Announcement

Collapse
No announcement yet.

How to perform actions after a File Reader poll is complete?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to perform actions after a File Reader poll is complete?

    This seems like something that shouldn't be hard to do, so I'm either being blind & missing an option, or just being stupid and not thinking of something obvious.

    I have a channel with a source type of File Reader. This channel runs hourly to poll for all files in one directory and move them to another directory. After it's completed processing all of the files, I need to change the value of some global variables (not after each file, but only after all of the files in the current poll have been processed).

    If it were a Database Reader source, there's an obvious "After all messages" option to run script, but I don't see the same for File Reader. Is there some trick to accomplishing this same functionality via a transformer or JavaScript Writer destination?

  • #2
    Personally I would run it all in a Source Transformer javascript. Then you know when files are done being written and even check to make sure. Then you can change all the variables you want from there.

    Comment


    • #3
      I don't think there is a way to do that with a file reader.

      You could do it with a javascript reader, but you'd basically need to load all of your files from that poll into memory, and I'm not sure if that's feasible. You could set a sourceMap variable on the last one as an indicator, and then check for the indicator in the response map of the destination or the post processor to know when the last one has been processed.

      If you share more about why you need to set a global variable after the polling batch is complete, maybe there is an alternate solution that is less messy.

      Comment


      • #4
        Originally posted by agermano View Post
        If you share more about why you need to set a global variable after the polling batch is complete, maybe there is an alternate solution that is less messy.
        We've recently started picking up a LOT more volume of ADT messages from certain senders. We have to do a batch send of those ADT messages to a recipient hourly at the most frequent. I had a multi-channel solution in place to assemble the batch message that had been working okay for awhile until the volume increase. In order to avoid running the heap out of memory, we had to cap the batches to 3,000 ADT messages per hourly run (and we're using all we are allotted on the server resources already).

        Now, we're getting enough messages that batching 3,000 messages hourly, 24 times a day, isn't enough to keep up, so I've tried working on this solution that will poll hourly, find all files received in that past hour, append them to a file on the file system (which doesn't cause the heap to run away with itself), and then my goal would be, after that process is done, use a 2nd channel (which would only run if the global variable flag is set) to move the assembled file to a location where it can be picked up via a cron'ned shell script to SFTP the files to the recipient.

        I don't want the batch file to be assembled in the directory where the cron script will pick it up, because it might run while the poller is still appending files to it, so I need to assemble it in a temp location and then move it only when the poll is finished.
        Last edited by jridderhoff; 03-07-2018, 04:03 PM.

        Comment


        • #5
          Hello,

          Originally posted by jridderhoff View Post
          I don't want the batch file to be assembled in the directory where the cron script will pick it up, because it might run while the poller is still appending files to it, so I need to assemble it in a temp location and then move it only when the poll is finished.
          Actually you could - without any programming.

          You can append incoming messages into a file named {DATE.get('YYYYMMDDHH')}.hl7, then set up other channel to execute hourly 5 after the hour and not to pick up files that were last modified in the last (let's say) 3-4 minutes.

          Another way to do this is to pick up individual messages without deleting them (set After processing action to None). Then in your source transformer get a File object from $('originalFilename') and $('fileDirectory') and read its properties. Finally you can delete relevant files using FileUtil.delete() method.
          Tom

          Comment


          • #6
            Are these files being moved around on the local file system? Are you doing any processing (reading or changing values) of the hl7 as they pass through? If you're strictly dealing with local files where you don't care about the content there are much faster and more memory/cpu/db efficient ways to move them than treating them as individual messages.

            Comment


            • #7
              Originally posted by agermano View Post
              Are these files being moved around on the local file system? Are you doing any processing (reading or changing values) of the hl7 as they pass through? If you're strictly dealing with local files where you don't care about the content there are much faster and more memory/cpu/db efficient ways to move them than treating them as individual messages.
              Nope, no processing of the individual messages themselves: they're literally being received from several senders, and then hourly batched together & sent to another recipient. None of the messages individually need to be loaded into Mirth, just moved around on the local file system.

              The only special processing we have to do is add the FHS/BHS & BTS/FTS segments to the batched message before it goes out, and name the file to a given standard that the recipient requires. Other than that, we don't do a thing with them.

              Comment


              • #8
                Implemented solution with JavaScript Reader

                Just to follow up, I was able to accomplish what I needed with a bit of custom FileUtils work in a JavaScript Reader source. There may be some ways to min/max this & make it a bit more efficient, but it's a vast improvement over the old way (tried with up to 10K messages, each ~1KB in size, and it handled it fine in about 10 seconds).

                The files to send get dropped into /path/to/files/in by another process, and then this channel assembles the batched message & moves it into the /path/to/cron folder where a cron job picks up the file & transmits it to the recipient via sFTP.

                Hopefully it'll help someone else, and I don't mind any kibbutzing either, if there's a more elegant/performant way to do this.

                Thanks all for the input!

                JavaScript Reader Source:
                Code:
                var pollDirectory = new java.io.File("/path/to/files/in/");
                var processedDirectory = new java.io.File("/path/to/files/out/");
                var assemblyDirectoryPath = "/path/to/files/assemble/";
                var finalizedDirectory = new java.io.File("/path/to/cron/");
                
                var foundFiles = pollDirectory.listFiles();
                
                if (foundFiles.length > 0) {
                	// found files to batch; build file names and IDs
                	var foundFileCount = 0;
                	var batchFileName = "FilenameToSendToReciptient" + DateUtil.getCurrentDate('dd-MM-yy_HH-mm-ss.S');
                	var outputFile = new java.io.File(assemblyDirectoryPath + batchFileName);
                	var currentDateStr = DateUtil.getCurrentDate('yyyyMMddHHmmssZ');
                	var controlId = DateUtil.getCurrentDate('yyyyMMddHHmmssS');
                
                	// add batch header segments
                	var headerSegments = "FHS|^~\&|SEND_APP|SEND_FACILITY|RCV_APP|RCV_FACIL|" + currentDateStr + "||||" + controlId + "\n";
                	headerSegments += "BHS|^~\&|SEND_APP|SEND_FACILITY|RCV_APP|RCV_FACIL|" + currentDateStr + "||||" + controlId + "\n";
                	org.apache.commons.io.FileUtils.writeStringToFile(outputFile, headerSegments, true);
                	
                	for each (var foundFile in foundFiles) {
                		foundFileCount++;
                		
                		// append each found file to the output file
                		org.apache.commons.io.FileUtils.writeStringToFile(outputFile, org.apache.commons.io.FileUtils.readFileToString(foundFile), true);
                		
                		// move each found file to processed directory
                		org.apache.commons.io.FileUtils.moveFileToDirectory(foundFile, processedDirectory, true);
                	}
                
                	// add batch footer segments
                	var footerSegments = "BTS|" + foundFileCount.toString() + "\n";
                	footerSegments += "FTS|1\n";
                	org.apache.commons.io.FileUtils.writeStringToFile(outputFile, footerSegments, true);
                
                	// move finalized file to be picked up by scheduled SFTP sender
                	org.apache.commons.io.FileUtils.moveFileToDirectory(outputFile, finalizedDirectory, true);
                
                	var results = <batchMessage><batchCount>{foundFileCount}</batchCount><batchFileName>{batchFileName}</batchFileName></batchMessage>;
                
                	return results;
                } else {
                	return null;
                }

                Comment


                • #9
                  This is probably more a mid-term than a short-term solution:

                  I altered the Mirth server code to provide a property in the source map that indicates polling is finished when the last file is processed by a file connector.


                  The property is called 'pollingFinished' and is only present for the last file of a poll. There it provides the total number of polled files. It can be checked from anywhere within the channel e.g. via
                  Code:
                  if(sourceMap.containsKey('pollingFinished')){
                       // do your clean-up work here
                       logger.info('Polling finished. Polled ' + sourceMap.get('pollingFinished') + ' files.');
                  }
                  Please find enclosed a patch file that can be applied to the latest codebase of mirth.

                  Not sure if the dev-team is willing to accept community contributions (it's not even possible to upload a patch-file, thus I had to zip it...)

                  However it would be great to see this in the upcoming release.
                  Attached Files

                  Comment


                  • #10
                    Maybe open a Jira ticket for a feature request and post your patch there?

                    http://www.mirthcorp.com/community/issues/browse/MIRTH

                    Comment


                    • #11
                      That's a pretty good solution you came up with. One thing you might consider is how recovery will work in case something happens during processing (exception thrown, server crash, whatever.)

                      A couple nitpicky things,

                      You could set foundFileCount to foundFiles.length instead of incrementing it in the loop.

                      You should probably return results.toString() since a JS Reader should return a String or RawMessage (or a List of them for multiple messages.)

                      Comment


                      • #12
                        Originally posted by agermano View Post
                        Maybe open a Jira ticket for a feature request and post your patch there?
                        JIRA-ticket can be found here: http://www.mirthcorp.com/community/i...wse/MIRTH-4252

                        Comment


                        • #13
                          Feature was added to Mirth Version 3.6.0:

                          Nick Rupley added a comment - 25/May/18 1:29 PM

                          The following variables are now put into the source map when using the File Reader:
                          • pollId: Unlike batchId, pollId is set to a unique string based on nanotime
                          • pollSequenceId: Similar to batchSequenceId, contains a number starting at 1 and incrementing
                          • pollComplete: Only present for the final message in a poll window, value is always tr
                          ue

                          Comment

                          Working...
                          X