Announcement

Collapse
No announcement yet.

Unable to manage Retries with FTP FileWriter

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Unable to manage Retries with FTP FileWriter

    Hi,

    when the FTP destination server is down or due to the network we get the following error. We have all permissions on the destination directory. If I resend this messages, it works fine, but the order of HL7 messages is not respected, this is a major feature.

    ERROR-403: File connector error
    ERROR MESSAGE: Error writing file
    java.net.SocketException: Connection reset


    It's impossible to retry sending automatically the message before processing the next message in case of error.

    Could you add to the FileWriter two additionnal parameters to manage retries.
    - maxRetries : number of retries
    - RetryWaitingTime : waiting time between 2 retries

    see post : http://www.mirthcorp.com/community/f...ead.php?t=5581

    Most integration engines have this functionnality.

    Thanks for your response

  • #2
    Mirth does provide this functionality for its TCP, LLP, HTTP, and Web Service Senders. I'm sure different people will have different views on this, but FTP by its very nature is just dumping files into a remote directory either one at a time or in daily batches, etc. So, it's not really tailored to be used as a live feed where order needs to be preserved.

    That said, there's an easy way around it in Mirth. Set up an inbound and outbound channel. The inbound channel sends to the outbound channel via LLP, with persistent queues turned on. Then, the outbound channel writes the file via FTP, and if it was successfully written, it can create a custom response variable which can then be sent back to the inbound channel.

    I've attached a couple of sample channels to show you what I mean. Here's the code I used in the postprocessor; feel free to alter it as you see fit:

    Code:
    if ($r('FTP_TEST_OUT').getStatus() == 'SUCCESS') {
    	var msg = new XML(SerializerFactory.getHL7Serializer().toXML(messageObject.getRawData()));
    	var ack = <HL7Message/>;
    	ack.MSH['MSH.1'] = msg.MSH['MSH.1'].toString();
    	ack.MSH['MSH.2'] = msg.MSH['MSH.2'].toString();
    	ack.MSH['MSH.3'] = msg.MSH['MSH.5'].copy();
    	ack.MSH['MSH.4'] = msg.MSH['MSH.6'].copy();
    	ack.MSH['MSH.5'] = msg.MSH['MSH.3'].copy();
    	ack.MSH['MSH.6'] = msg.MSH['MSH.4'].copy();
    	ack.MSH['MSH.7']['MSH.7.1'] = DateUtil.getCurrentDate('yyyyMMddHHmmss');
    	ack.MSH['MSH.9']['MSH.9.1'] = msg.MSH['MSH.9']['MSH.9.1'].toString();
    	ack.MSH['MSH.9']['MSH.9.2'] = msg.MSH['MSH.9']['MSH.9.2'].toString();
    	ack.MSH['MSH.9']['MSH.9.3'] = 'ACK';
    	ack.MSH['MSH.10'] = msg.MSH['MSH.10'].copy();
    	ack.MSH['MSH.11'] = msg.MSH['MSH.11'].copy();
    	ack.MSH['MSH.12'] = msg.MSH['MSH.12'].copy();
    	ack.MSA['MSA.1']['MSA.1.1'] = 'AA';
    	ack.MSA['MSA.2']['MSA.2.1'] = msg.MSH['MSH.10']['MSH.10.1'].toString();
    	ack.MSA['MSA.3']['MSA.3.1'] = 'File sent successfully';
    	responseMap.put('ACK',ResponseFactory.getSuccessResponse(SerializerFactory.getHL7Serializer().fromXML(ack)));
    }
    return;
    Attached Files
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      This is great info from narupley.. I am going to steal some of these ideas to retain msg persistency & retries in some of my LLP feeds.

      If you are after a file writer that has retries here is one that works for me:

      Step 1:
      Create a function in code templates so you can re-use this on other channels:
      Code:
      function ftpWrite(ftpserver, ftpport, ftpuser, ftppass, ftppassive, ftptimeout, 
      						ftpfile, ftppath, ftpappendfile, ftpByteArray,
      						maxRetries, reconnectWaitSeconds) {
      
      	// Do the ftp connection and send file
      	var returnBoolean = false;
      	var connected = false;
      	var retryCount = 0;
      
      	// setup ftp connection
      	while (! connected && retryCount++ < maxRetries) {
      
      		if (retryCount > 1) {
      			// not the first attempt wait a bit, maybe it will come back online shortly
      			java.lang.Thread.sleep(reconnectWaitSeconds * 1000);
      		}
      
      		try {
      
      			ftpConn = new Packages.com.mirth.connect.connectors.file.filesystems.FtpConnection(ftpserver, 
      												ftpport, ftpuser, ftppass, ftppassive, ftptimeout);
      		
      			if ( ftpConn == null || ! ftpConn.isConnected() ) {
      				connected = false;
      			} else {
      				connected = true;
      			}
      		} catch(e) {
      				logger.error("Ftp connection attempt failed for server:  " + ftpserver
      					+ ".  Reconnection attempt #" + retryCount + " of " + maxRetries
      					+ ", waiting "
      					+ reconnectWaitSeconds + " seconds before retry.");
      				connected = false;
      		}
      	}
      
      	// if ftp connection was successful then write the file to the ftp server
      	try {
      		if ( connected ) {
      		
      			ftpConn.writeFile(ftpfile, ftppath, ftpappendfile, ftpByteArray);
      			returnBoolean = true; 
      			ftpConn.destroy();
      		} else {
      			logger.error("Unable to connect to ftp server:  " + ftpserver
      					+ ".  Max retry limit exceeded. ");
      		}
      	} catch(e) {
      			logger.error("ftp connection OK, however file write failed. Server: " 
      						+ ftpserver + ", Filename: " + ftpfile );
      	}
      
      	// return the status of the function
      	return returnBoolean;
      
      }
      Step 2:
      In your channel, create a destination "JAVASCRIPT WRITER" and re-use this code with your params.
      Code:
      var ftpserver = "myftpserver";
      var ftpport = 0;				// zero for default ftp port
      var ftpuser = "myuser";
      var ftppass = "mypass";
      var ftppassive = true;			// ftp connection type
      var ftpappendfile = true;		// append or overwrite
      var ftptimeout = 10000;		// in milliseconds
      var ftpfile = "ftpfilename-" + DateUtil.getCurrentDate('yyyy-M-d') + ".txt";
      var ftppath = "myfilepath/";
      var maxRetries = 1500;
      var reconnectWaitSeconds = 60;
      var ftpSuccess = false;
      
      // build string that is going to be written to ftp file
      // it needs to be stored in a byteArray
      var msgString = new java.lang.String( $('resultString') );
      var ftpByteArray = msgString.getBytes();
      
      // call function that writes ftp file
      ftpSuccess = ftpWrite(ftpserver, ftpport, ftpuser, ftppass, ftppassive, ftptimeout, 
      						ftpfile, ftppath, ftpappendfile, ftpByteArray,
      						maxRetries, reconnectWaitSeconds);
      
      if ( ! ftpSuccess ) {
      
      	logger.error("Error writing ftp file with message: " + msgString);
      	responseMap.put(connector, ResponseFactory.getFailureResponse("Error writing ftp file with message: " + msgString));
      }
      In the above case it retries 1500 times and waits 1min between retries, plus it logs the failure in mirth. Since I monitor mirth this works fine for me in practice as the msg has to be stuck for 25 hours before it goes on to try the next one.. we monitor 24/7 so this should never happen.

      p.s. $('resultString') is the string of data I am passing from a transformer.. so it's the data I am wanting to write to the ftp server.

      Comment


      • #4
        Automatic reprocess of errored messages

        Another solution to automatically reprocess errored message is proposed here:



        http://www.mirthcorp.com/community/f...5487#post25487


        .

        Comment


        • #5
          Originally posted by tactics View Post
          Another solution to automatically reprocess errored message is proposed here:



          http://www.mirthcorp.com/community/f...5487#post25487


          .
          I'll just copy my response here just in case:

          Originally posted by narupley View Post
          This is a great alternative!

          It looks like this is tailored specifically for channels that are using both a vanilla File Reader (reading from local directory) and a File Writer. So, if you used a File Reader that's polling via FTP from a remote server, that method wouldn't work (though you could make it work with some extra code). Also, what if you had a different source connector, like an LLP Listener?

          In your postprocessor, instead of placing the file back in a source directory, another way to simulate this is simply to re-route the message directly to your channel:

          Code:
          if ($r('Destination 1').getStatus() != 'SUCCESS')
          	router.routeMessageByChannelId(channelId,messageObject.getRawData(),true);
          However, it's important to note that in general, this method is basically a rotating queue. That is, if the order of messages is important (as it certainly is in most registration and lab HL7 feeds) then you would probably want to stick to a persistent queue simulation that isn't rotating.

          Just my two cents!

          EDIT: Also, as far as overwriting statistics goes, this might help: http://www.mirthcorp.com/community/f...13&postcount=4
          Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

          Nicholas Rupley
          Work: 949-237-6069
          Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


          - How do I foo?
          - You just bar.

          Comment


          • #6
            Hi Narupley,
            thanks for your tip.

            In order to decrement the error statistics, I added this to my code:



            var channelStatisticsController = Packages.com.mirth.connect.server.controllers.Chan nelStatisticsController.getInstance();
            channelStatisticsController.decrementErrorCount(ch annelId);

            Comment

            Working...
            X