Announcement

Collapse
No announcement yet.

SFTP error due to file size - trying to set up an alert.

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • SFTP error due to file size - trying to set up an alert.

    We have some issues with large files and SFTP. While others are looking at a solution, I was trying to get some kind of alert as opposed to just checking the logs. The error I get is:

    [2019-12-02 10:05:00,040] ERROR (com.mirth.connect.connectors.file.FileReceiver:43 1): Unable to dispatch message to channel XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX. File: smb://SERVERNAME001P/Files/App/Outbound/Charge_20191202.txt
    java.io.IOException: File Charge_20191202.txt is too large. Unable to read files greater than 2147483647 bytes.
    I can't get an alert to fire, because (I'm guessing) that this is failing before it gets to the channel. Can I alert on the FileReciever? I'm looking, but can't get that sorted. We are on 3.8.1 with Platinum support and extensions.

    (And we are trying to get smaller files sent, but that is a battle in itself)

    Thanks!
    Last edited by Brad_AZ; 12-02-2019, 08:17 AM.
    SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America

  • #2
    Just to verify, are you trying to pick files up from an SFTP location? If so, can you set the maximum limit on the file size on the source tab?
    Best,

    Kirby

    Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

    Appliance Version 3.11.4
    Mirth Connect Version 3.8.0
    Java Version 1.6.0_45-b06
    Java (64 bit) Version 1.6.0_45-b06
    Java 7 (64 bit) Version 1.7.0_151-b15
    Java 8 (64 bit) Version 1.8.0_181-b13
    PostgreSQL Version 9.6.8

    Comment


    • #3
      Sorry - we generically call our channels SFTP if they are either sending or picking up files via SFTP.

      This is a file pickup that SFTPs to a destination. So this issue is a file reader using smb where it can't ingest the file due to size.

      File size is set to 0 min, and Ignore Maximum is selected.
      SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America

      Comment


      • #4
        is this a valid file you expect to be able to process? If not, can you set the maximum file size to something appropriate?

        Does this file contain a batch of messages that you can parse using the batch feature in Mirth?

        Also, review the Mirth logs for errors.
        Best,

        Kirby

        Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

        Appliance Version 3.11.4
        Mirth Connect Version 3.8.0
        Java Version 1.6.0_45-b06
        Java (64 bit) Version 1.6.0_45-b06
        Java 7 (64 bit) Version 1.7.0_151-b15
        Java 8 (64 bit) Version 1.8.0_181-b13
        PostgreSQL Version 9.6.8

        Comment


        • #5
          Originally posted by kirbykn2 View Post
          is this a valid file you expect to be able to process? YES If not, can you set the maximum file size to something appropriate?

          Does this file contain a batch of messages that you can parse using the batch feature in Mirth NO

          Also, review the Mirth logs for errors.
          So the file size is a point of contention between the producers of the file and us, but it is a legit file. However, we might set the max to just below what Mirth can accept - then at least we should be able to capture the error at the channel level and alert on it.

          The only error I found was the original which included "Unable to read files greater than 2147483647 bytes.". So I will make the max just under that and lobby to have them send us smaller files. Maybe as them to switch to daily as opposed to weekly.

          Thanks!~
          SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America

          Comment


          • #6
            The issue is that the message needs to be converted to a byte[] at some point, and the max size for an array in Java is Integer.MAX_VALUE (2147483647.)

            If it is a delimited file that you can read as a batch, it will actually stream the file and you should be ok since no single message will go over the 2GB mark.

            The only other way to stream and split the file would be to process it yourself in a javascript reader.

            Comment


            • #7
              If you aren't parsing the file (just moving it) you can try setting your Data Types to RAW. That won't load the file into memory at all.

              -= Jack Haines : Founder/CEO of Healthcare Integrations, LLC
              -= [email protected]
              -= Mirth Connect (Advanced)-certified
              -= Gold member of HL7.org
              -= Available for Mirth Connect channel development and consultation! Schedule a FREE call with me at https://calendly.com/jackhaines

              Comment


              • #8
                Thanks for the suggestions. The main issue here was how to alert since it is failing to even get to the channel. I think I have that sorted now by setting max file size to just under the max value pointed out by agermano. This should generate an error and we will be alerted that it did not pass. Before, I found it by seeing the error in the dashboard.

                It was already set to RAW, so this error seems to appear earlier on in the process.

                Anyway - my task is to get this to alert, and I can do that now. Another team member is chasing the file size issue. If they come up with some solution that is out of the box. I'll do a separate post.
                SIG|1|Brad|Mirth Certified Interface Analyst^Cancer Treatment Centers of America

                Comment


                • #9
                  I believe if the file is bigger than the max size it will just get skipped by the File Reader unless you have tested and found otherwise.

                  I don't know if it is an oversight or what, but Mirth does not generate an event for Alerts to pick up when it encounters that problem. It will generate an error event of type SOURCE_CONNECTOR if it can't read the file or has issues moving or deleting the file.

                  The error you are seeing will still trigger the error action to allow you to move the file to a different folder (but only after you have downloaded it, and it fails.)

                  Comment

                  Working...
                  X