Announcement

Collapse

Mirth Connect 4.0.1 Released!

Mirth Connect 4.0.1 is now available as an appliance update and on our GitHub page. Mirth Connect 4.0.1 is a patch release containing a bug fix which includes fixing a Jetty keystore regression that caused Connect servers using a PKCS12 keystore containing a wildcard certificate and/or a certificate with a SAN to throw an exception on startup. See the release notes for the list of fixes and updates.

Download | See What's New | Upgrade Guide | Release Notes

For discussion on this release, see this thread.
See more
See less

File Writer in SFTP mode errors out reading/writing large files

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • File Writer in SFTP mode errors out reading/writing large files

    Hi All,

    I have a very basic channel that reads a local file and then writes it out to an sftp remote. The file is about 200M and the error I get pretty consistently is shown below. Is there any way to deal with this? I've increased heap size in mirth configs to 2G, but that does not seem to help.

    Thank you,
    Stan

    ---
    [2019-08-20 06:34:53,748] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 288): Error processing message in channel largeFileTest (86583101-d0ee-4982-bc30-7c3075de3463).
    com.mirth.connect.donkey.server.channel.ChannelExc eption:
    at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1243)
    at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:192)
    at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:170)
    at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:418)
    at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:328)
    at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:239)
    at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)Caused by: com.mirth.connect.donkey.server.data.DonkeyDaoExce ption: java.sql.BatchUpdateException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
    at com.mirth.connect.donkey.server.data.jdbc.JdbcDao. executeBatchInsertMessageContent(JdbcDao.java:254)
    at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.executeTasks(BufferedDao.java:112)
    at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.commit(BufferedDao.java:85)
    at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.commit(BufferedDao.java:67)
    at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1723)
    at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1221)
    ... 8 moreCaused by: java.sql.BatchUpdateException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl .newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl .newInstance(NativeConstructorAccessorImpl.java:62 )
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessor Impl.newInstance(DelegatingConstructorAccessorImpl .java:45)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCalle r(Constructor.java:500)
    at java.base/java.lang.reflect.Constructor.newInstance(Construc tor.java:481)
    at org.apache.derby.impl.jdbc.Util.newBatchUpdateExce ption(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedStatement.executeL argeBatch(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedStatement.executeB atch(Unknown Source)
    at com.mirth.connect.donkey.server.data.jdbc.JdbcDao. executeBatchInsertMessageContent(JdbcDao.java:251)
    ... 13 moreCaused by: java.sql.SQLException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.g etSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.newEmbedSQLExcepti on(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.javaException(Unkn own Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl .wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl .handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleE xception(Unknown Source)
    at org.apache.derby.impl.jdbc.ConnectionChild.handleE xception(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedStatement.executeS tatement(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedPreparedStatement. executeBatchElement(Unknown Source)
    ... 16 moreCaused by: java.sql.SQLException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.get SQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.w rapArgsForTransportAcrossDRDA(Unknown Source)
    ... 25 moreCaused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.derby.iapi.services.io.DynamicByteArray OutputStream.<init>(Unknown Source)
    at org.apache.derby.impl.store.raw.data.BasePage.inse rtAllowOverflow(Unknown Source)
    at org.apache.derby.impl.store.raw.data.BasePage.inse rt(Unknown Source)
    at org.apache.derby.impl.store.access.heap.HeapContro ller.doInsert(Unknown Source)
    at org.apache.derby.impl.store.access.heap.HeapContro ller.insertAndFetchLocation(Unknown Source)
    at org.apache.derby.impl.sql.execute.RowChangerImpl.i nsertRow(Unknown Source)
    at org.apache.derby.impl.sql.execute.InsertResultSet. normalInsertCore(Unknown Source)
    at org.apache.derby.impl.sql.execute.InsertResultSet. open(Unknown Source)
    at org.apache.derby.impl.sql.GenericPreparedStatement .executeStmt(Unknown Source)
    at org.apache.derby.impl.sql.GenericPreparedStatement .execute(Unknown Source)
    ... 18 more
    ---

  • #2
    If you are only moving and not processing files, have you tried using an attachment?
    • Set your attachment option on the channel summary tab to "entire message." You can set the mime type in the attachment options.
    • Set your inbound and outbound types for all of your connectors to Raw.
    • Set your File Writer template to ${message.encodedData}
    • Make sure you have "Reattach Attachments" turned on (it should already be on by default.)


    When you view your message in the browser, all you should see is the replacement token, but there should also be an "Attachments" tab where you can see your attachment.

    With files this size, you may also want to check "Remove Attachments on Completion" under the storage options on your channel summary tab. In that case, you probably won't see the attachment tab in the message browser.

    An alternate option that could be even faster, but more work for you, is to do it entirely in javascript. If you directly access the file and start your own sftp session with jsch (the library mirth uses) you could stream data from the file to the remote without needing to read the entire file into memory in the first place or store it in the mirth database. The only case in which a File Reader does not read the entire file into memory at once is if you have your inbound data type set to Delimited and are splitting the batch line by line.

    Comment


    • #3
      We ended up using a shell script to move the files after all, but that isn't an option today. Tried attachment approach, but I still get OutOfMemoryError exception, appears to be happenning in the source, so the message does not even appear in the Message Browser. It's a text file, perhaps I need to set mime type to something for this to work?

      [2021-01-15 10:10:39,827] ERROR (com.mirth.connect.connectors.file.FileReceiver:27 ): Error processing file in channel: fd3b1cbc-9ce1-42df-bb8f-090587f53ce2
      com.mirth.connect.connectors.file.FileConnectorExc eption: Error reading file C:\tmp\OrderProducts.json Java heap space
      at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:436)
      at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:328)
      at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:239)
      at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)Caused by: java.lang.OutOfMemoryError: Java heap space

      Any help would be greately appreciated!

      Comment


      • #4
        Interesting, I can transfer the mirth tar.gz (219M) with agermano 's instructions.

        image.png

        And:


        image2.png

        and heap size on the server side is


        image3.png
        Diridium Technologies, Inc.
        https://diridium.com

        Comment


        • #5
          Hi everyone, I'm new here but have been using Mirth for several years. This month we migrated from windows server 2012 to 2019 and from Mirth 3.4 to 3.11.
          Everything seems fine excepts for the attachments handlers: with the 3.4 version we could transfer files to sftp up to 2gb (that was the Java Heap Size defined), but with the 3.11 version - in spite of having configured up to 4gb heap space - files bigger than 700mb aren't processed due to OutOfMemory java heap size error.
          The channel is configured the way agermano described but the error persists...
          I've checked every topic about this kind of issue over the internet but I keep stuck with this problem. Any clue will be really appreciated...

          Comment


          • #6
            What kind of message logging do you have the channel set to? Have you tried disabled logging or only keeping metadata for the messages to see if it helps? I know each Mirth channel makes a few copies of each message by default.

            Comment


            • #7
              Everything is at the bare minimum in order to work. Message storage is at RAW level, less than that and attachments can't be reattached...
              You do not have permission to view this gallery.
              This gallery has 1 photos.

              Comment


              • #8
                I do see similar behavior - although I am able to write up to 1G so far. acardoso - in my opinion, mirth is not the best tool for this kind of thing. I would use a command line SFTP client that has smart recovery, retries, etc. I would say something similar for large CSV imports - in that case use a native DB CLI option to import, it is far faster than mirth.
                Diridium Technologies, Inc.
                https://diridium.com

                Comment


                • #9
                  Maybe change the log level to none in the channel summary and see if it helps.
                  HL7v2.7 Certified Control Specialist!

                  Comment

                  Working...
                  X