No announcement yet.

memory usage filereader

  • Filter
  • Time
  • Show
Clear All
new posts

  • memory usage filereader

    Since we migrated to 3.2.1 (from 2.X.X) we've seen that memory usage has increased, to the point that even 6GB isn't enough anymore.

    We're getting the infamous out of memory error for the heapspace.

    now, we DO read in (filereader source) files of up to about 200MB.
    and use a filewriter destination to another directory on the same system (with renaming etc of file).

    This reading seems to result in memory usage of far more than just the 200MB of the file itself.
    and what's worse, it seems like that memory isn't freed up after it's been used.
    resulting in out of memory.

    We have a channel that call system.gc(), but that doesn't have an effect.

    I've used a testchannel to read in these files using javascreipt souces.
    using listfiles
    and java.lang.Runtime.getRuntime().exec("cmd.exe /c move from_file to_file")

    this seems to work without all the overhead, but is less easy to maintain etc.

    is there some way of using a filereader filewriter solution that doesn't use so much memory?


  • #2
    Unfortunately, as far as I understand, Mirth keeps several different versions/copies of each message that is processed through a channel. So, if you read in 1 200 MB file, the memory impact is actually much higher as it stores a Raw, Transformed, and Encoded version of the message.


    • #3
      Mirth Connect 3.0 has additional features that require more memory usage than 2.x did.

      The message data is copied in memory at various points to allow for message transformations, so if you do not need to do any transformations on that 200MB, it is better to extract it as an attachment using an Attachment Handler (in the Channel Properties section when editing a channel).

      The attachment handler will extract the data when the message is received, before the pre-processor or any transformers are run, then will automatically re-attach on send. This will minimize the amount of memory usage.


      • #4
        yeah, that's what I expected.
        However, I can live with that, as long as that memory gets freed as soon as possible.
        Anecdotal evidence seems to suggest to me that that doesn't happen, and that therefore memory full errors are a matter of waiting.

        Sounds interesting.
        so how should I work this?
        The channel in question does a filereader of a specific directory.
        Renames some of the files found.
        Moves all of them to another directory.

        I'm not seeing where this attachment setting enters the scene.
        Please feel free to answer: RTFM.



        • #5
          I had similar issues with much smaller files, unfortunately I actually process those files so I need to split it out. My issue was with reprocessing, even displaying the message would cause the out of memory error. I was not able to get it resolved unfortunately.

          The only thing that I could suggest is to split the fie as soon as possible if it is a batch file, minimize/eliminate transformations, maybe even reduce the message storage slider (i'm not sure what that would do for memory consumption, but it would help with database usage if I understand correctly).


          • #6
            it's even worse for us, we are in this case not even interested in the data itself, since we are only interested in moving the file from one directory to another.

            possibly we'll just start using bat files etc which we wil start using Mirth.

            Thanks for replying.


            • #7
              Sounds interesting.
              so how should I work this?
              Since you aren't doing any transformations on the content, you can extract the entire contents of each file as the attachment. This means as soon as the file is read, the content will be extracted and excluded from the transformation steps, which is what duplicates the content in memory. When the file is written back out it will re-attach the content immediately before writing. That's how it keeps the memory usage to a minimum.

              The easiest way to do that would be to use a JavaScript attachment handler with the following one-liner:
              return addAttachment(message, "text/plain").getAttachmentId();
              See attached screenshot
              Attached Files


              • #8
                that works, on our system it takes less memory AND no problems wit memory overflow), thanks.

                In the userguide (3.1, october 2014) it didn't make it clear for me how attachments work, this goes quite a ways to do that.
                Is there somewhere I can read some more (including examples if possible) about these (for me) obscure parts of MirthConnect

                We'll try this out on our clients installation soon, and see where that takes us.

                Thanks to all responders.



                • #9
                  Is there somewhere I can read some more (including examples if possible) about these (for me) obscure parts of MirthConnect
                  For non-subscribers / open-source users, the resources you have available are:

                  More resources are available with a paid subscription.

                  In many cases, just searching the forums will get you the info you need. The user guide does give an overview of the attachment handlers, however it's not clear that attachment handlers are the way to resolve memory issues with large messages. I'll make a note to clarify that in future revisions of the user guide.


                  • #10
                    once again, thanks.