Announcement

Collapse
No announcement yet.

Parsing 834 out of memory

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Parsing 834 out of memory

    I'm a newbie of EDI trying to parse an 834 file and just simple turn it into an XML file. However Mirth keeps saying it's running out of memory.

    The 1st error:
    [2018-02-22 17:10:07,974] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 771): Java heap space
    java.lang.OutOfMemoryError: Java heap space

    The second:
    [2018-02-22 17:10:07,974] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 258): Error processing message in channel EDI_Test (7fe4cec7-8678-4163-bd45-332b12aa53dd).
    java.lang.OutOfMemoryError
    at com.mirth.connect.donkey.server.channel.Channel.ha ndleDestinationChainThrowable(Channel.java:1772)
    at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1718)
    at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1191)
    at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:192)
    at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:170)
    at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:354)
    at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
    at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
    at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)

    I'm enclosing the channel definition on this thread. Any advice/help on how to parse an EDI to it's component parts would be great.
    Attached Files

  • #2
    How large is the file? It may be that you just have to increase the server-side max heap size. If the file contains large embedded blob data (like an embedded image or PDF) you should consider using an Attachment Handler to extract that before the message processes through the channel. Or if it's a batch file containing many messages within, you may want to consider turning on Process Batch on the source settings. There's an example batch script for EDI/X12 on the public Slack group.
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      Tried to raise Heap Size

      Thanks for your reply. The file size is just 7 MB. (I added the new channel definition)

      I increased the heap size to 2 GB, I also set the channel to do batching and took a script that I think is the right one from the slack site and I still get this error:

      [2018-02-23 09:51:21,169] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 771): Java heap space
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Unknown Source)
      at java.lang.AbstractStringBuilder.ensureCapacityInte rnal(Unknown Source)
      at java.lang.AbstractStringBuilder.append(Unknown Source)
      at java.lang.StringBuffer.append(Unknown Source)
      at java.io.StringWriter.write(Unknown Source)
      at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:74)
      at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:113)
      at com.mirth.connect.plugins.datatypes.edi.EDIReader. parse(EDIReader.java:154)
      at com.mirth.connect.plugins.datatypes.edi.EDISeriali zer.toXML(EDISerializer.java:96)
      at com.mirth.connect.donkey.server.channel.FilterTran sformerExecutor.processConnectorMessage(FilterTran sformerExecutor.java:93)
      at com.mirth.connect.donkey.server.channel.Destinatio nConnector.transform(DestinationConnector.java:360 )
      at com.mirth.connect.donkey.server.channel.Destinatio nChain.doCall(DestinationChain.java:114)
      at com.mirth.connect.donkey.server.channel.Destinatio nChain.call(DestinationChain.java:63)
      at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1716)
      at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1191)
      at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchBatchMessage(SourceConnector.java:25 4)
      at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchBatchMessage(SourceConnector.java:19 6)
      at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:331)
      at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
      at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
      at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)
      Attached Files
      Last edited by tkurian; 02-23-2018, 06:01 AM. Reason: Adding New Channel Definition

      Comment


      • #4
        That must mean the file is not actually a batch of multiple messages, but instead one big message right?

        Is there any large blob content in the message that would make sense to extract as an attachment?

        Also just to confirm, how exactly did you increase the heap size? If you're launching the service/daemon, did you edit mcservice.vmoptions? If you're launching mcserver directly, did you edit mcserver.vmoptions? Or did you use the Server Manager? Or are you passing in command line options to Java?

        I also noticed you're doing this in the destination transformer:



        Was that just testing, or else why is that there? The "message" variable is not available from transformers to start with. And then there's no need to serialize the EDI/X12 to XML since the transformer will have already done that for you... the "msg" variable is your E4X XML object. Finally, you're taking that XML and putting it into the channel map, needlessly double-storing the entire message.
        Last edited by narupley; 02-23-2018, 07:28 AM.
        Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

        Nicholas Rupley
        Work: 949-237-6069
        Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


        - How do I foo?
        - You just bar.

        Comment


        • #5
          Next Attempt

          The message is an 834 EDI Enrollment file. The message allows for a repeatable section (INS segments) for individual enrollments. So yes it's one EDI file with multiple enrollments. There are no blobs, attachments or anything other than pure EDI in the file.

          Yes the destination transformer was for testing and I forgot to remove it. I have removed it now. (I've attached the new channel definition)

          I have also removed the batching and script this time and tried to drop the message. When I did this it didn't error out but just put out the same message I got from the source. In the hope of doing the basic I switch the outbound datatype to XML to see if I can just spit out EDI --> XML but I still get the infamous memory error again.

          I just have one basic question:

          How do I split this 834 to different individual enrollments (members - EDI INS segments) so that I can store each one in the database?


          [2018-02-23 12:05:09,250] ERROR (com.mirth.connect.donkey.server.channel.Channel:1 771): Java heap space
          java.lang.OutOfMemoryError: Java heap space
          at java.util.Arrays.copyOf(Unknown Source)
          at java.lang.AbstractStringBuilder.ensureCapacityInte rnal(Unknown Source)
          at java.lang.AbstractStringBuilder.append(Unknown Source)
          at java.lang.StringBuffer.append(Unknown Source)
          at java.io.StringWriter.write(Unknown Source)
          at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:74)
          at com.mirth.connect.model.converters.XMLPrettyPrinte r.endElement(XMLPrettyPrinter.java:113)
          at com.mirth.connect.plugins.datatypes.edi.EDIReader. parse(EDIReader.java:154)
          at com.mirth.connect.plugins.datatypes.edi.EDISeriali zer.toXML(EDISerializer.java:96)
          at com.mirth.connect.donkey.server.channel.FilterTran sformerExecutor.processConnectorMessage(FilterTran sformerExecutor.java:93)
          at com.mirth.connect.donkey.server.channel.Destinatio nConnector.transform(DestinationConnector.java:360 )
          at com.mirth.connect.donkey.server.channel.Destinatio nChain.doCall(DestinationChain.java:114)
          at com.mirth.connect.donkey.server.channel.Destinatio nChain.call(DestinationChain.java:63)
          at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1716)
          at com.mirth.connect.donkey.server.channel.Channel.di spatchRawMessage(Channel.java:1191)
          at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:192)
          at com.mirth.connect.donkey.server.channel.SourceConn ector.dispatchRawMessage(SourceConnector.java:170)
          at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:354)
          at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
          at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
          at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
          at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
          at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)
          Attached Files

          Comment


          • #6
            It sounds like what you want to do is modify that sample batch script so that it splits by enrollment. Right now it just splits by ISA segment.
            Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

            Nicholas Rupley
            Work: 949-237-6069
            Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


            - How do I foo?
            - You just bar.

            Comment


            • #7
              Thanks

              As you said I changed it to INS. I think now it's debatching it without errors not 100% in the right way but I think I'll take a look at the script a little closer and figure out why it's not putting 1 file for each enrollment.

              However it's still putting everything out in EDI format is there a way that I can dump it to the folder in XML format?

              Thanks for all your help
              Attached Files

              Comment


              • #8
                If you set your destination outbound data type to XML, the encoded data should automatically be converted to XML for you.
                Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

                Nicholas Rupley
                Work: 949-237-6069
                Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


                - How do I foo?
                - You just bar.

                Comment


                • #9
                  Thanks

                  Thanks it worked great!

                  Comment


                  • #10
                    I'm using Mirth 3.8.1 channel and I'm new to EDI/X12. Could you please share a sample channel to convert 837 files to their corresponding XMLs.

                    Regards,
                    Divya.

                    Comment

                    Working...
                    X