Announcement

Collapse
No announcement yet.

Mirth message volume

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Mirth message volume

    I am investigating using Mirth to replace an existing HL7 interface. The current interface has hit the proverbial wall with message volume...to the point where it's throwing errors, failing to process some messages, and even completely dropping some messages.

    The current interface receives hospital ADT data (roughly 4000-6000 message per day), and maintains multiple databases (SQL Server) for various department functions as well as interactive reporting services. It's very database intensive. Virtually every ADT message that comes over prompts some type of datawrite activity.

    Should I expect Mirth to handle this volume?

  • #2
    Re:Mirth message volume

    4000-6000 per day? Thats it? :P

    We've got a Mirth instance running on light hardware (VM with Ubnutu 7.10, 512MB RAM and 10GB disk) that handled 30000 messages per day during an import operation without blinking. Those operations were largely simple routing (message over VPN, into our datacentre, routed to correct customer EMR instance) and basic database writes.

    What is your current interface doing? Is it simply stuffing messages into a database or is it applying some conversion/transformation/logic to those messages? I assume your reporting is being done with a reporting tool and not your existing interface?

    You say that the existing setup is "database intensive". Does that simply mean that most of the HL7 data gets stuffed into a database or that the interface itself is required to do a bunch of work with the database before writing the data?

    In the last 3 years we've only come across one case where we decided that it was better to roll our own interface instead of using Mirth and that was largely because of a janky deployment scenario than technical limitations.

    In addition to being technically superior, Mirth comes with a reasonably active community (as you can see) and the Mirth Team/Webreach is very responsive in fixing bugs and offering support. So if you're able to provide more details about what your interface needs to do, there are likely people around who have done something similar (or at least one portion of it) with Mirth already.

    I keep telling people this in my company, at my customers, and the people I know in the software community. Mirth kicks ass.
    Jon Bartels

    Zen is hiring!!!!
    http://consultzen.com/careers/
    Talented healthcare IT professionals wanted. Engineers to sales to management.
    Good benefits, great working environment, genuinely interesting work.

    Comment


    • #3
      Re:Mirth message volume

      Mirth kicks ass
      So far, I'm inclined to agree with you on that...

      The current interface takes an incoming ADT stream from our HIS, and fires a script which determines the message type (A01, A02, etc...) and preforms the required manipulations of a remote SQL Server database.

      That database is the backend for several in-house systems that support multiple department's data needs (Bed Control, Env Service, Case Management, Quality Management, etc..) and also serves as the data warehouse for multiple reporting tools.

      Our current interface has hit the wall in terms of message volume processing. It randomly throws errors when it tries to open a connection to the database that hasn't finished closing from the prior message execution...and it randomly drops messages completely (presumably becuase it's tied up processing something).

      Both issues are related to volume. Until the interface was recently expanded to cover outpatient messages, it clicked along like clockwork for several years with virtually no issues. If I modifying it to reduce the number of messages it needs to process, the errors go away.

      We've explored upgrading the machine the interface runs on, but it's not really a memmory issue, and the software isn't able to take advantage of multiple processors. That's primarily why we're looking into alternative interface solutions (not to mention that problem of the current interface being a huge memory leak...nessecitating the server be rebooted every 12 hours).

      Probably more than you wanted to know...but that's how we arrived at testing Mirth.

      I'm hoping someone has some practical experience using Mirth for similar needs...I'm trying to avoid spending a weekend converting my Mirth testbed into a full-blown load testing model

      Comment


      • #4
        Re:Mirth message volume

        MichaelP wrote:
        The current interface takes an incoming ADT stream from our HIS, and fires a script which determines the message type (A01, A02, etc...) and preforms the required manipulations of a remote SQL Server database.
        [/quote]

        So if I'm reading that right, all it does is run a few queries or stored procedures. Any heavy lifting is done by that query/stored proc on the database server and no heavy logic is done at your interface.

        Probably more than you wanted to know...but that's how we arrived at testing Mirth.
        Thanks for the review. My team writes software that has to interoperate with things like Mirth, PMs, EMRs, and HISs so the more I hear about issues the more I can do to make our code not suck to make everyones life a lot easier.

        I'm hoping someone has some practical experience using Mirth for similar needs...I'm trying to avoid spending a weekend converting my Mirth testbed into a full-blown load testing model
        If you look at my posts from about 2 years ago I developed a plan to load test Mirth. It never got implemented. The easier way to test it is to siphon off messages from your existing interface (either with Mirth as a simple router or some TCP load-balancer hacking), have Mirth run those messages into a testing database, and then compare the testing database with the production one.

        Another option that will leverage Mirth and allow you to gradually phase it in would be to use it as a throttle on your traffic. If you can spread those 6000 messages out from 8 hours to 24 hours your existing interface gets to live a little longer and Mirth gets a shakedown. It'd take a bit of thought to get it right, but you can simply set up a wait-loop in a Mirth channel using javascript.

        As far as deployment, multi-processor, etc. Mirth runs on Java. As long as you can deploy Java apps on a server with a good JVM (Sun-Java only IMHO) Mirth will run quite well. Like all Java apps it WILL suck up all your free RAM if you let it, set the XMax option appropriately. You'll also want to run Mirth itself on something other than its internal Derby database. I've had good results with both Postgres and SQL Server.
        Jon Bartels

        Zen is hiring!!!!
        http://consultzen.com/careers/
        Talented healthcare IT professionals wanted. Engineers to sales to management.
        Good benefits, great working environment, genuinely interesting work.

        Comment


        • #5
          Re:Mirth message volume

          MichaelP wrote:
          Mirth kicks ass
          So far, I'm inclined to agree with you on that...

          The current interface takes an incoming ADT stream from our HIS, and fires a script which determines the message type (A01, A02, etc...) and preforms the required manipulations of a remote SQL Server database.

          That database is the backend for several in-house systems that support multiple department's data needs (Bed Control, Env Service, Case Management, Quality Management, etc..) and also serves as the data warehouse for multiple reporting tools.

          Our current interface has hit the wall in terms of message volume processing. It randomly throws errors when it tries to open a connection to the database that hasn't finished closing from the prior message execution...and it randomly drops messages completely (presumably becuase it's tied up processing something).

          Both issues are related to volume. Until the interface was recently expanded to cover outpatient messages, it clicked along like clockwork for several years with virtually no issues. If I modifying it to reduce the number of messages it needs to process, the errors go away.

          We've explored upgrading the machine the interface runs on, but it's not really a memmory issue, and the software isn't able to take advantage of multiple processors. That's primarily why we're looking into alternative interface solutions (not to mention that problem of the current interface being a huge memory leak...nessecitating the server be rebooted every 12 hours).

          Probably more than you wanted to know...but that's how we arrived at testing Mirth.

          I'm hoping someone has some practical experience using Mirth for similar needs...I'm trying to avoid spending a weekend converting my Mirth testbed into a full-blown load testing model
          First of all, you're going to want to make sure to use postgresql for the backend of mirth.

          Next, if you are getting any problems connecting to the database, then there might be a problem establishing multiple connections to the database. Perhaps there are some DB settings you could mess with. Another thing you might want to look at for increasing performance is implementing a form of database connection pooling. You can create a database connection and then store it in the global map on deploy. To execute a javascript database update/query, simply grab the connection out of the globalmap, and call your execute method on it. You may also want to make sure it's still connected each time, and if not, reconnect. This alone may really help performance if you have any issues.

          Good luck!

          Post edited by: jacobb, at: 05/08/2008 13:16
          Jacob Brauer
          Director, Software Development
          NextGen Healthcare

          sigpic

          Comment


          • #6
            Re:Mirth message volume

            And on a side note, we've gotten huge performance increases running Mirth on multiple processors or multiple core machines. Our performance tests on systems with 8 cores have had considerable increases over systems with 4 cores.
            Jacob Brauer
            Director, Software Development
            NextGen Healthcare

            sigpic

            Comment


            • #7
              Re:Mirth message volume

              So if I'm reading that right, all it does is run a few queries or stored procedures. Any heavy lifting is done by that query/stored proc on the database server and no heavy logic is done at your interface.
              Sadly, the existing (legacy) system does a fair amount of the heavy lifting itself. Over the years I've supported it I've incrementally moved things over to the database side. A complete re-write would do it good, but there are other issues with the software and I'd just as soon move to something better anyway.

              If you look at my posts from about 2 years ago I developed a plan to load test Mirth. It never got implemented. The easier way to test it is to siphon off messages from your existing interface (either with Mirth as a simple router or some TCP load-balancer ), have Mirth run those messages into a testing database, and then compare the testing database with the production one.
              I'm mirroring over my production database structure onto a test server now. Once I fully configure Mirth's Channel to accept the data I can configure a duplicate outbound interface from our HIS and give Mirth a full load test.

              You'll also want to run Mirth itself on something other than its internal Derby database. I've had good results with both Postgres and SQL Server.
              That's good to know.

              And on a side note, we've gotten huge performance increases running Mirth on multiple processors or multiple core machines. Our performance tests on systems with 8 cores have had considerable increases over systems with 4 cores.
              That's exactly what I wanted to know. I'm budgetting for a new interface machine this summer.

              Comment


              • #8
                Re:Mirth message volume

                MichaelP wrote:
                That's exactly what I wanted to know. I'm budgetting for a new interface machine this summer.
                http://www.webreachinc.com/products/mirthappliances
                Jon Bartels

                Zen is hiring!!!!
                http://consultzen.com/careers/
                Talented healthcare IT professionals wanted. Engineers to sales to management.
                Good benefits, great working environment, genuinely interesting work.

                Comment


                • #9
                  Re:Mirth message volume

                  MichaelP, I too would recommend looking into an Enterprise cluster from us. They are all optimized for Mirth, so you'll get great message throughput and a really stable machine with a lot of neat extra features.
                  Jacob Brauer
                  Director, Software Development
                  NextGen Healthcare

                  sigpic

                  Comment


                  • #10
                    Re:Mirth message volume

                    In addition to the above, I've got a rate of +300.000 msg/hour taking data with a SOAP listener and writing the message to a remote SFTP server. All the messages are stored in the database (Oracle 10) installed in the same box.

                    These tests were performed with a Dell 8 core box, linux ... etc. Is an enterprise class server.

                    I think that Mirth really doesn't have any problem with the performance, as 300.000msg/hour is at least 1000 times higher than any rate I've never seen in a production environment.

                    Comment


                    • #11
                      Re:Mirth message volume

                      jbartels wrote:
                      Like all Java apps it WILL suck up all your free RAM if you let it, set the XMax option appropriately. You'll also want to run Mirth itself on something other than its internal Derby database. I've had good results with both Postgres and SQL Server.
                      I tried java -X to review the advanced options, I don't see anything like XMax unless you are talking about -Xmx? What do you set it to? Example? Thanks
                      Mike Caldwell
                      Alliance HealthCare - GE VAR
                      Rocklin, CA
                      Centricity PM/EMR Support - Developer - Network Engineer

                      Comment


                      • #12
                        Re:Mirth message volume

                        MichaelP wrote:
                        I am investigating using Mirth to replace an existing HL7 interface. The current interface has hit the proverbial wall with message volume...to the point where it's throwing errors, failing to process some messages, and even completely dropping some messages.

                        The current interface receives hospital ADT data (roughly 4000-6000 message per day), and maintains multiple databases (SQL Server) for various department functions as well as interactive reporting services. It's very database intensive. Virtually every ADT message that comes over prompts some type of datawrite activity.

                        Should I expect Mirth to handle this volume?
                        FYI. With the standard config, last night I seen Mirth pull in 68000+ messages in a couple minutes if that long and didn't sweat it and continues to do 12-15,000 a day without problems that I can see so far and these are LABs as well (ORU's) with quite a bit more data than the ADT's which it is also doing.
                        Mike Caldwell
                        Alliance HealthCare - GE VAR
                        Rocklin, CA
                        Centricity PM/EMR Support - Developer - Network Engineer

                        Comment


                        • #13
                          Re:Mirth message volume

                          quimicefa wrote:
                          In addition to the above, I've got a rate of +300.000 msg/hour taking data with a SOAP listener and writing the message to a remote SFTP server. All the messages are stored in the database (Oracle 10) installed in the same box.

                          These tests were performed with a Dell 8 core box, linux ... etc. Is an enterprise class server.

                          I think that Mirth really doesn't have any problem with the performance, as 300.000msg/hour is at least 1000 times higher than any rate I've never seen in a production environment.
                          Does your channel have a source transformer or filter? How about a destination transformer or filter? What version of mirth are you using? I'm interested to see what type of channel is getting you these numbers in your environment. If you wouldn't mind, perhaps you could e-mail me the channel at [email protected].

                          A different channel makes all the difference in the world. Some connectors take longer to establish connections or receive data. If you have a filter/transformer then your data is generally serialized from HL7 to XML which also greatly slows down processing. With a simple optimized routing channel, I wouldn't be entirely surprised to see numbers higher than 1000 messages per second.
                          Jacob Brauer
                          Director, Software Development
                          NextGen Healthcare

                          sigpic

                          Comment


                          • #14
                            Re:Mirth message volume

                            M56969 wrote:
                            I tried java -X to review the advanced options, I don't see anything like XMax unless you are talking about -Xmx? What do you set it to? Example? Thanks
                            My bad. I was referring to -Xmx.

                            You set the -Xmx option for Mirth indirectly in <mirth dir>/conf/wrapper.conf on the line that reads "wrapper.java.maxmemory=128" . On my testing/demo systems I usually use 128-256MB. On the dedicated Mirth boxes I set it as high as possible for the other processes and available RAM in the system.

                            That limit is there to keep Java from sucking up all of your free memory so be careful with it. It is usually best to leave it alone and only tune it if you notice a slow-down or if the machine in question still has available memory when it is under load.

                            It's not exactly a scientific approach, but it usually works to eke out the best performance from a Java application while ensuring it doesn't crowd out other services running on a machine.
                            Jon Bartels

                            Zen is hiring!!!!
                            http://consultzen.com/careers/
                            Talented healthcare IT professionals wanted. Engineers to sales to management.
                            Good benefits, great working environment, genuinely interesting work.

                            Comment


                            • #15
                              Re:Mirth message volume

                              The test were performed with Mirth 1.7.1, and a simple channel that listens on a webservices and writes the raw data to a remote SFTP. There are no filter or transformers at all ... I didn't repeated the tests with filters/some JS, ...

                              Comment

                              Working...
                              X