Announcement

Collapse
No announcement yet.

Newest version of Mirth still deadlocks

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Newest version of Mirth still deadlocks

    Hello,

    I was getting no deadlocks with 2.2, a lot with 3.0.0 and now less with the newest version - 3.0.2.7140.

    Is this still an open bug? Any resolution?

    Thanks,
    Scott

  • #2
    You're going to need to supply more information than that.
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      My situation is about the same as this:

      http://www.mirthcorp.com/community/i...wse/MIRTH-3042

      Inbound channels that are MLLP listeners sending to multiple channel-reader type destination - two in my case. Second destination is already set to wait for previous destination.

      Code:
      [2014-05-13 17:06:01,093]  ERROR (com.mirth.connect.donkey.server.channel.Channel:1530): An error occurred in channel AHC-HDL-ORU-001-IN Observation Result (05eeb396-f945-4302-8fae-df165bd4ceec) while processing message ID 646 from the source queue
      com.mirth.connect.donkey.server.data.DonkeyDaoException: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
      	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.updateStatus(JdbcDao.java:728)
      	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.executeTasks(BufferedDao.java:107)
      	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:74)
      	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:56)
      	at com.mirth.connect.donkey.server.channel.Channel.process(Channel.java:1394)
      	at com.mirth.connect.donkey.server.channel.Channel.processSourceQueue(Channel.java:1528)
      	at com.mirth.connect.donkey.server.channel.Channel.run(Channel.java:1515)
      	at java.lang.Thread.run(Unknown Source)Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
      	at java.lang.reflect.Constructor.newInstance(Unknown Source)
      	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
      	at com.mysql.jdbc.Util.getInstance(Util.java:386)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1066)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
      	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
      	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
      	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2815)
      	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
      	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2458)
      	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2375)
      	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2359)
      	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.updateStatus(JdbcDao.java:724)
      	... 7 more

      Comment


      • #4
        Can you post a copy of your channel that is deadlocking (make sure you remove any passwords/PHI/etc)

        Also what version of MySQL are you running as your backend database?

        Comment


        • #5
          Hi wayneh and narupley,

          I am using MySQL 5.6.14 X86

          Ill be happy to post my channel, but let me ask one question first. Is there an easy way to queue and retry on deadlock for source TCP - MLLP listeners?

          I usually only get 1-2 deadlocks in a row, then it works; so if I can get an easy workaround by retrying a few times then that is fine with me.

          Thanks,
          Scott

          Comment


          • #6
            Please find channel attached.
            Attached Files
            Last edited by spaterson; 05-14-2014, 07:32 AM.

            Comment


            • #7
              Is your MySQL server running on Windows or Linux? I'm able to reproduce but only on Linux

              Comment


              • #8
                Hi wayneh,

                It's running on Windows - 03 R2 x86. Same server as Mirth.

                Even with Linux, do you have a solution? Or advice for me?

                Thanks,
                Scott
                Last edited by spaterson; 05-19-2014, 09:28 AM.

                Comment


                • #9
                  I wasn't able to reproduce the exact same deadlock you mentioned, but it seems to have something to do with the queues. In your case it looks like its deadlocking at the source queue. The first step you can try is disabling the source queue for that channel and see if it makes a difference.

                  I've created an ticket in our issue tracker so we can look into it
                  http://www.mirthcorp.com/community/i...wse/MIRTH-3280

                  Comment


                  • #10
                    Hi Wayneh,

                    The deadlocking is occurring on the source.

                    If I set the source queue to: Off (Respond after processing), and I have my response set to: Auto-generate (before processing). It never sends an ACK back and my HL7 sender eventually times out. I have tried changing the response to other options, but an ACK is never sent back unless source queue is set to: On.

                    Thanks,
                    Scott

                    Comment


                    • #11
                      Do you have control of the source system and if so, is it also a Mirth Connect channel? If it is you can increase the Response Timeout on the TCP Sender. That setting determines how long the sender will wait for a reponse. It's possible that the channel simply takes more time to process than how long its currently waiting for the response.

                      Comment


                      • #12
                        OK a lot of progress has been made on this in the last day. First of all, you can read about why things are deadlocking here and why its really only a MySQL issue. We may not be able to prevent the deadlocks from happening altogether, but we will be dealing with them more gracefully in 3.1.0.

                        The deadlock itself is pretty harmless, but in 3.0.0, 3.0.1, and 3.0.2 the message that it happened on basically gets skipped until the channel is restarted. In 3.0.3 (which will be released very soon) this will longer happen. The current message will simply be retried and should complete successfully in this case so you won't need to intervene manually by restarting the channel. So you will still see the deadlocks in the error log but they will automatically be recovered. However, note that when this happens, the preprocessor and source transformer will execute again when that message is retried.

                        Furthermore, as a temporary workaround in 3.0.2, you can try adding the following line to a source transformer Javascript step on channels that are deadlocking with the source queue on

                        java.lang.Thread.sleep(20);

                        This will probably alleviate at least 90% if not more of the deadlocks you are seeing. The reason is because the deadlocks occur because certain queries are happening within too close a proximity of each other. By modifying the time where one of the queries occurs, the deadlocks become much more unlikely. Obviously this slows the throughput of the channel a bit, but for now its probably better than having to restart your channel all the time. If it doesn't work, try increasing the sleep time and see if it helps.

                        Comment


                        • #13
                          Hi wayneh,

                          I do not have control over the source system.

                          So you think the error would not happen if I use MS SQL? I am going to try it now with SQL 08, and ill update this thread with the outcome.

                          Thanks,
                          Scott
                          Last edited by spaterson; 05-20-2014, 08:29 AM.

                          Comment


                          • #14
                            Update. So Mirth is now running on MS SQL 08 on my server. I have done some testing (30 or so messages through) and I have not had the error once yet! Will do more testing tomorrow and hopefully won't have any errors.

                            Thanks,
                            Scott

                            Comment


                            • #15
                              Just to give some closure, this bug will finally be fixed in the next upcoming release (3.2.1). See MIRTH-3280 for more information.

                              Comment

                              Working...
                              X