Announcement

Collapse
No announcement yet.

Mirth Connect 3.0: Deadlocks

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Mirth Connect 3.0: Deadlocks

    We have numerous inbound channels that are MLLP listeners sending to multiple channel-reader type destinations. Everything is queued and none of it is synchronized.

    Frequently, this configuration produces a deadlock while updating the message statistics for the channel and destinations:

    Code:
    [2013-11-03 18:47:36,366]  ERROR (com.mirth.connect.connectors.vm.VmDispatcher:510): Error processing queued message 215-3 (SENT) for channel 8c89cb69-b26d-450e-a9fc-d09207d9b6eb (Destination 3). This error is expected if the message was manually removed from the queue.
    com.mirth.connect.donkey.server.data.DonkeyDaoException: java.sql.SQLException: Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.addChannelStatistics(JdbcDao.java:369)
    	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.commit(JdbcDao.java:1581)
    	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.executeTasks(BufferedDao.java:131)
    	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:74)
    	at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:61)
    	at com.mirth.connect.donkey.server.channel.DestinationConnector.run(DestinationConnector.java:498)
    	at java.lang.Thread.run(Unknown Source)Caused by: java.sql.SQLException: Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    	at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
    	at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2894)
    	at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2334)
    	at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:643)
    	at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:614)
    	at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:573)
    	at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:707)
    	at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.addChannelStatistics(JdbcDao.java:346)
    	... 6 more
    Turning the synchronization (wait for previous channel...) on makes this issue go away, but makes our processing sequential -- which we would like to avoid.

    I've attached the offending configuration, and I can get it to reliably trigger using a set of about 500 messages pumped at it.

    I've started to look at the way the code for the channels queues up the events for processing based on the Deadlock Graph included in the ZIP file -- but that is as far as I have gotten thus far.

    Just wanted to see if anyone else had seen this and if anybody was out ahead of me on the analysis.

    Oh -- the messages do get delivered, it is just the stat updates that are failing.

    Environment:
    • Windows Server 2008 R2
    • Mirth Connect Server 3.0.0.6931
    • Built on September 30, 2013
    • Server ID: 2f1516e3-8ed6-443e-914f-c670e8fde5cf
    • Java version: 1.7.0_45 (64-bit)
    • SQL Server 2008 R2 & SQL Server 2012


    Cordially,

    Jonathan Lent
    Attached Files

  • #2
    Thank you for finding this, I've opened an issue for it: MIRTH-3042
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      So what is the resolution to the Deadlock in SQL?

      Comment


      • #4
        Originally posted by csummer View Post
        The issue has been fixed, so the resolution is to upgrade your Mirth Connect instance to the latest.
        Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

        Nicholas Rupley
        Work: 949-237-6069
        Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


        - How do I foo?
        - You just bar.

        Comment

        Working...
        X