Announcement

Collapse
No announcement yet.

What happens when Mirth Connect loses the connection with its internal database?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • What happens when Mirth Connect loses the connection with its internal database?

    Dear people,
    about one and a half month ago one of our clients finally listened to us and replaced their embedded derby database with a MS SQL (2016 / 11.0.5058.0) database.
    That database is located on a different server in their serverpark, although we had indicated that perhaps a local version would be better for performance.
    (side question: is this true?).

    Since they started using this database they have recurring problems with Mirth Connect (3.2.1.750) 'hanging' i.e. doing nothing.
    There are no errors in the dashboard, just channels doing nothing.
    not reading, not writing, just nothing.
    The Mirth machine itself is not doing much (processor load is low).

    In the logging (mirth.log) we see this:
    ERROR 2017-05-14 20:14:38,983 [Timer-46] com.mirth.connect.donkey.server.channel.Destinatio nChain: Error processing destination Verplaats file als verwerking is gelukt for channel 0f99951c-3386-4749-92bd-622ce35cbe13.
    com.mirth.connect.donkey.server.data.DonkeyDaoExce ption: java.sql.SQLException: Invalid state, the Connection object is closed.
    at com.mirth.connect.donkey.server.data.jdbc.JdbcDao. close(JdbcDao.java:1983)
    at com.mirth.connect.donkey.server.data.buffered.Buff eredDao.executeTasks(BufferedDao.java:140)


    more details in the attached log excerpt (Mirth_log_excerpt.txt)

    And yes, because there are no alerts etc. to warn the admins it took them 90 minutes to react (and restart the mirth machine)!

    The strange situation is that restarting the Mirth machine solves the problem (for the time being).

    The destination first mentioned in the logging (Verplaats file als verwerking is gelukt) doesn't even use our user database, one of the reasons we have a feeling it's the internal Mirth database that's acting up.


    Is there ANY way to find out in logging or something is that's the case?
    In other words: how can we find out what the problem is here?

    Any help / hints are greatly appreciated.
    Thanks
    Jurjan
    Attached Files

  • #2
    Locating the database on the same server as Mirth Connect would certainly help performance while processing messages.

    You can enable more log output around database access by adding this to the end of the log4j.properties file under the /conf folder in the Mirth Connect installation:

    Code:
    log4j.logger.com.mirth.connect.donkey.server.data.jdbc = DEBUG
    This will produce a lot of output in your logs, so be aware of that.

    Also, updating to the latest version of Mirth Connect (3.5.0) could help. There was a performance issue that was addressed in version 3.4.0 that could be related to what you observed: http://www.mirthcorp.com/community/i...wse/MIRTH-3447

    Comment


    • #3
      OOOOPS,
      I'm sorry, I just noticed your answer.

      (seems the mail notifying me of your reply never got to me).

      I will forward your suggestions to my customer.

      Thanks

      Comment


      • #4
        We have had a new occurrence of this problem at our clients location.

        The logs are full of this:
        DEBUG 2017-09-26 23:19:17,227 [qtp173832146-32 - /channelstatus] org.eclipse.jetty.io.nio.ssl: [Session-1, SSL_NULL_WITH_NULL_NULL] [email protected] SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {[email protected],g=HttpGenerator{s=2, h=0,b=0,c=-1},p=HttpParser{s=0,l=12,c=44},r=108138} NOT_HANDSHAKING filled=0/0 flushed=0/0


        DEBUG 2017-09-26 23:59:18,831 [qtp173832146-34 - /channelstatus] org.eclipse.jetty.io.nio.ssl: [Session-1, SSL_NULL_WITH_NULL_NULL] [email protected] SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {[email protected],g=HttpGenerator{s=2, h=0,b=0,c=-1},p=HttpParser{s=0,l=12,c=44},r=108498} NOT_HANDSHAKING filled=0/0 flushed=1030/0
        DEBUG 2017-09-26 23:59:18,831 [qtp173832146-34 - /channelstatus] org.eclipse.jetty.io.nio.ssl: [Session-1, SSL_NULL_WITH_NULL_NULL] [email protected] SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {[email protected],g=HttpGenerator{s=2, h=0,b=0,c=-1},p=HttpParser{s=0,l=12,c=44},r=108498} NOT_HANDSHAKING filled=0/0 flushed=0/0
        DEBUG 2017-09-26 23:59:18,831 [qtp173832146-34 - /channelstatus] org.eclipse.jetty.io.nio.ssl: [Session-1, SSL_NULL_WITH_NULL_NULL] [email protected] SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {[email protected],g=HttpGenerator{s=2, h=0,b=1026,c=-1},p=HttpParser{s=0,l=12,c=44},r=108498} NOT_HANDSHAKING filled=0/0 flushed=0/0
        DEBUG 2017-09-26 23:59:18,831 [qtp173832146-34 - /channelstatus] org.eclipse.jetty.io.nio.ssl: [Session-1, SSL_NULL_WITH_NULL_NULL] wrap OK NOT_HANDSHAKING consumed=1026 produced=1055



        Any idea what the problem could be?
        And, perhaps more importantly: what would the solution (direction) be?

        Thanks!

        Comment


        • #5
          It happended again.
          some extra logging info:


          See attached log extract!


          Please help/advise
          Attached Files

          Comment


          • #6
            Mirth Connect maintains an pool of connections to its internal database. It seems that its failing to detect when connections to the SQL Server database are closed and new connections need to be established.

            Mirth Connect ships with two connection pool libraries, "HikariCP" and "DBCP". HikariCP is the default, but you can switch to DBCP by adding this to your mirth.properties file:

            Code:
            database.pool = DBCP
            I'd recommend giving that a try to see if it fixes this problem.

            Comment


            • #7
              Brent,
              thanks!

              I'll forward your suggestion to the client (and their application support).

              I'll let you know if this helps or not (hopefully the former!)

              Comment


              • #8
                com.mirth.connect.donkey.server.data.DonkeyDaoExce ption: java.sql.SQLException: Invalid state, the Connection object is closed.

                I am not sure how mirhtconnect maintains the connections in the connection pool. I faced the same problem in JBoss server when there is a glitch in the network connection / server between jboss server and SQL. the solution was to configure the connection pool (*-DS.xml) with more information such as validate the connection everytime fetched from the pool & flush the pool if there are not valid.

                hope this might give some idea.
                Mirth Certified

                Comment

                Working...
                X