Announcement

Collapse
No announcement yet.

How rollback Mirth connect cluster

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • How rollback Mirth connect cluster

    I have two appliance that was configured in Cluster (service cluster), after upgrading to connect 3.3.0 I decided to migrated to Mirth Connect Cluster, after doing that I started to have all kind of problems... cluster crashing, connection lost... messages lost, etc...

    I'm trying to rollback the configuration and use the old cluster configuration, I already removed the Database replication and auto-failover, and made change on the connect configuration to point to the local database, but when I try to start the cluster service it keep showing "The service failed to start. Clustering cannot be started while Mirth Connect Clustering is installed" How do I remove the "Mirth connect cluster"???

    Please help ....

  • #2
    Do you have snapshots prior to the upgrade? You could save your channels off and reload your snapshots and then reupload the current state of your channels. This is what we did while having trouble installing the advanced clustering.

    Comment


    • #3
      Nope... we have a policy to delete any snapshot after two days... the cluster worked fine for a little time... it crashed after 56 hours running...

      At this point I opened a ticket with Mirth, they are trying to figure out why it is crashing...

      For those that adventure to try this new cluster be aware that it won't use the cluster VIP to connect to your destination, the new cluster configuration use each server IP when connecting, so if you destination is trusting only one IP as source you will have a big problem here... recommendation wait for better version before move on this.

      Comment


      • #4
        Originally posted by cguanes View Post
        Nope... we have a policy to delete any snapshot after two days... the cluster worked fine for a little time... it crashed after 56 hours running...

        At this point I opened a ticket with Mirth, they are trying to figure out why it is crashing...

        For those that adventure to try this new cluster be aware that it won't use the cluster VIP to connect to your destination, the new cluster configuration use each server IP when connecting, so if you destination is trusting only one IP as source you will have a big problem here... recommendation wait for better version before move on this.
        From my experience, I do not think this add-on is production ready either, at least for our current setup. We have had multiple problems trying to configure it to our environment. We also had troubles with this VIP.

        Comment


        • #5
          They released as production release...
          Anyway, here the process that worked for to roll-back

          This worked for me, it may not work you... take it as it, and when you have not other option...

          1) uninstall the "Basic Clustering" plugin, on Mirth Connect, under Extensions on each one of the servers
          2) Stop the Mirth Connect on the Mirth Dashboard web, applications, STOP
          3) stop the failover (remove) (in all servers)
          4) stop/remove the replication database (in all servers)
          5) on the dashboard, application, connect manager, mirth connect setting, change back the database URL to point to original server
          6) restart the servers...
          7) check if the Mirth connect started (On my didn't start the first time)
          8) start the mirth connect on all servers
          9) start the cluster services

          I got my old cluster configuration running...

          I'm still seeing an error on booting time at the console about mirth connect cluster but I didn't care anymore as longer I have it running.

          Comment


          • #6
            how did you determine this?
            "the new cluster configuration use each server IP when connecting, so if you destination is trusting only one IP as source "

            Comment


            • #7
              because we do have another company receiving SIU hl7 from us and they do have an specific ip set to trust as source...

              They called us that nothing was coming to them, on the troubleshooting I asked to try another ip (primary server ip) and they started to receive all messages...

              Comment


              • #8
                Thank you.

                Comment


                • #9
                  I would like to share this too because I cannot believe what I read....
                  this come from the mirth support when I asked for how long the old cluster will be there.

                  "In any case, I spoke to the support manager who updated me that the old clustering will NOT be deprecated. We have found that the newer clustering is not suitable for all clients. "

                  They made me to feel too special, really? :-(

                  My mirth connect cluster was crashing after losing the heartbeat and they don't know why..

                  I'm glad that I could do roll-back.

                  Comment


                  • #10
                    Issues with advanced clustering

                    Thanks for the post cguanes! We have also had multiple issues installing, configuring, and using the advanced clustering plug-in.

                    We have had an IP routing issue in out test environment since installing advanced clustering. The ticket has been opened for three weeks with no solution. I believe the VIP routing issue you are describing maybe related to what we are seeing.

                    After reading your post I verified that the outgoing VIP is no longer working as expected. Mirth is not presenting the Cluster VIP address as the source IP.

                    I've wasted numerous hours troubleshooting this issue with Mirth support.


                    This is basic functionality. It's unfortunate this was released as production ready.

                    It is our understanding that the old clustering is being deprecated and we need to move to the advanced clustering plug-in. If this is not the case, I would expect Mirth to notify there customers. We will need to redesign our environment strategy.
                    Best,

                    Kirby

                    Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

                    Appliance Version 3.11.4
                    Mirth Connect Version 3.8.0
                    Java Version 1.6.0_45-b06
                    Java (64 bit) Version 1.6.0_45-b06
                    Java 7 (64 bit) Version 1.7.0_151-b15
                    Java 8 (64 bit) Version 1.8.0_181-b13
                    PostgreSQL Version 9.6.8

                    Comment


                    • #11
                      How rollback Mirth connect cluster - Crashing

                      You are welcome, I hope people read this thread before doing the same mistake that I did.

                      I have a PICO for development so I couldn't try the cluster first (I just test channels code compatibility), and also the cluster ran for two days so I didn't care to move my snapshot to a safe place...

                      And don't trust that mirth support will get you ASAP... they don't care much about your emergency or your patients

                      Comment


                      • #12
                        Advanced Clustering update

                        Hello Carlos,

                        Do you know if Mirth has resolved the issues with advanced clustering yet?
                        Best,

                        Kirby

                        Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

                        Appliance Version 3.11.4
                        Mirth Connect Version 3.8.0
                        Java Version 1.6.0_45-b06
                        Java (64 bit) Version 1.6.0_45-b06
                        Java 7 (64 bit) Version 1.7.0_151-b15
                        Java 8 (64 bit) Version 1.8.0_181-b13
                        PostgreSQL Version 9.6.8

                        Comment


                        • #13
                          Originally posted by kirbykn2 View Post
                          Hello Carlos,

                          Do you know if Mirth has resolved the issues with advanced clustering yet?
                          I don't know, I didn't try it anymore... Let see if they answer here..

                          Comment


                          • #14
                            First of all we appreciate your patience as we work on ironing out the issues with database replication + clustering.

                            Here is a status update based on the main issues I was able to gather from reading this thread:

                            1) Outbound traffic not coming from the cluster VIP.
                            2) Cluster crashing unexpectedly.
                            3) Problems configuring.

                            Comments on each:

                            1) Outbound traffic not coming from the cluster VIP.

                            We are aware that this is a showstopper for those wanting upgrade to the new db replication-based clustering. It is on the Mirth Appliance roadmap to bring back this feature from the old clustering service.

                            2) Cluster crashing unexpectedly.

                            To be able to narrow this down to a specific cause, we need more information about your specific environment, what the servers were doing at the time of the crash and what was shown in the mirth connect server logs at the time (preferably with debug logging turned on).

                            That being said, I suspect that an unexpected shutdown of the whole cluster (due to all servers heartbeats' timing out) could occur if the shared database is inaccessible for longer than the heartbeat timeout period (default 5 mins). The auto-shutdown on heartbeat timeout is intended to protect against concurrent access to shared resources, which is a new danger presented by the shared / replicated database architecture. However if the database itself stalls, the whole cluster shouldn't have to halt. This is also on our roadmap to address in an upcoming version of the Mirth Connect clustering plugin and is actively being worked on.

                            3) Problems configuring.

                            After the initial release of database replication + clustering, many issues related to installation/setup on Mirth Appliances were brought to our attention. For those issues that we have documented, some have already been fixed and rolled out via Mirth Appliance updates and the remainder will be fixed in subsequent updates.

                            If there are other issues or concerns, please let us know as we'd like to make sure all the problems in the initial rollout have been documented and added to our development roadmaps. And please remember that providing as much detail as possible is critical to help us determine the root cause of the issue! Thanks.

                            Comment


                            • #15
                              Date

                              Thanks for the detailed explanation.

                              Do you have any idea when this fixes will be in place?
                              Best,

                              Kirby

                              Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

                              Appliance Version 3.11.4
                              Mirth Connect Version 3.8.0
                              Java Version 1.6.0_45-b06
                              Java (64 bit) Version 1.6.0_45-b06
                              Java 7 (64 bit) Version 1.7.0_151-b15
                              Java 8 (64 bit) Version 1.8.0_181-b13
                              PostgreSQL Version 9.6.8

                              Comment

                              Working...
                              X