Announcement

Collapse
No announcement yet.

Clearing out old socket connections

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Clearing out old socket connections

    Scenario:
    A live connection exists between Mirth (using TCP Listener) and the Client. The Client, (for whatever reason), decides to close the connection and open a new connection to send messages.

    Perception:
    We've set the [Max Connection] setting = 1 because we do not want the Client sending messages to the channel thru more than one connection simultaneously. Now, because of this setting, it appears that Mirth will not see the new connection established by the client unless the Mirth channel is redeployed. So anytime the Client closes the connection and opens a new connection, it appears that Mirth is still looking at the old connection that was originally established with the Client.

    Question:
    If the Max Connection setting = 1, is there a way that Mirth can detect if the Client has closed its existing connection - and then automatically re-connect
    to the new active Client connection without having to re-deploy the channel?


  • #2
    If the client closes its connection, then the server (TCP Listener) will detect that and shutdown the local socket as well, so that another client connection can connect. I'm not able to reproduce the scenario you described. Can you do a network capture to see exactly what's going on?
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      I have something similar but I'm not able to reproduce this. but I have channel with 4 connections connected (this is wat Mirth Shows) but there is no client connected at this moment.

      only a restart of the mirth Service will close the connections.
      Stefan

      Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

      Comment


      • #4
        Hi narupley,

        Here is my theory, I believe the issue is the Client not closing the connection when is done sending a message.

        Possible flow:
        Client sends a message and does not close the socket, Mirth waits for the timeout and then closes it. The client then tries to send a message using the same socket, but this time it is closed, so it opens a new connection and starts sending out messages to that new socket but it encounters an issue because mirth is configured to have a maximum of 1 connections ( we want to maintain the order of precedence) and in theory there's another connection already stablished with them.

        We would like to have some guidance on this matter because the only way we know the channel is not listening is when the client makes us aware of the issue. The way we momentarily fix the issue is by manually redeploying the affected channels.

        I am pretty sure this is related to the same issue but we get a lot of connection reset errors (Error Message = Connection reset Error Type = Source Connector)
        Last edited by DannyCortes; 05-12-2014, 12:15 PM.

        Comment


        • #5
          Yes, that theory is sound. It's not a problem with Mirth Connect, it's a problem with the connecting client. If you configure your server to have a maximum of X connected sockets and the client isn't closing sockets appropriately, then the maximum will be reached and there's nothing the server can do, because it has no idea whether the client is actually "done with" the socket or not.
          Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

          Nicholas Rupley
          Work: 949-237-6069
          Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


          - How do I foo?
          - You just bar.

          Comment


          • #6
            After some research and with the help of Wireshark, I was able to find the exact moment when Mirth stops ingesting messages but I don't know how to interpret this information and why this is happening.
            The last received message and normal communication that we had with the Client was at 5:23 AM then at 5:41 the client changed the sending port and Mirth stopped ingesting messages, then around 7:23 We got a connection reset email from Mirth and we started ingesting messages again.

            Port 7154 is the Mirth Channel Port (Receiver)
            Port 60843 is the Client Port (Sender)
            Port 52425 is another Client Port (Sender)

            I couldn't upload the screen shot but here is the chain of events.
            Can you please point out what's going on here?

            5:23 AM 60843 > 7154 [PSH, ACK] Seq=14746495 Ack=107641 Win=130048 Len=173
            5:23 AM 7154 > 60843 [ACK] Seq=107641 Ack=14746668 Win=211712 Len=0
            5:23 AM 7154 > 60843 [PSH, ACK] Seq=107641 Ack=14746668 Win=211712 Len=78
            5:23 AM 60843 > 7154 [ACK] Seq=14746668 Ack=107719 Win=130048 Len=0
            5:41 AM 52425 > 7154 [SYN] Seq=0 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1
            5:41 AM 7154 > 52425 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
            5:41 AM 52425 > 7154 [ACK] Seq=1 Ack=1 Win=131072 Len=0
            5:41 AM 7154 > 52425 [FIN, ACK] Seq=1 Ack=1 Win=131072 Len=0
            5:41 AM 52425 > 7154 [PSH, ACK] Seq=1 Ack=1 Win=131072 Len=200
            5:41 AM 7154 > 52425 [RST, ACK] Seq=2 Ack=201 Win=0 Len=0
            5:41 AM 52425 > 7154 [PSH, ACK] Seq=201 Ack=1 Win=131072 Len=200
            5:41 AM 7154 > 52425 [RST] Seq=1 Win=0 Len=0 |After this Mirth stopped ingesting messages and we started sending RST back to the client and not ACK’s but the client kept sending us messages.
            5:41 AM - 7:23 AM A whole bunch of [PSH,ACK] From the client and [RST] From Mirth, A couple of [SYN] From the Client and [SYN,ACK] From Mirth and Two Retransmitions
            7:23 AM [TCP Keep-Alive] 7154 > 60843 [ACK] Seq=107718 Ack=14746668 Win=211712 Len=1 | We had 10 of this back to back
            7:23 AM 7154 > 60843 [RST, ACK] Seq=107719 Ack=14746668 Win=0 Len=0 |After this We went back to normal and got flooded with messages.
            7:23 AM 53978 > 7154 [SYN] Seq=0 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1
            7:23 AM 7154 > 53978 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
            7:23 AM 53978 > 7154 [ACK] Seq=1 Ack=1 Win=131072 Len=0
            7:23 AM 53978 > 7154 [PSH, ACK] Seq=1 Ack=1 Win=131072 Len=200

            Comment


            • #7
              I forgot to mention that we changed the timeout from 10 seconds to 0 (infinite) and we kept the maximum connections set to 1 and keep connection open set to yes

              Comment


              • #8
                I wonder if the client is changing the connection but not closing the original connection, so Mirth is sending RST over and over:
                7:23 AM 7154 > 60843 [RST, ACK] Seq=107719 Ack=14746668 Win=0 Len=0
                to tell the Client to close the original connection - while they are sending messages over the new connection?

                Comment


                • #9
                  What about this
                  7:23 AM [TCP Keep-Alive] 7154 > 60843 [ACK] Seq=107718 Ack=14746668 Win=211712 Len=1

                  We had 10 of those back to back and they are directed to the port they stopped using at 5:23 AM. I am wondering why this is being sent two hours later.

                  Comment


                  • #10
                    The network capture you posted confirms my suspicions that the client is simply not closing its socket like it should be. Therefore since you have a maximum of 1 connection set on the TCP Listener, what you're seeing is the expected behavior.

                    Client 1 (60843) sends a message and gets an ACK, but never sends a FIN packet. When Client 2 (52425) connects, the server sees that the maximum number of connected sockets (1) has already been reached, so the server-side socket gets closed (you see the FIN,ACK sent back to Client 2). Client 2 ignores the FIN packet and tries to send data, and the server obviously returns a RST because that connection is no longer valid.

                    Let me reiterate that this is what should happen when you have "Max Connections: 1" set. The problem lies not with Mirth Connect but instead with the connecting client that is not closing its socket when it is done sending a message. The Keep-Alive ACKs you get further prove that the network layer on the remote client side still thinks that the connection is valid. You need to either increase the maximum number of connections on the TCP Listener, or have the remote side fix its application to close connections correctly.
                    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

                    Nicholas Rupley
                    Work: 949-237-6069
                    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


                    - How do I foo?
                    - You just bar.

                    Comment


                    • #11
                      I believe the issue occurs when there are networking issues between mirth and the client. I was able to reproduce using 2 separate virtual machines; one with mirth, the second with a test client. Both VMs were hosted on the same machine.

                      Mirth Connect Server 3.0.2.7140
                      Java version: 1.7.0

                      1. Create and deploy a new Mirth channel, set the source to be a TCP Listener on port 7000. All other options are default. (Channel export attached)
                      2. Establish a connection from the client VM
                      3. On the client VM, disconnect the network adapter
                      4. From the client VM attempt to send an HL7 message. This message will error out obviously - as the VM no longer has a network connection.
                      5. Reconnect the network on the client VM
                      6. Call close and cleanup on the client socket
                      7. Mirth still sees the connection

                      Note: After step #4 is completed, nothing done on the client side seems to have any effect on Mirth. Mirth does cleanup the connection - but not until 2 hours have elapsed.
                      Attached Files

                      Comment


                      • #12
                        That is the expected behaviour, because there's no way for the server to know whether the socket is actually still valid or not. Just to reiterate:

                        Originally posted by narupley View Post
                        The network capture you posted confirms my suspicions that the client is simply not closing its socket like it should be. Therefore since you have a maximum of 1 connection set on the TCP Listener, what you're seeing is the expected behavior.

                        Client 1 (60843) sends a message and gets an ACK, but never sends a FIN packet. When Client 2 (52425) connects, the server sees that the maximum number of connected sockets (1) has already been reached, so the server-side socket gets closed (you see the FIN,ACK sent back to Client 2). Client 2 ignores the FIN packet and tries to send data, and the server obviously returns a RST because that connection is no longer valid.

                        Let me reiterate that this is what should happen when you have "Max Connections: 1" set. The problem lies not with Mirth Connect but instead with the connecting client that is not closing its socket when it is done sending a message. The Keep-Alive ACKs you get further prove that the network layer on the remote client side still thinks that the connection is valid. You need to either increase the maximum number of connections on the TCP Listener, or have the remote side fix its application to close connections correctly.
                        Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

                        Nicholas Rupley
                        Work: 949-237-6069
                        Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


                        - How do I foo?
                        - You just bar.

                        Comment


                        • #13
                          Because our client sends messages over different port/Sockets at random without closing them, my settings are:
                          Max Connections: 30
                          Receive Timeout: 5000 ms
                          Keep Connection Open: Yes

                          So if the client only has 1 connection open - it should stay open indefinitely because [Keep Connection Open] = Yes; however, if the client opens a second connection without closing the first connection, and then stops sending messages over the second connection for 5 seconds, which connection will Mirth close -the first or second connection? The reason I'm asking is because I don't see either connection closing.

                          Thanks :-)
                          Last edited by marke; 06-03-2014, 04:54 AM.

                          Comment


                          • #14
                            Originally posted by marke View Post
                            Because our client sends messages over different port/Sockets at random without closing them, my settings are:
                            Max Connections: 30
                            Receive Timeout: 5000 ms
                            Keep Connection Open: Yes

                            So if the client only has 1 connection open - it should stay open indefinitely because [Keep Connection Open] = Yes; however, if the client opens a second connection without closing the first connection, and then stops sending messages over the second connection for 5 seconds, which connection will Mirth close -the first or second connection? The reason I'm asking is because I don't see either connection closing.

                            Thanks :-)
                            That is expected, because you have Keep Connection Open enabled. When that is enabled, the only time the connection is actually closed is when the client closes the connection, or if an error occurs. The receive timeout in that case indicates how long to read from the socket on any given attempt. That is useful when you're receiving via Basic TCP without any start or end frame characters, because there the receiver has no way of knowing when one message ends and the next begins. The receive timeout can act as a delimiter in that case.

                            In your case, if you want connections to close after 5 seconds of idle time, then set Keep Connection Open to No.
                            Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

                            Nicholas Rupley
                            Work: 949-237-6069
                            Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


                            - How do I foo?
                            - You just bar.

                            Comment


                            • #15
                              what happens if you set the Max Connection = 0? Could that mean provide an infinite number of connections?

                              Comment

                              Working...
                              X