LLP Listerner Channel stops listening...

[[email protected]]

My LLP listerner channel seems to stop listening after a while and the sending end blocks waiting for an ACK that it never gets.

I seem to get messages for a while, but then the status changes on the Channel from Connected(1) to Waiting, and I don't get any more messages.

What is interesting, is that if we reset the sending system (their communications adapter), without touching the Mirth channel, then messages start to flow again without any problem. This is without having the Mirth channel or server restarted.

What I suspect is that the Mirth channel drops the connection after some long period of time, though why I don't know (maybe a network level timeout, something to do with the VPN connection, who knows?)....but the sender doesn't know this so it tries to send on the old connection and blocks. By reseting the sender it opens a new connection from scratch and then Mirth is happy again.

Here are some details on the setup:

Sender: GE TriWin Lab System with TOC connector

Mirth: Recent 1.7.0 SVN Build (from about a week ago).

LLP Channel:

Receive Timeout: 0
Use Strict LLP Validation: Yes
Send Ack: Enabled


It seems like a repeat of the old Keep Connection Open issue...at least the symptoms are similar.

Any ideas on what might be causing this?

Would changing the Receive Timeout to something like 60000 (1 minute) fix the issue?

Any other recommendations?

Post edited by: [email protected], at: 11/06/2007 08:01

[[email protected]]

Re:LLP Listerner Channel stops listening...

I'm also curious as to what the following setting actually do (both are turned off in my channel):

MSH-15 ACK Accept?

ACK on New Connection?

Muchos gracias!

[[email protected]]

Re:LLP Listerner Channel stops listening...

Since the system sending the LLP messages is a GE Centricity TriWin lab system, would this post be relevant?

http://www.mirthproject.org/index.ph...d=2686&catid=2

That is, would turning Strict LLP Validation off help?

Since I do receive a bunch of messages, and then stop getting them after a longish period of the lab system not sending anything, I don't think it's a Strict Validation issue....probably a timeout/keep connection open thing.

Thanks all!

[jacobb]

Re:LLP Listerner Channel stops listening...

Thanks for your testing before the release comes out, we really appreciate it!

It's an interesting issue, and I think you are right about the connection. However, this connection being dropped might be something that we don't have control over. We set the Java Socket SO_TIMEOUT to 0 if your receive timeout is 0. This value makes it so it should never disconnect. If the socket ends up dying for some reason and the sender doesn't know (although it should take of this), then I could see how this would be a problem. Does centricity attempt to establish a new connection if the connection is explicitly closed? If it does, then perhaps you could try setting the receive timeout to a short amount of time. This would cause it to explicitly close the connection every time there is a pause in messages. If centricity doesn't do this, is there any way to tell it to create a new connection for every message? At the very least, centricity should have a way of reestablishing connection if the socket connection dies.

Let me know your findings!

[jacobb]

Re:LLP Listerner Channel stops listening...

Also, the only situation that I can think of in which you would want to set "Use Strict LLP Validation" to "No" is if you receive a single message over multiple socket connections. This works if you turn of strict llp validation and turn on wait for end of message char.

[[email protected]]

Re:LLP Listerner Channel stops listening...

Here is what I saw in the Mirth logs this morning, when the status changed from Connected(1) to Waiting:

INFO 2007-11-07 11:59:19,324 [ca14e70b-d7f2-4c6c-87d7-65870a6f31ae_source_connector._mllpEndpoint#129290 5300.receiver.1] com.webreach.mirth.connectors.mllp.MllpMessageRece iver: TCP connection from /111.111.111.1:2810 on port 2100

INFO 2007-11-07 14:10:34,499 [ca14e70b-d7f2-4c6c-87d7-65870a6f31ae_source_connector._mllpEndpoint#129290 5300.receiver.4] com.webreach.mirth.connectors.mllp.protocols.LlpPr otocol: SocketException on read() attempt. Socket appears to have been closed: Connection timed out

Does that look like the lab end is closing the socket to you? Or maybe timing out at the lab end?

Thx!

Post edited by: [email protected], at: 11/07/2007 11:55

[jacobb]

Re:LLP Listerner Channel stops listening...

It definitely looks like the connection is made by the lab, and then something breaks that connection in the middle of it. It could be as simple as a network error, rather than Mirth or Centricity actually closing the connection.

Does this seem to happen at a specific interval (like every 2 hours), or is it completely random?

[[email protected]]

Re:LLP Listerner Channel stops listening...

It doesn't seem to be following much of a pattern. Since I turned logging on, I got a connection at around 1:30pm yesteday which timed out at about 12:30am this morning, around 11 hours later. Then it connected at noon and dropped at 2:10, after a bit more than 2 hours.

Doesn't seem consistent even with just two data points.

I wonder if there is a timeout in the firewall or the VPN connection that is used to connect the lab and Mirth? Have to look into that.

[jacobb]

Re:LLP Listerner Channel stops listening...

Have you noticed anything special about the last message to be processed? Does the last message sent by Centricity end up being processed by Mirth, or does the connection die in the middle of it? Any Mirth errors? A few questions like this might help figure out what exactly is going on.

[chrisl]

Re:LLP Listerner Channel stops listening...

Are you using the MIK with Centricity?

[[email protected]]

Re:LLP Listerner Channel stops listening...

The last message seems to come through fine. The connection gets dropped many hours later. I suspect it's a network error or timeout, given all the VPN crap in between the two systems.

No mirth errors at all!

[[email protected]]

Re:LLP Listerner Channel stops listening...

We're using the TOC connector with an older TriWin system.

Don't know what MIK is.

[cabonis]

Re:LLP Listerner Channel stops listening...

Was there ever any resolution to this issue? I am having the same issue with communication between GE Centricity and Mirth. The connection will be lost for whatever reason and GE thinks the connection never ended. GE is in an 'UP' state but isn't sending and Mirth is waiting. This also happens if we restart Mirth for any reason. Help!

[[email protected]]

Re:LLP Listerner Channel stops listening...

'Fraid that the only fix we've found is to restart the GE TOC connector. We're putting in place an automated procedure that will restart it at regular intervals, to coincide with when our VPN connection resets to re-negotiate keys (about every 8 hours for us).

I'ld love to see the source code for GE's communications stuff. But then I think I know what I would find, and I severely doubt it would be very pretty. ;-)

[cabonis]

Re:LLP Listerner Channel stops listening...

We're working with our GE implementation engineer to see if there is anything that can be done to resolve the issue on their end. I'll let you know if we find anything.