CLOSE_WAIT leak

[StickyBandit]

We run Mirth connect on linux and when mirth is running in our TEST environment the CLOSE_WAIT number keeps climbing and climbing everyday. I know that mirth connect is the culprit because when I shut it down this number goes back to 0 zero.
Seen by this linux command...
$/sbin/lsof | grep '>itx' | grep CLOSE_WAIT | wc -l
239370


I have added the ability to use the new Oracle DB connection pool with...
:POOLED at the end of the URL and the new oracle driver...
oracle.jdbc.pool.OracleDataSource from the ojdbc8.jar file replacing the ojdbc7 that gets installed with mirth connect. I also added multi-threading to several channels.

I tried to address any and all changes that were needed for these new features, but I discovered this CLOSE_WAIT leak still exist.

Any help or ideas on how to find what is causing this issue would be greatly appreciated?


Mirth Connect Server 3.6.1
Built on July 13, 2018
Java version: 1.8.0_181

[StickyBandit]

Anybody read this?

[agermano]

Are you possibly forgetting to close your database connections?

[MrGesan]

I have added the ability to use the new Oracle DB connection pool with...
:POOLED at the end of the URL and the new oracle driver...
oracle.jdbc.pool.OracleDataSource from the ojdbc8.jar file replacing the ojdbc7 that gets installed with mirth connect. I also added mult-threading to several channels.

It would be interesting to know how your channels work. Do they open sockets only with the oracle server or use other connectors o javascript codes that open sockets?



As you know the CLOSE_WAIT status means that the other side has initiated a connection closure, but the application on the local side has not yet closed the socket. Maybe the mirth connector or javascript code (if you open sockets by code) has released the connection handler before completing the closure

[StickyBandit]

I thought it was also that some connections were not getting closed.
But after verifying that no channels were even getting any messages and yet
the CLOSE_WAIT number was still climbing by the hundreds, I am beginning to think that maybe Mirth Connect is somehow not compatible with ojdbc8.jar.

Not sure what the heck is going on...

I have even stopped channels, trying to pin down which channels are causing it, it didn't seem to matter or help.


I may have to rollback to ojdbc7.jar soon.

[StickyBandit]

Does anyone at Mirth think the new version Mirth Connect Server 3.8.0
would fix this issue?

[StickyBandit]

What I mean is that all the channels are idle and the CLOSED_WAIT count still goes up by another 100 every few minutes.

[agermano]

Where are you finding your information about how to use the driver in that manner?

It looks like the options to get pooled connections from that class have been deprecated since at least 2012 and using a real connection pool manager is advised. Oracle provides UCP for that purpose. Mirth ships with HikariCP and uses it internally.

https://stackoverflow.com/questions/...ion-pool-class

[StickyBandit]

https://docs.oracle.com/cd/B28359_01...htm#ADMIN12351

OLD
oracle.jdbc.driver.OracleDriver

NEW POOLED
oracle.jdbc.pool.OracleDataSource

[agermano]

This pool is on the server side. You'll potentially reduce server resources by using it (Edit: if you have many clients/channels all connecting to the same server pool,) but it looks like the middle-tier client (mirth) still has to set up and tear down connections with the connection broker. It says this complements middle-tier connection pools that share connections between threads in a middle-tier process.

https://docs.oracle.com/database/121....htm#CNCPT1896


In this guide, they still use Universal Connection Pool (UCP) on the client side.

https://docs.oracle.com/database/121...htm#JJDBC29029