Announcement

Collapse
No announcement yet.

SFTP Polling

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • SFTP Polling

    I have an issue with 3 of my channels that are constantly polling an SFTP server for data, one channel is polling every 2 seconds, another channel every 10 seconds, and another every 5 minutes.

    For some weird reason, and I am not sure why, the channels just randomly stop polling and data is not getting picked up for hours and hours. At this point I have to halt the channel and then start it back up. Why would this happen?

    This is in production, and hence it is killing me. It has already happened 3 times in the last 30 days, and the customer is not happy at this point.

  • #2
    Originally posted by rpatel View Post
    I have an issue with 3 of my channels that are constantly polling an SFTP server for data, one channel is polling every 2 seconds, another channel every 10 seconds, and another every 5 minutes.

    For some weird reason, and I am not sure why, the channels just randomly stop polling and data is not getting picked up for hours and hours. At this point I have to halt the channel and then start it back up. Why would this happen?

    This is in production, and hence it is killing me. It has already happened 3 times in the last 30 days, and the customer is not happy at this point.
    What version are you using? Anything in the server log when that happens?
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      Mirth Connect Server 3.0.1.7051

      This is the last error that I see before it stopped:

      [2014-08-20 08:44:11,658] ERROR (com.mirth.connect.connectors.file.FileReceiver:19 8): Error polling in channel: 3f9204c3-8b74-475a-85bf-77ba5eb59fd7
      4:
      at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:17 21)
      at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:15 27)
      at com.mirth.connect.connectors.file.filesystems.Sftp Connection.listFiles(SftpConnection.java:140)
      at com.mirth.connect.connectors.file.FileReceiver.lis tFiles(FileReceiver.java:546)
      at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:194)
      at com.mirth.connect.donkey.server.channel.PollConnec tor$PollConnectorTask.run(PollConnector.java:141)
      at java.util.TimerThread.mainLoop(Unknown Source)
      at java.util.TimerThread.run(Unknown Source)Caused by: java.io.IOException: inputstream is closed
      at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java: 2871)
      at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.jav a:2895)
      at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:16 17)
      ... 7 more

      Comment


      • #4
        Any updates on this?

        Comment


        • #5
          We experience the same problem. We would like to know een solution, so it retries.

          Comment


          • #6
            Stop pulling from sftp

            Do we have a solution for this issue? I have the same problem on version 3.4.2 with a File Reader connector using sftp, it pass the "Test Read" (also accessible from a third party tool), it works fine for a few days, but randomly stops working until I re-deploy the channel. Not good on a production environment.
            The error is on the dashboard instead of the channel:
            ERROR (com.mirth.connect.connectors.file.FileReceiver:20 7): Error polling in channel: c1bad0ad-90c4-4194-aff3-aa710cc7a626
            4:
            at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:17 20)
            at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:15 26)
            at com.mirth.connect.connectors.file.filesystems.Sftp Connection.listFiles(SftpConnection.java:185)
            at com.mirth.connect.connectors.file.FileReceiver.lis tFiles(FileReceiver.java:555)
            at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
            at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
            at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
            at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)Caused by: java.io.IOException: inputstream is closed
            at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java: 2884)
            at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.jav a:2908)
            at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSft p.java:2446)
            at com.jcraft.jsch.ChannelSftp._sendCLOSE(ChannelSftp .java:2465)
            at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:16 97)
            ... 7 more

            The configuration is:
            Source Queue: OFF
            Response: None
            Process batch: No
            Max Processing Threads: 1
            Timeout (ms): 10000
            After Processing Action: Delete
            Error Reading action: None
            Error in response Action: After Processing Action
            Check File Age: Yes
            File Age (ms):1000
            File size (bytes): 0 - Ignore Maximum checked
            File Type: Text
            Encoding default

            The file to process is an hl7 with embedded PDF

            Comment


            • #7
              Do you face the same problem while using a third party tool?
              HL7v2.7 Certified Control Specialist!

              Comment


              • #8
                No, I used the third party tool just to verify connectivity. As I mention, the issue is happening just sometimes. Is actually working fine today, but it may occur again in a few days. It happened 4 times on the last month.

                Comment


                • #9
                  This is an ongoing issue for me too. I have opened tickets with support, but have been unable to get a resolution.

                  We have SFTP connections to different end points. They poll once per hour after different times. Occasionally we noticed that a channel stops picking up files from the SFTP endpoint. The channel appears to be functioning without issue. If I use a tool like Filezilla, I can connect to the endpoint and download files. If I restart the channel Mirth will pick up the files.

                  Perhaps its something in my configuration? What is the best practice for picking up files with Mirth from a SFTP end point?
                  Last edited by kirbykn2; 03-28-2018, 09:15 AM.
                  Best,

                  Kirby

                  Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

                  Appliance Version 3.11.4
                  Mirth Connect Version 3.8.0
                  Java Version 1.6.0_45-b06
                  Java (64 bit) Version 1.6.0_45-b06
                  Java 7 (64 bit) Version 1.7.0_151-b15
                  Java 8 (64 bit) Version 1.8.0_181-b13
                  PostgreSQL Version 9.6.8

                  Comment


                  • #10
                    Polling error

                    I also have this issue with two of my SFTP channels. I am using Mirth 3.3.0.7801. I would like to know if anyone has found a solution to the issue. Thanks!

                    Comment


                    • #11
                      I just set up some new channels last week using sftp file readers and had this happen over the weekend. Same error in the log file. I had to stop/halt/start the channel to kick start it again. The channel was stuck in a 'Polling' state.

                      I do see the Jsch errors more often than when the channel gets stuck polling, so that error is not freezing the channel every time. Every once in a while I'll see it come through with the error at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java :2208) or com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:35 9) instead of ls.

                      I'm connecting to a MOVEit server.

                      I noticed something using the OpenSSH client that occurs whether I connect to this server or my own OpenSSH server. If I do an ls with a wildcard, and there are no files matching the wildcard, I get a not found error rather than just returning nothing as if I had done a plain ls in an empty directory. Not sure if this is related to the errors in the server log or not.

                      I'm on 3.4.1.

                      Comment


                      • #12
                        I had to bounce my channels again. They had been stuck Polling since yesterday morning. I did a netstat this time and I could see them still connected, though they weren't doing anything productive.

                        This is also somewhat strange because when I connect manually, the site has a pretty short timeout before it boots me for inactivity. Is the channel really attempting to poll continuously for all that time?

                        Comment


                        • #13
                          This is definitely a tricky issue, I wish I could reproduce it locally so I could debug through and see exactly what's happening in JSch.

                          Do you happen to have any threaddumps (jstack) taken while the issue is happening? Not sure it'll help, but you never know.
                          Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

                          Nicholas Rupley
                          Work: 949-237-6069
                          Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


                          - How do I foo?
                          - You just bar.

                          Comment


                          • #14
                            A channel stopped processing message on 9/13. It appeared in the dashboard that the channel was stuck on "POLLING". I've included the error from the logs.

                            ERROR 2017-09-13 06:43:01,398 [File Reader Polling Thread on From_MY_Channel (80c21578-767d-4b29-abd3-6a33131e5358) < 80c21578-767d-4b29-abd3-6a33131e5358_Worker-1] com.mirth.connect.connectors.file.FileReceiver: Unable to dispatch message to channel 80c21578-767d-4b29-abd3-6a33131e5358: 4:
                            at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1 540)
                            at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1 290)
                            at com.mirth.connect.connectors.file.filesystems.Sftp Connection.readFile(SftpConnection.java:261)
                            at com.mirth.connect.connectors.file.FileReceiver.get BytesFromFile(FileReceiver.java:502)
                            at com.mirth.connect.connectors.file.FileReceiver.pro cessFile(FileReceiver.java:347)
                            at com.mirth.connect.connectors.file.FileReceiver.pro cessFiles(FileReceiver.java:247)
                            at com.mirth.connect.connectors.file.FileReceiver.pol l(FileReceiver.java:203)
                            at com.mirth.connect.donkey.server.channel.PollConnec torJob.execute(PollConnectorJob.java:49)
                            at org.quartz.core.JobRunShell.run(JobRunShell.java:2 13)
                            at org.quartz.simpl.SimpleThreadPool$WorkerThread.run (SimpleThreadPool.java:557)
                            Caused by: java.io.IOException: inputstream is closed
                            at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java: 2911)
                            at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.jav a:2935)
                            at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1 326)
                            ... 9 more
                            Best,

                            Kirby

                            Mirth Certified|Epic Bridges Certified|Cloverleaf Level 2 Certified

                            Appliance Version 3.11.4
                            Mirth Connect Version 3.8.0
                            Java Version 1.6.0_45-b06
                            Java (64 bit) Version 1.6.0_45-b06
                            Java 7 (64 bit) Version 1.7.0_151-b15
                            Java 8 (64 bit) Version 1.8.0_181-b13
                            PostgreSQL Version 9.6.8

                            Comment


                            • #15
                              Here's what I see in a thread dump for one of the channels while two of them were stuck polling. I'm assuming that the QuartzSchedulerThread is related since both stuck channels were adjacent to one.

                              I did not see a jsch session thread running for either of them.

                              Code:
                              "8f547f2f-c3b9-46b7-99e4-8ed5e594f201_QuartzSchedulerThread" #18910 prio=5 os_prio=0 tid=0x00000000568a0000 nid=0x15a0 in Object.wait() [0x000000005f2bf000]
                                 java.lang.Thread.State: TIMED_WAITING (on object monitor)
                              	at java.lang.Object.wait(Native Method)
                              	at org.quartz.simpl.SimpleThreadPool.blockForAvailableThreads(SimpleThreadPool.java:438)
                              	- locked <0x00000000f3470b18> (a java.lang.Object)
                              	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:255)
                              
                              "File Reader Polling Thread on XXXXXXXXXXXXXXXXXXX (8f547f2f-c3b9-46b7-99e4-8ed5e594f201) < 8f547f2f-c3b9-46b7-99e4-8ed5e594f201_Worker-1" #18909 prio=5 os_prio=0 tid=0x00000000568a0800 nid=0x474 waiting on condition [0x000000005d58e000]
                                 java.lang.Thread.State: WAITING (parking)
                              	at sun.misc.Unsafe.park(Native Method)
                              	- parking to wait for  <0x00000000f3478968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                              	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
                              	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
                              	at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
                              	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
                              	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                              	at com.mirth.connect.connectors.file.FileConnector.getConnection(FileConnector.java:174)
                              	at com.mirth.connect.connectors.file.FileReceiver.listFiles(FileReceiver.java:552)
                              	at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:203)
                              	at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
                              	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
                              	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
                              	- locked <0x00000000f3470b60> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                              
                              "e15d4b58-4fe2-4f5d-b9ec-ef47bfa7bba8_QuartzSchedulerThread" #18872 prio=5 os_prio=0 tid=0x00000000580fe000 nid=0x13b0 in Object.wait() [0x000000005ebdf000]
                                 java.lang.Thread.State: TIMED_WAITING (on object monitor)
                              	at java.lang.Object.wait(Native Method)
                              	at org.quartz.simpl.SimpleThreadPool.blockForAvailableThreads(SimpleThreadPool.java:438)
                              	- locked <0x00000000f3467bf8> (a java.lang.Object)
                              	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:255)
                              
                              "File Reader Polling Thread on XXXXXXXXXXXXXXXXX2 (e15d4b58-4fe2-4f5d-b9ec-ef47bfa7bba8) < e15d4b58-4fe2-4f5d-b9ec-ef47bfa7bba8_Worker-1" #18871 prio=5 os_prio=0 tid=0x00000000580fb000 nid=0xb18 waiting on condition [0x000000005de8e000]
                                 java.lang.Thread.State: WAITING (parking)
                              	at sun.misc.Unsafe.park(Native Method)
                              	- parking to wait for  <0x00000000f346b0e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                              	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
                              	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
                              	at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
                              	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
                              	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                              	at com.mirth.connect.connectors.file.FileConnector.getConnection(FileConnector.java:174)
                              	at com.mirth.connect.connectors.file.FileReceiver.listFiles(FileReceiver.java:552)
                              	at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:203)
                              	at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
                              	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
                              	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
                              	- locked <0x00000000f346fca0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                              Last edited by agermano; 02-28-2018, 06:55 AM. Reason: added threads for second stuck channel

                              Comment

                              Working...
                              X