Announcement

Collapse
No announcement yet.

Mirth increasing number of open threads on Windows

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Mirth increasing number of open threads on Windows

    Hi,
    I have the file reader and some javascript in Source Translator and destination translator. Also, I have some custom java code calls within the javascript. I am running the Mirth Server on Windows 2003 server.

    The following are the observations using the windows trask manager

    Before starting the Mirth Server the Thread Count : 493
    After Starting the Mirth Servwr Thread Count : 560
    After 40 minutes the Thread count is : 670
    After two hours the Thread count is : 1097

    We are not processing any messages only with that code we are running the Mirth Channels.

    It appears that some threads are not closed.

    Any one has any idea how to find the issue. I increased the thread count using the following parameter

    wrapper.java.additional.1=-Xss100k as I was getting OutOfMemorey issue after 4500 threads. However the above increase in the thread may be the issue.

    Can any one help to resolve the issue or How to narrow down the issue.

    Thank you for the help

    Thanks
    RAm

  • #2
    Re:Mirth increasing number of open threads on Windows

    If you remove the calls to your custom java code does the thread count keep increasing?

    What version is your JRE?

    Can you export your channel and post it here?

    I'm thinking that it would be best to get the custom code out of the picture and try to reproduce this with just Mirth.
    Jon Bartels

    Zen is hiring!!!!
    http://consultzen.com/careers/
    Talented healthcare IT professionals wanted. Engineers to sales to management.
    Good benefits, great working environment, genuinely interesting work.

    Comment


    • #3
      Re:Mirth increasing number of open threads on Windows

      I am using the following JRE
      Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
      Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)

      I am going to remove the code ( Java Native Code and run the process). Do you think the native code runs on a seperated thread? Do we need to do any additional thing to clean-up in the java code>

      Please let me know

      I will post the result soon after my testing.

      Thanks
      Ram

      Comment


      • #4
        Re:Mirth increasing number of open threads on Windows

        You might want to try with JRE 1.5.

        Mirth 1.7.1 Installation note:
        Note that Java 1.5 is required. Java 1.6 can be used, however it is not recommended nor supported
        Hope it helps!

        Comment


        • #5
          Re:Mirth increasing number of open threads on Windows

          rampratapa wrote:
          I am going to remove the code ( Java Native Code and run the process). Do you think the native code runs on a seperated thread? Do we need to do any additional thing to clean-up in the java code?
          I don't know, but having code that isn't part of Mirth running makes it harder to figure out what the problem might be.

          If that code is removed and the threading issue goes away, then the problem is either in that code or how it is used. If the code is removed and the problem is still there, then its an issue with Mirth.
          Jon Bartels

          Zen is hiring!!!!
          http://consultzen.com/careers/
          Talented healthcare IT professionals wanted. Engineers to sales to management.
          Good benefits, great working environment, genuinely interesting work.

          Comment


          • #6
            Re:Mirth increasing number of open threads on Windows

            Thank you for the response. At present I am wokring on removing the java calls from javascript and try to run the test so that I can located the issue.

            I will post my results as soon as I complete my testing.

            Thank you for the help

            Thanks
            Ram

            Comment


            • #7
              Re:Mirth increasing number of open threads on Windows

              It is the JavaScript calling java causing the thread issue. I have commented the java calling code and tested the channel and it is working fine.

              However if you call a java code from JavaScript then it is not closing the threades.

              Case : I have a channel with FileReader and Channel Writer. The is channel reades batch and split the batch into individual requests and feed to other channel with a single request. The following JavaScript code was written in the Source Transformer

              ---- JAVA SCRIPT --START---
              var inMessage = messageObject.getRawData();
              logger.error("Started Processing Batch:" + inMessage );
              try{

              var batchSplitter = Packages.com.service.impl.MirthFacade();
              var splittedBatch = batchSplitter.splitDocument(inMessage);

              logger.error("Completed Java calling");
              var OrignialFileName = $c('originalFilename');
              channelMap.put("OrignialFileName",OrignialFileName );
              var trackingId = 0 ;
              if (OrignialFileName == null){
              OrignialFileName ="";
              }
              if (splittedBatch != null){
              logger.error("In Message" + inMessage + "OrignialFileName" + OrignialFileName);
              header = "<?xml version\="1.0" encoding\="UTF-8"\?> <message> <http.request>/</http.request> <http.version>HTTP/1.1</http.version> <Host>localhost:8091</Host> <Content-Length>5815</Content-Length> <http.method>POST</http.method> <payload>";
              trackingStartTag = "</payload><trackingId>"
              trailer = " </trackingId> <batchId>" + OrignialFileName+"</batchId><User-Agent>Jakarta Commons-HttpClient/3.0.1</User-Agent></message>";

              logger.debug("Incoming Message" + inMessage );
              logger.info("Incoming Message Received" + splittedBatch.size() );
              // route single request to Individual Batch
              for (var index=0; index < splittedBatch.size(); index++) {
              trackingId++;
              payloadMessage = header + splittedBatch.get(index) + trackingStartTag + trackingId + trailer;
              logger.error("routed Message" + payloadMessage );
              router.routeMessage("Channel 2", payloadMessage,true,true);
              }
              logger.info("Batch Was Routed Channel 2" + OrignialFileName);

              }else {
              channelMap.put("ERRMESSAGE",inMessage);
              channelMap.put("ERROR","FATAL");
              }


              }catch (ex) {
              channelMap.put("ERRMESSAGE",ex);
              channelMap.put("ERROR","FATAL");
              logger.error(ex);
              }
              return true;

              ---- JAVA SCRIPT --END---

              MirthFacade is POJO and splitDocument is method that splits the document into indvidual messages.

              When I put 20 files to this channel the mirth creates 20 threads and is not closing the threads even after the process completed splitting and routing the messages. If Add another 20, it is creating 20 more threads now they became 40, so it is increasing the threads.

              NOTE : The java code is not opening any connections and it is pure POJO.

              Please let me know what you think

              Thank you for the help

              Thanks
              Ram

              Comment


              • #8
                Re:Mirth increasing number of open threads on Windows

                Ram,

                I'm finding this very interesting. Thanks for posting about it. I'm still at work, but when I get home I'll play around with it and see if I can reproduce the issue.

                Since removing the call to Java code resolves the threading problem that means that the root-cause is:

                1) The Java code that is being called
                2) Not closing out/destroying the Java objects created (not sure on this, my knowledge of the Java lifecycle is limited)
                3) Mirth (maybe Rhino or Mule) doesn't handle the code correctly

                If it were #1, you should be able to get the threading issue to happen again by just calling that Java code from another application.

                I think if it were #2 or #3 the issue would show up with a simple Java class. Like a "Hello World" class would cause it to break just the same as a "Do lots of buisness logic" class.
                Jon Bartels

                Zen is hiring!!!!
                http://consultzen.com/careers/
                Talented healthcare IT professionals wanted. Engineers to sales to management.
                Good benefits, great working environment, genuinely interesting work.

                Comment


                • #9
                  Re:Mirth increasing number of open threads on Windows

                  I ran a test to verify this and didn't get the same results. I used jconsole to monitor the number of live threads, by starting Mirth with the VM argument -Dcom.sun.management.jmxremote and then starting jconsole.

                  My JavaScript step did the following:

                  var a = Packages.com.webreach.mirth.util.StringUtil();
                  a.convertLFtoCR("asdfsdf");

                  A pretty simple test that also calls a Java class with a static method. After running a couple hundred messages through, the peak thread count never went above 71, and the live thread count seemed to stay around 66 and now rests at 61. Let me know if you have different results, jbartels.
                  Jacob Brauer
                  Director, Software Development
                  NextGen Healthcare

                  sigpic

                  Comment


                  • #10
                    Re:Mirth increasing number of open threads on Windows

                    jacobb,
                    Thank you for the quick test. I was using Non-Static methods ( instance methods) by getting the Instance using Spring Factory. So I have attached Spring Framework for business objects so that I can get them based on the context. I am continuing my research by deleting the Spring Framework and then see the results. Then I will introduce Spring Framework and see the results.

                    I will post the results as soon as I complete my testing.

                    Once again thank you for all your help

                    Thanks
                    Ram


                    UPDATE : 9:18 PM CST

                    I have deleted Spring Factory and it is working fine. I am using Process Explorer to track number of threads Mirth Opened for Processing. When it started it is opening 50 threads and once I put the load they are between 60 to 70 threades.

                    It is working fine now with Pure Java Class.

                    Thank you all of you for the help. Now I am going to work with Spring Framework and let you know what was the problem that is causing the thread leak.

                    Thanks
                    Ram

                    Post edited by: rampratapa, at: 11/18/2008 20:22

                    UPDATE : 11/19/2008

                    I am able to identify the issue. If the class has instance variables ( class level variables) and it is holding a object reference then Mirth is not releasing the thread and therefore the thread count increase over period of time. When I moved those to local to method then it started working fine.

                    I was under the view that if the javascript in the transformer or filter executed all the variable in that including the Java Class varibale will become out of scode and the thread is free from resources. I think it is not the case. However I am able to fix the issue and Spring FrameWork is working fine with Mirth.

                    I would like to thank you all for helping me in this issue.

                    Thank you
                    Ram

                    Post edited by: rampratapa, at: 11/19/2008 09:39

                    Comment


                    • #11
                      Re:Mirth increasing number of open threads on Windows

                      Update : I have eliminated Spring Frame work and Tested with Pure Java Class with logic and It is working Fine. I am using Process Explorer to see how many threads the Mirth Process is using in total and I am getting around 65 and I posted around 1000 messages and it is able to handle those without an increase in thread count.

                      I will be looking into the Spring Framework and where it is causing the problem and let you know.

                      Thank you all for the help and quick response.

                      I will update you further once I resolve the issue with Spring Factory.

                      Thanks
                      Ram

                      Comment


                      • #12
                        Re:Mirth increasing number of open threads on Windows

                        Ram, I tried to create a test case to reproduce your behavior using two classes and an instance variable that creates the second class. I could not get threads to stay around. Please let me know if you have a minimal test case that can reproduce this.
                        Jacob Brauer
                        Director, Software Development
                        NextGen Healthcare

                        sigpic

                        Comment


                        • #13
                          Re:Mirth increasing number of open threads on Windows

                          Jacobb,
                          I will try to create a test case and let you know soon.

                          thank you
                          Ram

                          Comment


                          • #14
                            Re:Mirth increasing number of open threads on Windows

                            Jacobb,
                            I have the following class

                            public class MirthFacade {

                            private static final Logger logger = Logger.getLogger(MirthFacade.class);
                            // Each service is defined and get the application context when ever needed
                            private BatchHandler batchHandle ;
                            public MirthFacade() {
                            }
                            /*

                            public List<String> splitDocument(String batchDocument) {
                            logger.debug("Started Splitting the batch:" );
                            batchHandle = (BatchHandler)ApplicationContextFactory.getApplica tionContext().getBean("BatchSplitter");
                            return batchHandle.SplitBatch(batchDocument);
                            }
                            }

                            which I have used to call from transformer of the a channel also I was using Spring Framework for getting the business object using ApplicationContextFactory. In the above case the threads were increasing when a new file was sent to file reader. Now when I changed to the follwoing way

                            public class MirthFacade {

                            private static final Logger logger = Logger.getLogger(MirthFacade.class);
                            // Each service is defined and get the application context when ever needed

                            public MirthFacade() {
                            }
                            /*

                            public List<String> splitDocument(String batchDocument) {
                            logger.debug("Started Splitting the batch:" );
                            BatchHandler batchHandle =null;
                            batchHandle = (BatchHandler)ApplicationContextFactory.getApplica tionContext().getBean("BatchSplitter");
                            return batchHandle.SplitBatch(batchDocument);
                            }
                            }

                            Then it is working fine. It is stable now.

                            Let me know if you need more information

                            Thanks
                            Ram

                            Comment


                            • #15
                              Re:Mirth increasing number of open threads on Windows

                              Ram,

                              Thanks for that code. Do you have any way of reproducing it without using anything related to Spring? I'd like to see it without getting a bean or using the ApplicationContextFactory. Something with just standard java libraries would be great. I tried the same thing with only two java classes doing simple tasks and didn't get the same thread behavior.
                              Jacob Brauer
                              Director, Software Development
                              NextGen Healthcare

                              sigpic

                              Comment

                              Working...
                              X