Announcement

Collapse
No announcement yet.

FileReader and "fileLastModified" based filter

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • FileReader and "fileLastModified" based filter

    Hello,

    I've a channel reading in a directory (using FileReader connector) that contains thousands of DICOM files.
    As expected, these DICOM files are heavy (from 20 MB to 1 GB).

    In the FileReader connector, I set up a filter to getting files that are older than a defined date (let's says 30 minutes) based on the fileLastModified property.

    I have search and read about these feature and it seems it first loads files before testing the fileLastModified properties.

    http://www.mirthcorp.com/community/f...ead.php?t=7487

    With DICOM files, it's not manageable.

    Is there a way to apply such a filter prior to load theses files ?

    Otherwise, I would develop a JS connector based on libs like JCIFS, but it's not IMO the best option.

    Thx for your help.

    PS : I use MC 3.1.1.7461
    Last edited by lionelroux; 01-29-2015, 05:33 AM. Reason: version

  • #2
    First, what exactly are you talking about with the "fileLastModified property"? Are you talking about the actual OS last modified timestamp for the file? Or are you talking about a DICOM-specific tag contained within the file?

    If you're talking about the OS last modified timestamp, then it's not true that files get loaded into memory before that value is checked. Only the OS file information (like the filename, last modified date, size, etc.) is loaded, but not the file contents.
    Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

    Nicholas Rupley
    Work: 949-237-6069
    Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


    - How do I foo?
    - You just bar.

    Comment


    • #3
      yes I talked about the OS property.
      So I'm conforted to know files are not loaded.

      So maybe it's because there are so many files in the directory that it takes so much time to read files and send them into my channel.

      So i'll apply this filter.

      Thank you.

      Comment


      • #4
        Back with my FileReader memory "issue".

        So today I tried to load some files in the directory. There are 483 DICOM files in it.
        I set up a filter based on the OS last modified property to get only files that are not older than 1 hour. This files subset is approx. 30 files.

        My channel has a queue set on for the source connector.

        The first 300 files are processed very quickly, let's say 75 files every 10 sec.
        But then, it takes more and more time, and at the end, I get an out Of Memory error from the server.

        Here is the Stacktrace :

        ERROR 2015-01-30 16:20:44,946 [Thread-126] Server: Exception in thread "Thread-126" java.lang.OutOfMemoryError: Java heap space
        ERROR 2015-01-30 16:20:44,947 [Thread-126] Server: at java.util.Arrays.copyOf(Arrays.java:2367)
        ERROR 2015-01-30 16:20:44,947 [Thread-126] Server: at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.java:130)
        ERROR 2015-01-30 16:20:44,947 [Thread-126] Server: at java.lang.AbstractStringBuilder.ensureCapacityInte rnal(AbstractStringBuilder.java:114)
        ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at java.lang.AbstractStringBuilder.append(AbstractStr ingBuilder.java:535)
        ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at java.lang.StringBuffer.append(StringBuffer.java:32 2)
        ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at java.io.StringWriter.write(StringWriter.java:94)
        ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at com.sun.org.apache.xml.internal.serializer.ToStrea m.characters(ToStream.java:1504)
        ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at com.sun.org.apache.xml.internal.serializer.ToStrea m.characters(ToStream.java:1618)
        ERROR 2015-01-30 16:20:44,948 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:244)
        ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:230)
        ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:230)
        ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:136)
        ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2T O.parse(DOM2TO.java:98)
        ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transformIdentity(TransformerImpl.java:
        683)
        ERROR 2015-01-30 16:20:44,949 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transform(TransformerImpl.java:727)
        ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transform(TransformerImpl.java:340)
        ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.model.converters.DocumentSeriali zer.toXML(DocumentSerializer.java:94)
        ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.model.converters.DocumentSeriali zer.toXML(DocumentSerializer.java:102)
        ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.plugins.datatypes.dicom.DICOMSer ializer.toXML(DICOMSerializer.java:181)
        ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.FilterTran sformerExecutor.processConnectorMessage(FilterTran
        sformerExecutor.java:91)
        ERROR 2015-01-30 16:20:44,950 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.Channel.pr ocess(Channel.java:1473)
        ERROR 2015-01-30 16:20:44,951 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.Channel.pr ocessSourceQueue(Channel.java:1738)
        ERROR 2015-01-30 16:20:44,951 [Thread-126] Server: at com.mirth.connect.donkey.server.channel.Channel.ru n(Channel.java:1725)
        ERROR 2015-01-30 16:20:44,951 [Thread-126] Server: at java.lang.Thread.run(Thread.java:722)


        My box is a Linux one, with 6 Go memory, 3Go for the JVM, 4 CPU-cores (all on a VM-ware env.).

        So it seems there is something with the memory even if the message is filtered.
        Any idea ?

        thx in advance

        Comment


        • #5
          The DICOM-to-XML process is the root cause of my OOM error.
          If I set datatypes as DICOM, without converting to XML, it's ok.

          So the workaround for my case is to set up a child channel that receives only the non-filtered messages. This one will convert DICOm messages to XMl so that I can alter some values.

          Comment

          Working...
          X