File reader processes files in random order

[nshaik]

=========
Mirth v1.1

Channel: File Reader to Database
=========

File reader randomly picks the files from the "Directory to read" folder.

Q: How to setup Mirth to process oldest files first (FIFO - First-In-First-Out order)?

More Details
=========

Setup Mirth to read files from a folder which had about 22,000 message files with datetime ranging from 10/10/06 3:30 PM to 10/12/06 6:00 PM. The messages were expected to be processed with oldest file first (Operating system file create date).

In this specific test case, it should process files from 10/10/06, 10/11/06, and then 10/12/06. But Mirth randomly picks the files from the "Directory to read" folder and process them (the files are getting moved in random order in the "Move-to Directory" folder).

This would make the updates incorrect. For e.g., A31 message for patient update on 10/10/06 and 10/12/06 for the same patient. If the 10/12/06 A31 message gets processed first and then 10/10/06, the patient data would be incorrect.

NOTE: This problem may not occur when the files are getting processed in real-time i.e., as and when they arrive. But when Mirth HL7 server is down due to hardware / network problems, the HL7 host would be sending the messages and they get accumulated in the target folder. Later, when the Mirth HL7 box is turned back on and is up and running, it needs to process the messages in the order they arrived.

Thanks in advance!

[chrisl]

Re: File reader processes files in random order

This is a good idea - can you add this as an issue to the JIRA? (www.mirthproject.org/jira).

Thanks!

[nshaik]

Re: File reader processes files in random order

Chris,

Added this problem to the issue tracker.

As I understand, a possible solution would be an option to process a static list. Details below:

Suggestions:
==================================
1. SHORT TERM TEMPORARY WORKAROUND

Utility / Script to move files in sorted order (by file-date) from a TEMP folder to the INBOUND folder with a time delay (in seconds)
==================================

* Stop Mirth
* Rename current inbound folder to temp and create a new empty inbound folder
This is basically to save time in moving large set of files to a temporary folder.
* Start Mirth
* Write a utility / script which will take TimeDelayInSeconds as parameter:
Move the files in sorted order (oldest file first) from temporary folder to
INBOUND folder with the specified delay

This should take care of the problem of processing the files in the order received.

=======================
2. A Possible Solution
=======================

* Mirth Admin to have the following option for "File Reader"
[Check box] Use static list of files

* Display message on ?Deploy Channel?
You selected "Use static list of files". Mirth will work only with the current
snap-shot of files and ignore other files that come in later.
Are you sure you want to continue?
* Process the static list of files in the order received (FIFO)
* mirth.log file to show

************************************************** *********
--> [Datetime stamp] Processing static list of files...

--> Total no. of files in the static list: [nnnn]

--> First file in list: [File name with the file create date] (oldest file in the list)

--> Last file in list: [File name with the file create date] (latest file in the list)

--> IMPORTANT:
--> Please log back to Admin module and turn "Use static list of files" option off.
--> Redeploy channel to continue normal processing.
************************************************** *********