No announcement yet.

Message destinations in FIFO order

  • Filter
  • Time
  • Show
Clear All
new posts

  • Message destinations in FIFO order

    Question: How can I make sure that Mirth handles destination messages in sequential order (FIFO principle = first in, first out), meaning that the 2nd message in the destination "queue" is processed after the 1st message has been fully processed?

    Situation: A system creates JPG images (medical study, taken with a camera) and sends a single HL7 message for each image. After all images have been created it finally sends an additional message. The number of messages is N +1 where N is the total number of images.

    My Mirth is receiving these messages and has 2 destinations with the following tasks:
    1. Destination 1 reads the JPG images, base64-encodes them and finally uploads them using a RESTful API (HTTP POST request). Due to the I/O and latency to the API this takes a while per image, also depending on the JPG filesize.
    2. Destination 2 handles the final message and reports to the same RESTful API that the upload of all images has been finished. Once this has been done no more images can be uploaded.

    Problem: The final message (N+1) is processed before all images have been uploaded. This prevents running uploads from being finished.

    Final question: How can I solve this problem and force Mirth to process the destination of the 2nd message after the 1st message destination processing has finished?

  • #2
    By default they should process in the order they were received. A number of configuration options can change that.

    Do you have more than one source thread configured?
    Do you have destination queuing turned on?

    If you could share your channel configuration that might help.


    • #3
      Yes, I'm sure they're processed in the order they were received but I assume the next message will not wait until the previous one is processed completely? That's probably why the 2nd message processing might finish earlier than the 1st one...

      Or it's a limitation of the HTTP Sender? Maybe the next message is processed right after the HTTP request of the previous message is sent out... but probably Mirth doesn't wait for the response (or until the defined timeout). Or a data-form multipart upload is a special thing and causes this behavior?

      No, just 1 thread and no queuing in destinations.

      Please find the channel attached reduced to the necessary code.

      ... by the way it includes a working method of how to do a data-form multipart upload of binary data.
      Attached Files


      • #4
        I haven't had a chance to look at your channel yet, but if your channel has a single source thread and no destination queuing, then it shouldn't be possible to start processing the next message before the previous one completes.

        An exception to that could possibly be if you're doing something in javascript that spawns an additional thread allowing the message to "finish" before it's actually done.


        • #5
          This may seem like a silly question, but have you confirmed that you are receiving the messages in the expected order?


          • #6
            Originally posted by agermano View Post
            This may seem like a silly question, but have you confirmed that you are receiving the messages in the expected order?
            Yes I can see in the dashboard that the messages are coming in the right order. If I ignore the last message then all image uploads get completed. When I process the last one the messages before are not fully processed - like cancelled because the API declined to finish the upload operation.


            • #7
              Is there maybe processing happening on the remote server that occurs after you upload the image files that is interrupted when you send the final message? Can you glean anything from the destination response times in the message browser as to the order they were received at the remote end?

              Maybe you just need to introduce an artificial delay before sending the final message if the remote system can't handle receiving them so closely together.

              You can put a java.lang.Thread.sleep(millis) in the destination transformer for the non-image destination.


              • #8
                Coming back to this because I still don't have a solution and somehow it's hard to test.

                But today I found this in the Mirth Guide, see on page 224:

                Source Settings

                Item: F
                Name: Max Processing Threads
                Description: The maximum number of messages that can process through the channel simultaneously. By default this is set to 1, meaning that only one message can process through a channel at any given time (does not include asynchronous processes like the destination queue). Increase this setting can greatly improve channel performance / throughput, at the cost of message order preservation.
                What does it mean?

                Compare to page 224:
                Scripts Tab

                Postprocessor Script: This script executes once for every message, after all destinations have completed processing (not including queued messages which are processed asynchronously). You have access to "message", which is an ImmutableMessage object containing information about the state of all connector messages. This script may be used as a general tool to perform a custom cleanup script. It can also be used to return a custom response that may be sent back to the originating system.
                Is this related only to QUEUED messages or in general to all outbound messages of all destinations?


                • #9
                  After some more investigation now I think I understood that I have to set in destination settings "Queue Messages: Always" instead of Never/On Failure so I can guarantee all messages are sent to the queue immediately and then will be processed/sent in the correct order. Right? And of course w/o rotating and 1 thread only in the advanced queue settings.

                  Additional question: Does every destination have its own queue?

                  If yes:
                  • They still could mix up because my channel has 2 destinations and each queue is running in an own thread...
                  • Can I force to send them all to the same queue?


                  • #10
                    If you have Queue Messages set to Never, and there is only one destination chain (multiple destinations always wait for previous) then your source and destinations will all process in the same thread. When using a single source thread, one message will completely finish before the next one starts.

                    If you have multiple destination chains, this will spawn extra threads, but your single source thread still shouldn't start the next message until all destinations have a response and the post-processor runs. If you are using queuing, then QUEUED is a response.

                    You are correct that setting both destinations to always queue could allow them to process out of order. You can't force them to the same queue. Each destination does have its own queue.