Announcement

Collapse
No announcement yet.

Dynamically control destination queue thread count?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Dynamically control destination queue thread count?

    Does anyone have an architecture that allows or simulates a dynamic thread size for a destination queue?
    Since the field does not accept a variable. The only method, so far, that I can think up is to have a channel with a bunch of destinations, one per thread, up to your max.
    And then somehow enable/disable them to grow or shrink the thread pool. Not even sure how or if that can be done.

    threads.png
    Attached Files

  • #2
    Hi,

    May I ask what is your use case?

    I don't think there is a way to do this from inside a channel - or from inside Mirth. Actually you should just set queue thread count based on expected trafic.

    The only way I could imagine doing something like this would be to change the value in a channel's xml definition file stored locally, then upload the channel through REST API or using CLI and finally redeploy the channel through REST API (not sure if possible using CLI). Seems a bit complicated, and you still have to define WHEN to trigger this, and if it will have any effect on currently queued - I suppose it should since "regenerate template" option is set.
    Tom

    Comment


    • #3
      The number of destination queue threads (each with their own in-memory cache) cannot be changed on a deployed channel. There isn't a way to start or stop individual threads, only the entire connector.

      I'm also interested in your use case. Having a few idle threads is not going to greatly impact anything. Definitely insignificant compared to them all processing messages simultaneously. If your system can handle that many threads active, it should be able to handle them idle with no problem.

      If you went the route you are suggesting with multiple destinations, you would also need to be somehow monitoring load to know when to "spin up" or shutdown a destination, and then figure out how to balance load between active destinations.

      Comment


      • #4
        My use case is this; we have an external service that we download data from. We have an agreement with this vendor that we will limit the number of threads we hit them with. Our production run for the evening varies in length. Some nights we download for an hour, others up to 3 hours. All good. Now, add the need for doing "backfills". A backfill for us is where we are downloading a bunch of initial data as part of a new integration. It's rare, but when it happens, we want to share the "thread pool" we have agreed to stay within. If we need a backfill tonight, we want to be able to "steal" those threads from our production server, after it finishes for the evening. Via some monitoring, etc, we know when the backfill server can "have the threads" for the rest of the evening, since production is done with them. If the thread count variable was dynamic, we could just set it via a configuration map entry and redeploy the channel. Our production server and backfill server are two different servers. We'll figure out the intra-server channel communication somehow. But right now, we are stuck having to set the thread count.

        I'm almost thinking this can be done with some creative use of the threadAssignmentVariable. I have used that in another setting to control a channel to run single threaded in a preprod environment vs multi-threaded in a prod environment. Simply by giving it a real calculation (using some message data) vs hardcoding basically a static value, forcing all message into a single thread, even though several others are available. Maybe it's as simple as using modulo. I might be overthinking this. Would a simple "mrn mod $(threadCount)" be the answer? Is it that simple.

        Comment


        • #5
          Hi,

          Thanks for sharing your use case.

          If I understand it correctly, production has finished (absolutely no more trafic) when you want to trigger the backfill, is this true? or is there still a small amount of trafic on prod ?

          In the latter case, the approach with intra-server bridge is probably the smartest. But if you want to assign a variable based on mrn mod threadcount I'm actually wondering if it is really necessary, because it would actually apply on both flows, because it means that the originating download (prod or backfill) is not relevant and you accept your prod flow being "throttled" by backfills when they occur. If not acceptable, a (maybe not that) creative approach is then necessary.

          In the former case, I'd simply do nothing but let the 2 servers run independtly and have the backfill server being triggered by your monitoring system.
          Tom

          Comment


          • #6
            That should work to use the thread assignment variable to limit the number of active threads, knowing that you'll always be using at least 1 thread if the channel is running. You could put the divisor for your modulo operation in the configurationMap, and that should immediately be picked up on the next message without requiring a redeploy. It's possible (though maybe a little clunky) to update the configurationMap via the Client API if you wanted to automate it.

            I would probably use the message ID instead of the MRN as your dividend to ensure you evenly spread your requests over the threads (and also avoid issues with MRNs that might have non-digit characters)

            Comment

            Working...
            X