Announcement

Collapse
No announcement yet.

DIY Queue Depth Alerts

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • DIY Queue Depth Alerts

    I thought I'd share way I have put together for Mirth Alerts triggered by Queue Depth.

    I would welcome any thoughts, comments or suggestions for improvements as it is far from perfect.

    Setup:
    I assumed the easiest way to accomplish this custom alert was with the postprocessor. My company has hundreds of instances of Mirth in the field with literally thousands of channels so in a plan to ease a future wide spread distribution, I put my code in the Global Postprocessor. It has some drawbacks that I will point out later.

    One big concern I had was that I knew I didn't want a simple threshold checker. This would have caused me to get an alert for threshold, threshold+1, ....,threshold+n blowing up my mailbox until it was fixed. To solve this, I used modulo of the threshold. In the example code below I used 100, so that for each multiple of 100 an alert is generated. Not great, but better than just a >100 strategy.

    Code:
    // This script applies across all channels
    var channelController = Packages.com.mirth.connect.server.controllers.ChannelController.getInstance();
    var channelName = channelController.getDeployedChannelById(channelId).getName();
    var channelNameUpper = channelName.toString().toUpperCase();
    
    //Open database connection
    var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
    var expression = 'SELECT queued FROM channel_statistics WHERE channel_id = \''+channelId+'\';';
    //channelResult is of java type ResultSet
    var channelStatResult = dbConn.executeCachedQuery(expression);
    var numOfQueued = 0;
    
    while(channelStatResult.next())
    {
    	numOfQueued = channelStatResult.getInt("queued");
    	if(numOfQueued % 100 == 0 && numOfQueued != 0)
    	{
    		//Generate an alert
    		alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);
    	}
    }
    
    return;
    We use postgres as our DB but I don't think this couldn't be adapted to any other. The key is that after each message, the post processor trawls the DB for the queued count. Admittedly, maybe not the most efficient process but I haven't noticed any performance issues. It might be possible, although I haven't explored it, to reference the message status. I know it can be done from the individual channel postprocessor as you have access to the messageObject variable as well as the responseMap which would tell you all you need to know. But I am not sure if you can do it from the Global Postproccessor. If you could, you might be able to improve efficiency by making the above code conditional on the message status being queued. Now that I think of it, you would probably have to set it up individually for each channel and its specific destinations because of the responseMap's getStatus arguments are literal strings. This wouldn't work for me because of how many Mirth servers and channels I have but might work for anyone who has only a handful of channels to maintain.

    In addition to the above postproccessor, you will need to set up Alert's as you normally would. With the SMTP settings and it should be set to monitor any channels you wish. The alerts.sendAlert line will throw a Custom Error 302 and the string within that line can be accessed through the ${error} variable. I did it this way because I couldn't get Global Variables to work, but this did the trick. My Email Body looks something like:

    PHP Code:
    Alert!
    -----------------------------------------
    Alert generated at : ${date}

    The following channel is experiencing a large volume of queued records to its destination(s).

    ${
    error}
    -----------------------------------------
    **
    This e-mail was automatically generated by the Alerting ServiceReplies to this message are not monitored.** 
    And that is about all it takes. Now, there are some rough edges. First, if alot of messages queue up very quickly, or if there is simply alot of traffic, then I have noticed some multiples of 100 get skipped and no alert is ever sent. This probably has something to do with the fact that the Global PostProccessor is having to manage the 200 channels on my test server. If that is the case, moving it to the channel's own postprocessor should fix this.

    Another issue I personally have, because of our setup, is that new channels can be added at any time to any of our instances of Mirth. I was wondering if anyone had any ideas about possible ways to utilize the global startscript to check for new channels and add them to the alert. I am currently thinking about trawling the DB for all the channel ID's then inserting into the requisite tables. I think there are three, alert, channel_alert, and alert_email. But I will need to check. Any thoughts or ideas would be welcomed.

    Lastly, when I was working on this, I found something that troubled me. Because the queuestore is not a part of the DB, but rather a mule file system, it looks like that when a channel or destination is deleted, its queuestore remains and is not deleted. I am thinking about modifying my postgres-channel.xml file to clean this up but I was curious if this was a known issue, not considered an issue, of if this is not how it is supposed to work. I don't mind submitting the JIRA request myself if neccessary but maybe someone from the Mirth Team can chime in. Jacob, I'm looking at you


    BTW, I am most certainly not the first to come up with something like this. In the interest of giving credit where credit is due, Chad recently came up with something similar here. I am sure there are others, but Chad's posting led me to submit what I had developed awhile back so that others could benefit and also maybe offer suggestions to ironing out the issues I mentioned.

  • #2
    Auto-magically Add Newly Created Channels to an Alert

    After I wrote the above post, I decided to pursue my idea of automatically adding new channels to this alert. I wrote the following script that is meant to run in the Global Deploy Script. The only requirement is that you must tell it what the name of your alert is.

    After a deploy, the script gets the alert id for the name you have entered. It then queries the DB to get all channels associated with that alert id from the table channel_alert. It then gets all the channel ids. Finally it compares the two arrays of channel id and if it finds a channel on the channel list that is not in the alert list, it inserts it into the database.

    I tested this and the Alert on Mirth 2.0.1 running Linux, but that shouldn't matter.

    Code:
    // This script executes once when the mule engine is started
    // You only have access to the globalMap here to persist data
    
    /***************************************\
    *	This adds any new channels to the	                *
    *	queue depth alert.					*
    ****************************************/
    
    //Get the ID of the alert
    //Open database connection
    var dbConn = DatabaseConnectionFactory.createDatabaseConnection('org.postgresql.Driver', 'jdbc:postgresql://localhost:5432/mirth', 'mirth', 'mirth');
    var expression = 'SELECT id FROM alert WHERE name = \'Queue Depth Alert\';';
    //channelResult is of java type ResultSet
    var alertId = dbConn.executeCachedQuery(expression);
    while(alertId.next())
    {
    	var id = alertId.getString("id");
    }
    
    //Get the list of channels
    expression = 'SELECT id FROM channel ORDER BY id ASC;';
    var channels = dbConn.executeCachedQuery(expression);
    //Build an Array of channels
    var channelArray = new Array();
    while(channels.next())
    {
    	channelArray.push(channels.getString("id"));
    }
    
    //Get the list of channels for the Alert
    expression = 'SELECT channel_id FROM channel_alert WHERE alert_id = \''+id+'\' ORDER BY channel_id ASC;';
    var alertChannels = dbConn.executeCachedQuery(expression);
    //Build an Array of channels
    var channelAlertArray = new Array();
    while(alertChannels.next())
    {
    	channelAlertArray.push(alertChannels.getString("channel_id"));
    }
    
    //Simple Exit Test
    if(channelAlertArray.length != channelArray.length)
    {
    	 //index for the channelArray
    	for(var i = 0; i < channelArray.length; i++)
    	{
    		var index = -1;
    		//index for the Alert Channel Array
    		for(var k = 0; k < channelAlertArray.length; k++)
    		{
    			if(channelAlertArray[k].toString() == channelArray[i].toString())
    				index = k;
    		}
    		if( index == -1 )
    		{
    			//Insert into the table
    			expression = 'INSERT IGNORE INTO channel_alert VALUES (\''+channelArray[i]+'\', \''+id+'\');';
    			dbConn.executeUpdate(expression);
    		}
    	}  
    }
    dbConn.close();
    return;

    Comment


    • #3
      This is a very nice setup. I'm going to take your idea and plug it into Icinga as a Mirth-queue monitor. Thanks for the hard work you put into this. If I get success, and you don't mind, I'll post my work to the Icinga (Nagios) plugins repo, and of course here.

      Comment


      • #4
        Originally posted by laidback_01 View Post
        This is a very nice setup. I'm going to take your idea and plug it into Icinga as a Mirth-queue monitor. Thanks for the hard work you put into this. If I get success, and you don't mind, I'll post my work to the Icinga (Nagios) plugins repo, and of course here.

        I don't mind at all. Let me know if do have success making it work with Icinga. I'd be curious to see how.

        Comment


        • #5
          Thanks for the example. I will try using this, but have simplified by removing the database lookup for the queue count as shown by using the channelStatisticsController:

          var channelStatisticsController = Packages.com.mirth.connect.server.controllers.Chan nelStatisticsController.getInstance();
          var stats = channelStatisticsController.getStatistics(channelI d);
          var numOfQueued = stats.queued;

          if(numOfQueued % 100 == 0 && numOfQueued != 0)
          {
          //Generate an alert
          alerts.sendAlert("\n\nChannel: "+channelName+"\n\nQueue Depth: "+numOfQueued);
          }

          Simon
          Last edited by seaston; 02-20-2011, 05:06 AM.

          Comment


          • #6
            Python

            In my experience the channel statistics queue count hasn't been quite what i'm interested in knowing (assuming that's the same value that shows up under the 'Queued' column on the dashboard). Instead when we get a lot of messages in to a channel the only way to see how many messages are backed up has been to count the files in the queuestore folder.

            So here is what we're currently doing to track our queues. It uses python, mutt, and runs on linux.

            Code is in python
            Code:
            import os
            import commands
            basedir = '/path-to-appdata/queuestore/'
            limit = 30
            queueToChannel = {'CHANNEL ID':'CHANNEL NAME'}
            #loop through queue folders
            for dirname in os.listdir(basedir):
                    # ls -l    lists 1 file per line,  wc -l   counts the number of lines returned
                    cmd = 'ls -l ' + basedir + dirname + ' | wc -l '
                    #get results of wc -l  
                    filecount = commands.getoutput(cmd)
                    try:
                            filecount = int(filecount) -1;
                    except:
                            filecount = 0;
                    if (filecount > limit):
                            commands.getoutput('echo "mirth queuestore ' + dirname + ' representing ' + queueToChannel[dirname] + ' has ' + str(filecount) + ' files" | mutt -s "mirth queuestore for ' + queueToChannel[dirname] + ' is above limit of ' + str(limit) +'" [email protected]')
            Anywho, this has been helpful for us but does require manual entry of the queueToChannel mappings. Not sure if it'll be helpful for you guys or not, but better to post and find out it wasn't needed than vice versa

            -Scott

            Comment

            Working...
            X