OpenX Blog

The database is an important part of Openads. Maybe it is running on the same machine as your adserver or maybe you have a larger site with a dedicated database server. Either way, you want to know everything is running smoothly and when things go wrong you want to know about it quickly.

This is where monitoring comes in. Network monitoring plays a vital role in any production system. You need to know that your web site is accessible and your database hasn’t crashed.

There are a number of different Network Monitoring Systems around, some free and some not so free. Nagios has been around for ages and lots of plugins are available for it, although it is not designed to be a replacement for a full-blown SNMP management application like OpenNMS. According to the OpenNMS website, OpenNMS is the first enterprise-grade network management platform to be developed under the open-source model.

I’m going to show you how to combine the power of OpenNMS with the extensibility of the Nagios plugin system to monitor the InnoDB free space in your database. I won’t go over installation of OpenNMS itself but the latest 1.3 version is pretty straightforward to get going. It comes bundled with Jetty so you don’t even need to worry about setting up a tomcat server to run it (unless you want to).

OpenNMS can perform many monitoring tasks remotely. For example, it can check your mysql service is running by polling the mysql port and it can use SNMP to collect all sorts of data, from network statistics to disk usage. However, when it comes to monitoring application specifics like the InnoDB free space things get a little harder. An easy way to do this is to use the Nagios Remote Plugin Executor (NRPE) capability of OpenNMS. OpenNMS will then see NRPE as just another service like MySQL or SSH which it will start polling.

NRPE is a daemon that you run on the remote (target) machine. The monitoring server then connects to this daemon which runs a plugin that can do, well, whatever you want it to really. There are loads of plugins available on NagiosExchange.

Setup NRPE

• On the machine to be monitored, install the required packages:


# yum install nrpe nagios-plugins-nrpe nagios-plugins


• Plugins go in /usr/lib/nagios/plugins
• On the monitoring machine install nagios-plugins-nrpe for testing.
• Unfortunately, OpenNMS can’t talk SSL to NRPE so we need to modify /etc/init.d/nrpe on the target machine to use the -n switch:


# See how we were called.
case "$1" in
start)
# Start daemons.
echo -n "Starting nrpe: "
daemon $NrpeBin -c $NrpeCfg -d -n
echo
touch $LockFile
;;


• Start nrpe on the target machine using ’service nrpe start’ (make sure you open port 5666 for your monitoring server)
• On your monitoring server, check you can connect (this should return the NRPE version):


$ /usr/lib/nagios/plugins/check_nrpe -n -H example.com \
-c _NRPE_CHECK


• Now we need to add the plugin. Grab this script by Sheeri Kritzer and put it in /usr/lib/nagios/plugins. Make sure it is executable and add it to the NRPE config file (/etc/nagios/nrpe.cfg). The command should all be on one line:


# Check innodb free space, warn at 1GB, critical at 500MB
command[check_innodbfree]= /usr/lib64/nagios/plugins/check_innodbfree.sh -H [hostname] -u [user] -d test -w 1000000 -c 500000


• Configure the parameters for host, user, password and database. The optional parameters are kB to warn at and kB to be critical at, however, OpenNMS makes no distinction between these two and will fire an event on warn as well as critical.
• Test it out by running the plugin from your monitoring server:


$ /usr/lib/nagios/plugins/check_nrpe -n -H example.com \
-c check_innodbfree
example.com has 82469888 kB free


Setup OpenNMS

• Take a look at the examples in /opt/opennms/etc/examples for poller-configuration.xml and capsd-configuration.xml
• Copy the service and monitor definitions from poller-configuration.xml and the protocol-plugin definition from capsd-configuration.xml to your live config files, modifying such attributes as command and service name as appropriate.
• When you restart OpenNMS it should detect the new service.

Now, whenever OpenNMS polls your new NRPE service, the check_innodbfree.sh script will be run and, depending on the return value, OpenNMS will trigger an event.

Obviously you will want to restrict who can connect to your nrpe daemon using firewall rules.