Munin is a system monitoring program designed to capture and graph performance statistics with minimal configuration. It is significantly easier to configure than MRTG and has a wide variety of plugins. The Ubuntu (Debian) installer will setup a reasonable configuration on installation. It requires one server, and a client daemon on all monitored servers including the server. Scalability to large (campus) environments has been reported to be an issue
Install the munin package on the server. This will install both the munin server code and the munin-node daemon.
The configuration for the server is located in /etc/munin/munin.conf. The more hosts you monitor the heavier the load on the server.
Adding Clients
Install the munin-node package on all clients. The client runs a daemon and has minimal resource requirements. You will need to add an allow line to the file /etc/munin/munin-node.conf. If multiple severs are used add additional allow lines. This should be based on the existing match for the localhost connection. Add a server specification for the client in /etc/munin/munin.conf on the munin server.
Adding Services
If you add a new service you want to monitor to a server you will need to adjust the configuration of that client. The server will automatically start monitoring the new service. The command munin-node-configure --shell will generate the script to add some new services. The command munin-node-configure --shell | sh will create the necessary links
Monitoring Apache2
To monitor your Apache web server you will need the libwww-perl package installed. You will also need to enable mod-status on the Apache server and ExtendedStatus in the /etc/apache2/mods-available/status.conf file. Add the following:
# Turn on extended status for munin ExtendedStatus On
Monitoring an OpenWRT Server
A shell version of munin is available in the repository. It is not as flexible as the full version, but provides reasonable monitoring. Install the muninlite package with opkg. This will install the xinetd package as well. xinet needs to be started and the /etc/init.d/xinetd command needs to be enabled. Then add an entry on the munin server with appropriate address restrictions. Monitoring will automatically configured.
This tool does not provide disk space monitoring. If you wish to disable the uptime monitoring, you will need to edit the script /usr/sbin/munin-node. Any changes will be overwritten by package updates.
If a network interface is very busy, then the its statistics may be distorted. This is due to overflows of the 32 bit counters.
Patch for missing hostname Command
If you get limited output (CPU usage only), muninlite may require a hostname command. You can test this by using telnet to connect to port 2424 on your OpenWrt system. If it does not include the correct name in the greeting you will need to patch your system. There is a posted patch for muninlite, but I prefer to create a hostname command.
Verify the hostname command is missing by running it before applying this patch. Create a hostname command by saving the following 2 line script as /usr/bin/hostname. Be sure to run chmod 755 /etc/bin/hostname after creating it.
#!/bin/sh cat /proc/sys/kernel/hostname
Tuning munin.conf
Examine the munin.conf file to see what changes you wish to make. Changes made before the first server section will apply globally. They can be specified on a per server or per domain basis by placing the changes in the appropriate server or domain section.
By default rate graphs use a period of one second. On less active servers using minutes may be more appropriate. Alter munin.conf to specify the graph_period in the global section. Add this line:
graph_period minute
Notifications when limits are exceed can be enabled by specifying a contact command. Email notifications can be enabled with a line like:
contact.email.command mail -s "Munin notification" admin@example.com
Tuning Graphics Colours
The default colours for the CPU usage graphs use bright colors for idle and nice, neither of which are indicative of load issues. I override the colours on a server by server basis in munin.conf. It does not appear to be possible on a more general basis.
For servers running boinc several colors are overridden.
# Override colors on CPU graphs - boinc running applications cpu.idle.colour FFFF66 cpu.nice.colour FFEEEE cpuiowait.colour BB2222
For other servers only overide idle time.
cpu.idle.colour FFFFBB
Implementing the ip_ plugin with Shorewall
Shorewall creates the rules required for the ip_ plugin. This simplifies implementation of the plugin. Setup Shorewall with the desired configuration. For a workstation, the one-interface example is a good starting point. If you don’t use Shorewall, you will need to ensure the iptables rules are created.
Implement the plugin by creating the a symbolic link into the plugins configuration directory. Also disable the old plugin, retaining error statistics
ln -s /usr/share/munin/plugins/ip_ /etc/munin/plugins/ip_eth0 rm /etc/munin/plugins/if_eth0
If you are monitoring more than one interface, repeat the above commands changing the interface as required.
Refactoring Server Groups
By default servers are grouped by DNS zone. You can change the group by prefixing the server name by a group name followed by a semi-colon in munin.conf. For example you could specify [DMZ;mail.example.com]. Existing statistics can be retained by moving the rrd files prior to changing the configuration.
To setup the above change use the following commands. (Assumes a standard Ubuntu configuration).
cd /usr/lib/munin mkdir DMZ chmod munin:munin DMZ ln example.com/mail.example.com* DMZ /etc/init.d/munin-node restart
After you have changed the configuration you can remove the old file names. Verify that the files have two links with ls -l before removing them. Files with one link will be lost.
cd /usr/lib/munin/example.com ls -l mail.example.com* rm mail.example.com*
Security Considerations
This service requires TCP access on port 4949 from the munin server. All traffic travels in the clear. The main risk is information leakage, although most of the information will be available from the munin monitoring site. The Munin FAQ contains information on tunnelling the traffic over ssh. In this case, the firewall would not need to allow access on the munin port.
The munin-node client runs as root. The protocol provides read only access to information provided by the plugins. Several monitors run under less privileged accounts limiting their risk.