RRDTool

The intention is to monitor our postfix installation, which also uses amavisd-new, spamassain and clamav, and then to display that information with graphs on a webpage. We ended up using a combination of rrdtool, mailgraph and cacti. Most of the testing that resulted in this page was performed on ubuntu and then ported to gentoo.

• RRDTool
  • Create an rrd
  • Update an rrd
  • Graph an rrd
• Mailgraph
• Cacti
• Collectd
• Project Woes
• Testing Links
  • Test Machine - RRDTool Help
  • Test Machine - Mail Statistics for Test Machine
  • Test Machine - Cacti statistics for Test machine

RRDTool Usage

Create an rrd

1. Create an rrd called, "datafile.rrd"
2. This rrd data has one Data Source(DS), called "packets", but additional data sources could be specified.
   "ABSOLUTE" refers to the data. Can also use options such as COMPUTE, COUNTER, DERIVE or GAUGE
   "900" sets the update period or heartbeat to 900 seconds (15 minutes)
   "0:10000000" defines the minimum and maximum valid value that can be entered.
3. This is the first of three Round Robin Archives(RRA) inside the rdd.
   "AVERAGE" stores the average of the data points stored. MIN, MAX and LAST are also options.
   "0.5" defines the xfiles factor(xff) and can be set from 0 to 1.
   "1" sets the steps to 1 or the actual data point every 15 minutes.
   "9600" sets the number of values or rows to store. 9600 rows at 15 minute intervals is 100 days of data.
   
4. The second RRA in our RRD. Only it stores the average of 4 reporting periods (1 hour) for 400 days at 1-hour accuracy.
5. The third and final RRA in our RRD. It stores the average of 24 reporting periods (6 hours) for 6000 periods for 1500 days of accuracy at 6-hour intervals.

Update your rrd

Values must always be entered in chronoligical order. The letter, "N" represents Now to timestamp the entry but, the unix time (seconds since January 1970 UTC) could also be used. Following the timestamp is the data, "123456" which is entered into the first variable in the rrd file (packets). If we had created additional variables in our rrd then they would follow with colons (:) for delimiters. For more information see 'man rrdupdate'.

Create an graph from your rrd

1. Create a graph file named "graph.png"
2. "DEF" defines a variable to be plotted named, "pkt", on the graph. "datafile.rrd:packets" defines the source rrd file and the data in the rrd file. "AVERAGE" is the consolidation method - MIN, MAX and LAST are also options. If using multiple rrd files as sources simply include more DEF lines.
3. "LINE1" is the type of graph to be drawn. Other options are LINE2, LINE3, AREA, STACK, etcetera.
   "pkt" is the variable in the graph to be plotted as defined in the DEF statement.
   "#ff0000" is the color to use (red).
   "Packets" is the name to use in hte graph's legend.

View rrd data/info

Links

• oss.oetiker.ch - RRDTool Site
• ds9a.nl - Simple rdrdtool how-to
• mailgraph.schweikert.ch - Mailgraph site
• arschkrebs.de - Queuegraph Site
• howtoforge.com - Postfix Monitoring With Mailgraph And pflogsumm
• old.nabble.com - RRDTool - Mailgraph: postfix + clamav-milter
• khmerict.com - Linux Howto-Postfix Mailgraph
• cacti.net - The complete RRDTool-based graphing solution (Cacti)

MailGraph

Mailgraph is a simple mail statistics RRDtool frontend in perl for Postfix and Sendmail that produces daily, weekly, monthly and yearly graphs of received/sent and bounced/rejected mail.

If you have integrated a content filter like amavisd (for spam and virus scanning) into Postfix like we do, then answer "No" to the "Count incoming mail as outgoing mail?" question during installation. This will prevent Mailgraph from counting email twice (because Postfix delivers emails to amavisd which then - after successful scanning - delivers the mails back to Postfix). If you don't use a content filter, then answer "Yes".

We are using a central loghost for all servers the path to the log files had to be updated in /etc/default/mailgraph. /usr/lib/cgi-bin/mailgraph.cgi needed to be updated with the correct hostname.

As I was testing on a VM instead of the log server and initially rsynced (1) over the logs periodically. Later I mounted it with sshfs (2) which provided much better results in cacti: (1) rsync -avz -e ssh user@server:/var/log/mail.log /var/log/server/ (2) sshfs server:/var/log /var/log/server (unmount with: fusermount -u /var/log/server)

Mailgraph was almost perfect and the webpage showing the mail graphs, /usr/lib/cgi-bin/mailgraph.cgi, could have been very easily customized to blend in with our existing managment pages but, for versatility, we decided to show this data in cacti.

Mailgraph with Multiple Log File Support

Based in large on Jacob Emcken's modifications to support multiple log files in mailgraph

1. Add a VIRUS_LOG definition to /etc/default/mailgraph:VIRUS_LOG=/var/logs/saffron/mail.log Note:The variable definitions are all in /etc/conf.d/mailgraph on gentoo as opposed to being split between the conf file and the init script on ubuntu
   
   
2. Update the mailgraph init script at /etc/init.d/mailgraph (changes are in red): #!/bin/sh MAILGRAPH_CONFIG="/etc/default/mailgraph" NAME="mailgraph" DAEMON="/usr/sbin/mailgraph" PID_FILE="/var/run/mailgraph.pid" PID_VIRUS_FILE="/var/run/mailgraph_virus.pid" RRD_DIR="/var/lib/mailgraph" IGNORE_OPTION="" [...] case "$1" in start) echo -n "Starting Postfix Mail Statistics: $NAME" if [ -f $VIRUS_LOG ]; then start-stop-daemon -S -q -b -p $PID_FILE -x $DAEMON -- --daemon-pid=$PID_FILE \ --only-mail-rrd -l $MAIL_LOG -d --daemon_rrd=$RRD_DIR $IGNORE_OPTION start-stop-daemon -S -q -b -p $PID_VIRUS_FILE -x $DAEMON -- --daemon-pid=$PID_VIRUS_FILE \ --only-virus-rrd -l $VIRUS_LOG -d --daemon_rrd=$RRD_DIR $IGNORE_OPTION else start-stop-daemon -S -q -b -p $PID_FILE -x $DAEMON -- -l $MAIL_LOG -d \ --daemon_rrd=$RRD_DIR $IGNORE_OPTION fi echo "." ;; stop) echo -n "Stopping Postfix Mail Statistics: $NAME" if [ -f $PID_FILE ]; then kill `cat $PID_FILE` rm $PID_FILE fi if [ -f $PID_VIRUS_FILE ]; then kill `cat $PID_VIRUS_FILE` rm $PID_VIRUS_FILE fi echo "." ;; [...]
3. Edit the process_line function in /usr/sbin/mailgraph to support clamav lines: elsif($prog eq 'clamd') { if($text =~ /.* FOUND$/) {
event($time, 'virus');
}
}
I also added some tweaks for amavisd-mx, amavisd-milter[-d], sophie, etc.

For details on setting up mailgraph specific to Gentoo click here.

Links

• mailgraph.schweikert.ch - Mailgraph homepage
• forums.cacti.net - Postfix monitoring with David Schweikert's Mailgraph?
• pineapple.com.hk - Monitoring email server with Cacti
• teodorarafailovic.com/files/Cacti/

Cacti

Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. We would like the mailgraph data to be available from cacti.

1. SNMP Method to add Mailgraph data to Cacti
2. Configure Cacti to read Mailgraph RRDs Directly
3. Cacti Links

1. Importing Mailgraph data into Cacti via SNMP

Initially, the easiest way to insert Mailgraph data into cacti that I found was with Curu Wong's Cacti graph template for mail server monitoring * via SNMP. Configuration instructions are here but, here's the highlights:

Then add the following two lines to /etc/snmp/snmpd.conf, make sure that it is the first exec rule in the file:

Note: You need snmpd and Net-SNMP perl module for this:

Finally, import the cacti_graph_template_postfix_[...].xml templates via Console > Import Templates on the cacti web interface^w3,4.

Adding Multiple Log File Support to SNMP Method

If you enabled multiple log file support in mailgraph (above) then you'll see that your cacti graphs starting to show alot of hard peaks and valleys but, the mailgraph page appears correct. This is because you are running two instances of the mailgraph daemon for viruses and mail data. Curu Wong's modifications re-write /var/tmp/mailstat with only the data from each rrd.

1. Edit /usr/share/cacti/site/scripts/mail_stat.pl to simply read the exisitng mailgraph rrd files and report the correct data. As all this is running on the local machine it would also allow me to bypass SNMP completely.
2. Edit the update_stat function in /usr/sbin/mailgraph to correctly handle the data (line 910-ish). Here's a quick and dirty example: sub update_stat()
{ my $stat_file = '/var/tmp/mailstat'; open (my $stath, "+<", $stat_file) or die "unable to open $stat_file to write mail statistic $!"; flock($stath, 2); my $line=<$stath>; chomp($line); my %old_stat = (); %old_stat =split /:| /, $line; if ( $opt{'only-virus-rrd'} ) { $sum_stat{sent} = $old_stat{sent}; $sum_stat{received} = $old_stat{received}; $sum_stat{bounced} = $old_stat{bounced}; $sum_stat{rejected} = $old_stat{rejected}; } if ( $opt{'only-mail-rrd'} ) { $sum_stat{virus} = $old_stat{virus}; $sum_stat{spam} = $old_stat{spam}; } seek($stath, 0, SEEK_SET); # Move to beginning of file print $stath "sent:$sum_stat{sent} received:$sum_stat{received} bounced:$sum_stat{bounced} [...]\n"; #truncate($stath, newSizeHere); flock($stath, 8); close($stath); return 1; }

2. Importing Mailgraph RRD Directly into Cacti

The SNMP method described above has some uses, but I would prefer to use the mailgraph rrds directly in Cacti. A method to do so is described in detail here. This method requires much less hacking and is more stable on my system.

3. Cacti Links

• cacti.net - Cacti homepage
• cacti.net - Cacti Scripts
• cacti.net - Cacti Manual: templates
• forums.cacti.net - Monitoring for Postfix
• forums.cacti.net - Postfix monitoring with David Schweikert's Mailgraph?
• forums.gentoo.org - Postfix stats?
• generationip.com - Best Cacti Templates for Application
• netexpertise.eu - Central Logging with Cacti

Collectd

collectd is a daemon which collects system performance statistics periodically and provides mechanisms to store the values in a variety of ways, for example in RRD files.

2. Collectd Links

• collectd.org - Collectd homepage

Woes

"0.8.7e" => "0019"
"0.8.7f" => "0020"
"0.8.7g" => "0021"