Monitoring postfix with Zabbix (LTS version 3.0)

As we’ve built new monitoring system based on Zabbix LTS version 3.0 I also was looking for solution how to monitor postfix servers. I’ve found few articles regarding this issue but they were either outdated or not very effective.

I decided to take the best from what I’ve found but build an own strategy.

The whole solution consists of several different parts:

• pygtail software
  This one reads the lines of mail log (this tools is written in Python and works great on Centos 6 and Centos 7. It doesn’t run on Centos 5 due to older Python libraries. You can download my spec file for Centos 6/7 here if you want to create an RPM package,
• pflogsumm script
  This script is part of postfix-perl-scripts package on Centos 6 and 7 and can read the Postfix logs and create nice summary. We’d use this one to get the summary and parse it for interesting values to be further read by Zabbix
• Zabbix temlate for Postfix monitoring
• create userparameter_postfix.conf file under /etc/zabbix/zabbix_agent.d with this content (you also have to enable remote commands to be run on Zabbix clients by adding enableEnableRemoteCommands=1 to main config file /etc/zabbix/zabbix_agentd.conf):

  # Postfix user parameter
  UserParameter=postfix.pfmailq,mailq | grep -v "Mail queue is empty" | grep -c '^[0-9A-Z]'
  UserParameter=postfix[*],sudo /usr/local/bin/zabbix-postfix-stats.sh $1
  UserParameter=postfix.update_data,sudo /usr/local/bin/zabbix-postfix-stats.sh
  

• we need to add this line to /etc/sudoers to grant zabbix user access to run the main script with root permissions (otherwise zabbix user would need access to read mail logs):

  zabbix  ALL=NOPASSWD: /usr/local/bin/zabbix-postfix-stats.sh *
  

• simple bash script which uses pygtail and pflogsumm and gets the interesting data from Postfix log to by further read by Zabbix:

  #!/usr/bin/env bash
  
  MAILLOG=/var/log/maillog
  PFOFFSETFILE=/tmp/zabbix-postfix-offset.dat
  PFSTATSFILE=/tmp/postfix_statsfile.dat
  TEMPFILE=$(mktemp)
  PFLOGSUMM=/usr/sbin/pflogsumm
  PYGTAIL=/usr/bin/pygtail
  
  # list of values we are interested in
  PFVALS=( 'received' 'delivered' 'forwarded' 'deferred' 'bounced' 'rejected' 'held' 'discarded' 'reject_warnings' 'bytes_received' 'bytes_delivered' )
  
  # write result of running this script
  write_result () {
  
          echo "$2"
          exit $1
  
  }
  
  # check for binaries we need to run the script
  if [ ! -x ${PFLOGSUMM} ] ; then
          echo "ERROR: ${PFLOGSUMM} not found"
          exit 1
  fi
  
  if [ ! -x ${PYGTAIL} ] ; then
          echo "ERROR: ${PYGTAIL} not found"
          exit 1
  fi
  
  if [ ! -r ${MAILLOG} ] ; then
          echo "ERROR: ${MAILLOG} not readable"
          exit 1
  fi
  
  
  # check whether file exists and the write permissions are granted
  if [ ! -w "${PFSTATSFILE}" ]; then
          touch "${PFSTATSFILE}" && chown zabbix:zabbix "${PFSTATSFILE}" > /dev/null 2>&1
  
          if [ ! $? -eq 0 ]; then
                  result_text="ERROR: wrong exit code returned while creating file ${PFSTATSFILE} and setting its owner to zaabbix:zabbix"
                  result_code="1"
                  write_result "${result_code}" "${result_text}"
          fi
  fi
  
  # read specific value from data file and print it
  readvalue () {
          local $key
          key=$(echo ${PFVALS[@]} | grep -wo $1)
          if [ -n "${key}" ]; then
                  value=$(grep -e "^${key};" "${PFSTATSFILE}" | cut -d ";" -f2)
                  echo "${value}"
  
          else
                  rm "${TEMPFILE}"
                  result_text="ERROR: could not get value \"$1\" from ${PFSTATSFILE}"
                  result_code="1"
                  write_result "${result_code}" "${result_text}"
          fi
  }
  
  # update value  in data file
  updatevalue() {
          local $key
          local $pfkey
  
          key=$1
          pfkey=$(echo "$1" | tr '_' ' ')
  
          # convert value to bytes
          value=$(grep -m 1 "$pfkey" $TEMPFILE | awk '{print $1}' | awk '/k|m/{p = /k/?1:2}{printf "%d\n", int($1) * 1024 ^ p}')
  
          # update values in data file
          old_value=$(grep -e "^${key};" "${PFSTATSFILE}" | cut -d ";" -f2)
          if [ -n "${old_value}" ]; then
                  sed -i -e "s/^${key};${old_value}/${key};$((${old_value}+${value}))/" "${PFSTATSFILE}"
          else
                  echo "${key};${value}" >> "${PFSTATSFILE}"
  
          fi
  }
  
  # is there a requests for specific value or do we update all values ?
  if [ -n "$1" ]; then
          readvalue "$1"
  else
          # read the new part of mail log and read it with pflogsumm to get the summary
          "${PYGTAIL}" -o"${PFOFFSETFILE}" "${MAILLOG}" | "${PFLOGSUMM}" -h 0 -u 0 --no_bounce_detail --no_deferral_detail --no_reject_detail --no_smtpd_warnings --no_no_msg_size > "${TEMPFILE}" 2>/dev/null
  
          if [ ! $? -eq 0 ]; then
                  result_text="ERROR: wrong exit code returned while running  \"${PYGTAIL}\" -o\"${PFOFFSETFILE}\" \"${MAILLOG}\" | \"${PFLOGSUMM}\" -h 0 -u 0 --no_bounce_detail --no_deferral_detail --no_reject_detail --no_smtpd_warnings --no_no_msg_size > \"${TEMPFILE}\" 2>/dev/null"
                  result_code="1"
                  write_result "${result_code}" "${result_text}"
          fi
  
          # update all values from pflogsumm summary
          for i in "${PFVALS[@]}"; do
                  updatevalue "$i"
          done
  
          result_text="OK: statistics updated"
          result_code="0"
          write_result "${result_code}" "${result_text}"
  fi
  
  rm "${TEMPFILE}"
  
  

This is a summary how the whole thing works:

1. All items from Zabbix postfix template refer to user parameters defined in userparameter_postfix.conf file. These are read on zabbix client and triggers the launch of the defined bash script zabbix-postfix-stats.sh
2. There is a item called ‘Postfix data update request’ in the template which is run every minute. This one calls the script zabbix-postfix-stats.sh without any arguments which makes the script to update all the statistics from the Postfix mail log file. To parse the log file we need both pygtail and pflogsumm tools – both mentioned at the beginning.
3. The rest of template items are not run so often and causes the script zabbix-postfix-stats.sh to be run with a specific argument depending on item type. If the postfix statistics were already made the value which is requested by each item is returned.

Posted in Monitoring