Zimbra 8, Ubuntu 12 LTS, OpenVZ – zmstat io and the Infinite Loop

I checked the zimbra installation tonight and found that the entire hard drive had been filled up, which caused zimbra to stop working. First step was to find the offending files, so I ran the command:

sudo du -h / | grep ^[0-9.]*G

This quickly revealed a file that took up 100+ GB, this single file was the following:

/opt/zimbra/zmstat/zmstat.out

The file contained the same error repeated in the entirety of  the 100+ GB file.

Use of uninitialized value $line in pattern match (m//) at /opt/zimbra/libexec/zmstat-io line 69.

A quick Google search turned up the following post on Zimbra forums with my exact problem. In summary, PROXMOX openvz containers have problems with iostat in Ubuntu 12.04 LTS, which results in an infinite loop that fills up the zmstat.out file . The solution is to downgrade iostat, a program that runs in the sysstat package. The post then proceeded to suggest extracting the older version from a debian package. I tried that, but was unsuccessful in getting the source to compile.

The Debian package that was linked was to sysstat 9.0.6.1. The quicker [and easier] alternative was to just download the precompiled package and install it.

I found the .deb package here.

Installing it was done via the following:

wget http://archive.ubuntu.com/ubuntu/pool/main/s/sysstat/sysstat_9.0.6-2_amd64.deb
dpkg -i *.deb

After restarting zimbra services, the file no longer filled up with the same error. Note: You can delete zmstat.out and recreate it if needed. Just make sure you set the owner to zimbra:zimbra after doing so.


I also ran into this Knowledge Base article on VMware’s site – which said to do the following:

THIS DID NOT WORK

To resolve this issue:

  1. Open these files in a plain text editor:/opt/zimbra/libexec/zmstat-allprocs
    /opt/zimbra/libexec/zmstat-nginx
  2. In both files, locate this line:$HAS_IO_ACCT = /proc/self/io
  3. In both files, change $HAS_IO_ACCT = /proc/self/io to:$HAS_IO_ACCT = /proc/self/no_io
  4. Save and close both files.
  5. Run this command to make the changes take effect:zmstatctl restart

UPDATE

Easiest way to fix this is to disable the stats service. Not really a fix if you need the stats service to update.

zmstatctl stop

Update #2

I noticed if the server reboots, zmstat starts up again. You can disable the service by logging in as the user zimbra and then editing crontab to stop the service.

15 * * * * /opt/zimbra/bin/zmstatctl stop