Using Systemd to monitor server’s processes instead of Monit

systemd. Many developers get shivers down their spine upon hearing/reading the word. Couple of years ago it replaced init scripts in Linux distributions and now it has found its way to almost all the modern ones. I personally feel that the hate it receives is similar to users moving away from SVN to GIT.

monit (https://mmonit.com/monit/), on the other hand, is a handy tool that does server level monitoring (remote or local). It is commonly installed to production servers to make sure that certain daemons are always running in the background. If they crash, Monit will restart them again and notify users. It can also notify users if the space is running low or the CPU is higher than usual.

Setting up nginx to be restarted after a crash

This short guide naturally applies to any systemd service but I’ll use nginx for example purposes.

$ systemctl nginx edit

Make sure that you have your $EDITOR variable set up because this command opens up the default editor with supposedly empty document if you are running the command for the first time.

Add these lines:

[Service]
Restart=always

You can also specify a delay in seconds to restart the service but personally I don’t see any benefits.

RestartSecs=30

Save the document. This will save it to /etc/systemd/system/nginx.service.d/overrides.conf.

Reload the daemons:

$ systemctl daemon-reload

You can test the configuration now by:

$ systemctl start nginx
$ ps -A | grep nginx
$ killall nginx
$ ps -A | grep nginx

…you should see new processes running.

The good stuff

When integrated straight into the system-level daemon management, the restarts happen immediately when the processes crash. Monit uses cron to check things every one or five minutes. This also means that Systemd will not take any actions when the user shuts the daemon down manually whereas Monit will quickly start it again and notify users. There is a workaround for this by running $ monit service stop nginx. Systemd will also make sure that all the required services are running also so the hierarchy is a plus.

Not perfect, far from it

Monit is handy because you can actually monitor URLs (http://example.com/status) that the service is up and responding. Also pinging some addresses and ports for an appropriate answer works. This might be possible with Systemd but at least it requires more digging. For example having a shell script that does the checking and Systemd could use that for service status. Also setting up a notification system will be a hassle (see OnFailure) whereas this comes out-of-the-box in Monit. I wouldn’t be surprised if someone had made a generic unit for this purpose already.

Conclusions

All in all, using Systemd instead of Monit will probably eventually kill the need of third-party service monitoring on servers. But we are not there yet. It’s awesome to see that service process management ships with Systemd.

Also, one of the reasons why I wrote the article was that currently there is no reasonable documentation on doing this. You’ll basically have to dig trough ArchLinux systemd wiki page which isn’t the most convenient way.

Thoughts by