systemd resource limiting for mortals

Motivation

I’ve recently gotten completely fed-up with my backup process (using duplicity) in the way it gobbles up huge amounts of memory (6GB+ out of 8GB physical memory) when doing its daily backup. This annoyance is amplified by code in the display driver that bails out when there is intense memory pressure and exits out of my Wayland session (likely bug).  I’m documenting my solution for Fedora 26, as I haven’t found trivially-understandable articles elsewhere.

Context

Duplicity is a quite a nice program, and I am quite thankful for its development. While I am unhappy with the way its resource usage scales with 200GB backups and limited available knowledge on best tunings for various scales greater and smaller, I have not found any superior backup solution. I just need to do something about its ability to gobble memory undeterred by the Linux kernel.

Intel video hardware is probably the best low-power graphics with “reasonable” Linux support in its open-source drivers, but unfortunately, some debilitating bugs are being triggered with  the recent Wayland migration. Most recently, its eagerness to bail out (and kill the user session) under memory pressure has been a concern.

Resource management tools have been available on Linux (and other POSIX-style OSes) for quite some time, so why can’t you just do a “nice -n “19 and be done? Some notes suggest that systemd may be of help.

Solution

TL;DR: We need to create a systemd slice and put the backup process systemd unit into that slice.

Create a slice by creating the file in /etc/systemd/system/. Here’s what mine looks like:

$ cat /etc/systemd/system/duplicity.slice 
[Unit]
Description=Duplicity resource-limited slice
Before=slices.target

[Slice]
MemoryAccounting=true
MemoryHigh=2G

I’ve set MemoryHigh to 2G, which is supposed to mean that systemd will start preferentially swapping pages out from the slice, should tasks inside that slice use more thatn 2GB physical memory. Other pages suggested setting  MemoryMax, but as I still wanted my backup to complete, and a re-run of the backup process might even demand more, I did not want the MemoryMax behavior of “kill processes with OOM if the max is reached”. MemoryAccounting=true is a necessary prerequisite for any memory tracking controls in that slice.

With the slice file created, then I just needed to add:

Slice=duplicity.slice

to the corresponding duplicity.service file. While I was there, I also added (Nice=19, IOSchedulingClass=3, IOSchedulingPriority=7).

To apply the changes:

$ systemctl daemon-reload

When the task is running, you can verify that it’s running in your new slice.

$ systemd-cgls # list running services by cgroup slice

$ systemd-cgtop # list running services sorted by resource usage

So far, the resource limiting has prevented the abrupt session-exits that used to consistently coincide with the backup trigger, but I will update this if I find anything new.

References

  • The necessity of creating a slice was only clear after reading through a particular systemd issue, which thankfully included a concise example. If this issue were more findable to people looking for a HOWTO, I may not have bothered with this post.

About Daniel

I write distributed database software. Coding is fun. Love learning languages (spoken and computer). Always looking for opportunities to use advanced math in work and daily life. github: wangd
This entry was posted in development. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *