Restic running extremely slowly from systemd service

I just solved a problem that had been frustrating me for a while as I looked in all the wrong places, so I thought I would share the solution and hopefully include enough detail that others with this problem will find the solution.

When I run my backup script from the command line, it finishes in less than a minute. Running it from systemd, it was taking over an hour; two orders of magnitude slower. Yet it was clear from strace that it was making slow-but-steady progress.

Barking up the wrong tree, I looked for AVC errors, because I wondered if SELinux was blocking something that was causing a timeout or slow path. I finally collected stderr while it was running and saw this which put me on the right path:

unable to open cache: unable to locate cache directory: neither $XDG_CACHE_HOME nor $HOME are defined

I updated my service file and now it’s just as fast as the command line.

# cat /etc/systemd/system/mastodon-backup.service
[Unit]
Description=Mastodon - backup service
After=mastodon.service

[Service]
Type=oneshot
Environment="HOME=/root"
StandardError=file:/var/log/mastodon-backup.err
StandardOutput=file:/var/log/mastodon-backup.log

WorkingDirectory=/opt/mastodon
ExecStart=/bin/bash /opt/mastodon/mastodon-backup

[Install]
WantedBy=multi-user.target

The context for this service file, including the timer file that fires it off and the script it calls, is described here:

Thanks so much for restic! I’ve been using rdiff-backup personally for many years, and I’ve been deeply grateful for it, but the requirement to run the same version on both ends of the connection means that I ended up with three different versions running on one backup target system. Having gotten started with restic, I’m now in the process of moving everything over. :tada:

2 Likes

Replying to myself only to mark that the OP has the solution. :smiley:

One small nitpick. If the only issue is cache directory, you can also use --cache-dir parameter explicitly to restic command, instead of relying auto-resolution from HOME variable.

Even better! I’d missed that option, thank you!

Also, instead of setting HOME to an explicit path, you might generally prefer %h which gets resolved by systemd for that user running the service.

Especially for services with privacy enabled it will also ensure that the folder are created appropriately and have correct permissions set.

Yeah, but I can’t edit the OP, so I can’t actually change it to be what I actually changed to.

I quit using $HOME and moved to using --cache-dir and pointing to a directory under /var for the cache dir.

1 Like