Overheating -- is there a way to slow down the backup?

I run an sftp backup of /home/$USER over wifi every hour on my Linux laptop, and often when I have multiple screens plugged in and a few windows open, the backup is actually enough to overheat it and shut it down.

I’m trying to figure out why my CPU isn’t throttling enough, but meanwhile is there a way I can slow down the backup? I know of --limit-upload, but I imagine it’s the disk scan that’s computationally intensive, and I don’t think limiting the upload speed would throttle that. Any ideas on how to slow restic down?

Welcome to the forum!

I’m wondering: is there so much data that needs to be read by restic on each invocation? Does so much data change between runs? Maybe restic is run in a way so that it re-reads all data for each invocation, that’d explain the CPU load. Usually, I’d expect that subsequent (“incremental”) backups only read the data that has changed.

You can find out if restic works right by looking for the line using parent snapshot [...] when running a subsequent backup (explanation here).

In order to dig into this further:

  • Can you maybe paste the whole invocation (including the paths you save)?
  • Is the set of paths you save stable? Or do the paths change on each invocation?
  • Is the hostname always the same? (check by running restic snapshots)

On Linux there’s ionice which can throttle the IO bandwidth used by restic, but that’ll only change the priority of the IO relative to other processes. Other than that you could try to set the environment variable GOMAXPROCS=1, which should restrict restic to roughly one CPU core at most.

Here’s the entire restic invocation:

restic -r sftp:kyle@remote:/mnt/restic-bak -p /home/kyle/restic/password.txt --exclude-file=/home/kyle/restic/exclude.txt backup /home/kyle

And here’s the output:

using parent snapshot d7d2fb5c
scan [/home/kyle]
[0:17] 170298 directories, 988816 files, 49.458 GiB
scanned 170298 directories, 988816 files in 0:17
warning for /home/kyle/sensors/temperature.log: expected 69250208 bytes, saved 69250234 bytes
[2:05] 100.00%  49.458 GiB / 49.458 GiB  1159094 / 1159114 items  49 errors  ETA 0:00 

duration: 2:05
snapshot 8b077b4e saved

I am always saving /home/kyle. The hostname in this repository is not always the same because I have multiple computers backing up to a single repository:

kyle@leibniz: ~$ restic -r sftp:kyle@remote:/mnt/restic-bak -p /home/kyle/restic/password.txt snapshots
password is correct
ID        Date                 Host             Tags        Directory
<sections are cut out for brevity>
9027b7ab  2020-09-30 18:00:02  DESKTOP-65472FU              C:\Users\bacon
181b0381  2020-09-30 18:00:22  leibniz                      /home/kyle
6fa60195  2020-10-01 00:10:01  spaceship                    /home/kyle
fffb9bab  2020-10-01 01:10:01  spaceship                    /home/kyle
0c8a3f37  2020-10-01 02:10:01  spaceship                    /home/kyle
3bbfea27  2020-09-30 20:40:23  DESKTOP-65472FU              C:\Users\bacon
6adb907c  2020-09-30 21:00:02  DESKTOP-65472FU              C:\Users\bacon
b75c322c  2020-10-01 03:10:01  spaceship                    /home/kyle
d7d2fb5c  2020-10-02 21:01:43  leibniz                      /home/kyle
3a4c4ec4  2020-10-03 08:10:01  spaceship                    /home/kyle
df43e40a  2020-10-03 02:15:16  marionberry-pi               /home/kyle
9c8e2f74  2020-10-03 11:10:01  spaceship                    /home/kyle
8b077b4e  2020-10-03 06:00:39  leibniz                      /home/kyle
3baf01aa  2020-10-03 12:10:01  spaceship                    /home/kyle
1833 snapshots

For now I think setting GOMAXPROCS makes sense, but based on your questions it does seem that something inefficient is happening here.

Which restic version do you use? The output looks like 0.8.3 or older. Please upgrade to restic 0.10.0 or at least 0.9.6. restic 0.9.0 includes a rewrite of the archiver code, so there’s no point in debugging older versions.

Oh! I had no idea I was that far behind. I’ll see what that does. Will there be any backward compatibility issues moving from restic 0.8.3 to 0.10.0, and moving rest-server from 0.9.7 to 0.10.0? I have other computers backing up through the REST server to the same repository.

The repository format itself is compatible, although tracking of file changes works differently which requires restic to re-read all data again. restic 0.10.0 now exits with status code 3 when a warning occured during the backup.

You could take a quick look at the change summaries at https://github.com/restic/restic/releases or take a look at the release blog posts at https://restic.net/blog/ .

Ok, after updating to 0.10.0 and running it once to allow it to re-read all the data, here’s the output:

Files:         627 new,   738 changed, 480570 unmodified
Dirs:            5 new,   100 changed, 96271 unmodified
Added to the repo: 136.428 MiB

processed 481935 files, 26.097 GiB in 1:37
snapshot 8d71dac8 saved

The restic invocation is the same as before, with the additional prefix GOMAXPROCS=2. With htop I can see restic is running 5 processes. I imagine that’s an implementation detail of the Go runtime.

Half a million files in 97 seconds look like reasonable performance to me. Do you still have the problem with overheating?

Go internally has a lock to ensure that at most GOMAXPROCS threads are running at the same time. When one thread is blocked in a syscall then another one is started to continue processing. So you’ll end up with 2+ threads.

I do still have overheating, but my laptop is also supporting 3 monitors with integrated graphics, so there’s a lot of load on the CPU and the cooling isn’t great. I bet I just need to beef up the exclude file so the whole thing is over sooner.

Thank you so much for your patient help!

Processing of excludes is currently not overly efficient, see for example https://github.com/restic/restic/issues/2694 . A few dozen excludes should be ok, everything beyond that will probably slow down your backup. Did you try setting GOMAXPROCS=1 ? With small number of changes this probably doesn’t make much of a difference, but it might be worth a try if you haven’t done so yet.

Yes, setting GOMAXPROCS keeps it from overheating. Works for me!

You could also consider limiting your CPU’s maximum clock speed, either all the time (if overheating is generally a problem) or only while restic is running.

Yeah, I recently learned from the ArchWiki that my computer has a thermal shutdown problem that can be solved with thermald. So that fixes that!