Backup running forever (24hrs+) after forget/prune

Getting Help
1

Hi all,

Scenario: Backups were running fine, daily for a couple of years. I noticed recently that I was running out of local drive space - turns out I was backing up happily, but never removing old snapshots (anyone need 600+ daily snapshots? not me…), 200Gb of disk space would be nicer.

I ran several forget/prune runs… the first with forget --keep-last 300 --prune which (eventually) seemed to finish fine. The second with forget --keep-weekly 7 --keep-monthly 1 --keep-yearly 1 --prune (from the docs) … though that seems to toss out all but 7 snapshots, not quite what I expected, but finished also.

Now when the nightly cronjob runs, which look like this: 59 01 * * * RESTIC_PASSWORD_FILE="/root/bb-pw.txt" B2_ACCOUNT_ID=XXX B2_ACCOUNT_KEY="YYY" restic -o b2.connections=10 -r b2:insel-:allthespace --verbose backup /mnt/allthespace - keep running for what seems like all day, and don’t finish/send a cronjob email.

Before the forget/pruning, it was finishing in ~1hr (its a 1TB drive, 97% full).

This is a gentoo linux install, current restic: restic 0.17.3 compiled with go1.24.1 on linux/amd64

… Hmm writing this all out suggests to me I should run it manually (instead of via cronjob) and see if/how it fails. Any other thoughts?

2

running manually should work, also you can add > /tmp/resticlog 2>&1 to the command to see what does it say maybe?

1 Like
3

@castaway you report the 1TB drive is still 97% full, that is only around 30GByte of free space. If you have a large backup then restic cache and temporary space to chunk and pack and could exceed that. Also if you do not have enough RAM to run all your apps then the OS would claim swap space.

Agree it is best to run manual first and see what happens.

EDIT: restic cache and tmp are not likely to exceed 30GB.

1 Like
4

Those parameters are redundant when combined with --keep-weekly 7. For the --keep-yearly 1 parameter, restic groups all snapshots by year and for the latest year (== the 1 passed to the option) keeps the latest snapshot. The parameter does NOT mean keep 1 snapshot per year or so. It just keeps 1 yearly backup.

The backup command shows that you’re uploading to B2. I don’t really understand how that matches with your local disk usage (I really hope that the repository cache wasn’t that large).

1 Like
5

I’d agree, except that’s a 2nd drive, .cache is on a different one :slight_smile:

6

Yes its the cache, it was 300Gb+ (on a different drive)

7

Ah… The documentation seems to indicate this is how to use it, maybe I misread it, this section: Removing backup snapshots — restic 0.17.3-dev documentation paragraph starting For example, suppose you make one backup every day for 100 years

8

I was referring to exactly --keep-monthly 1 and --keep-yearly 1 (option + value!). --keep-weekly 7 --keep-monthly 1 --keep-yearly 1 has exactly the same effect as --keep-weekly 7.

1 Like
9

Aha I just kinda discovered what’s going on…

Mar 27 23:11:02 insel kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null
),cpuset=936,mems_allowed=0,global_oom,task_memcg=/945,task=restic,pid=30879,uid
Mar 27 23:11:02 insel kernel: Out of memory: Killed process 30879 (restic) total
-vm:57383504kB, anon-rss:25106696kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables
:100236kB oom_score_adj:0
Mar 27 23:11:04 insel kernel: oom_reaper: reaped process 30879 (restic), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
10

Turns out, was nothing to do with the pruning (unsurprisingly) - it helps if I don’t try and back up a /proc bind (hanging around from a postmarketos chroot) - oops! (This time it took 57minues…)