How much space is needed to prune

danrest · January 30, 2018, 7:58pm

Hello,

I use restic 0.8.1 with rest-server as backup destination.

I try to recover backup, which wasn’t completed due lack of space on destination. I had to unlock the destination, rebuild-index and start new backup, this seems be working quite correct. Only the first backup after repository fix runned much longer than regular backups.

Now I try to run prune to remove unneeded files (I didn’t use the forget) and it says:

repository contains 954787 packs (8928751 blobs) with 4.076 TiB
processed 8928751 blobs: 4521593 duplicate blobs, 2.121 TiB duplicate
load all snapshots
find data that is still in use for 26 snapshots
[6:40] 100.00% 26 / 26 snapshots
found 4398972 of 8928751 data blobs still in use, removing 4529779 blobs
will remove 0 invalid files
will delete 1 packs and rewrite 923625 packs, this frees 2.125 TiB

but after performing about 10% of the prune I see, that it is eating more and more space, about 400GB for the 10% of prune. Is it expected, that the command need in fact double the storage needed for backup? It looks like the prune will copy all data to the new blobs and then remove the old ones?

I am also performing the maintenance (rebuild-index, prune) locally on backup server with rest-server process stopped. I hope this isn’t any problem.

fd0 · February 5, 2018, 7:37pm

Running some operations locally on the server is not an issue. What’s happening here is that restic repacks files, so that data that is still needed is loaded and written to new pack files. For safety reasons, restic will only ever delete data at the very end of the process, so while the process is still running the repo will grow.

I have a branch ready that runs a more aggressive prune, but that may not be safe and is experimental.

danrest · February 5, 2018, 7:58pm

Thank you for reply.

I can try aggressive prune, because the repository is lost. I allocated already all available space to them, but I am not able to finish the prune operation. So the only option is to start with a fresh one. And it’s not the only backup, so I am not afraid to lost them.

fd0 · February 6, 2018, 7:57am

You can try the branch prune-aggressive, clone the repo, run git checkout prune-aggressive, then compile restic as usual: go run build.go. Then run restic prune as usual. When you abort prune, the repo will be in a non-consistent state, you can fix that by running restic rebuild-index.

It will then first remove all pack files that are completely unneeded (this gives you some space) and then starts repacking things. Maybe that’ll help. I’m working on a long-term solution for that, but it’s not ready yet.

Please report back if that restored the repo!

danrest · February 7, 2018, 9:19am

Hello,

thank you for the suggestion, but the action isn’t successful. The prune-aggressive branch ignores stale lock on the repository, but other ways work as the normal one. The log is:

storage ID fc9a52c8
counting files in repo
building new index for repo
[1:10:42] 100.00% 1245383 / 1245383 packs
repository contains 1245383 packs (11608689 blobs) with 5.331 TiB
processed 11608689 blobs: 7201532 duplicate blobs, 3.375 TiB duplicate
load all snapshots
find data that is still in use for 26 snapshots
[7:04] 100.00% 26 / 26 snapshots
found 4398972 of 11608689 data blobs still in use, removing 7209717 blobs
will remove 0 invalid files
will delete 0 packs and rewrite 1214222 packs, this frees 3.379 TiB
Save(<data/f716bb31ca>) returned error, retrying after 520.700203ms: Write: write /mnt/restic.old/server/data/f7/f716bb31caa50e14bf9beb43a2eb6cc1c7c38e43161a97689300a6dc50087917: no space left on device

and crashes. I tried to add about 10GB space to the repo, but this isn’t enough.

rawtaz · February 13, 2018, 9:05pm

That’s odd. Are you entirely sure that you built restic from scratch on that special branch, and also that you really ran that new restic binary and not some other one?

fd0 · February 14, 2018, 8:32pm

It’s possible that restic behaves this way: The aggressive-prune branch will only remove files earlier that are completely unneeded, so they do not contain a single blob that is still referenced. It may happen that no such files exist…

Mebus · October 28, 2018, 6:59pm

Hey,

now I need the prune-aggressive branch again

Would it be possible to rebase the prune-aggressive branch onto a newer version of restic or include the feature in the standard version? The branch is already quite old.

Mebus

moritzdietz · October 28, 2018, 8:02pm

I think you would be able to fork the repo and apply the single commit yourself quicker than fd0 rebasing it onto the current master if I am being honest

Mebus · October 28, 2018, 8:38pm

Theoretically, yes. But I’m not familiar enough with the code, to know if that won’t brake my repo

fd0 already told me that he had rebased it recently.

Mebus

olc · January 7, 2019, 1:15pm

Hi!

I’m in the same situation: lack of available space in order to perform the prune. I decided to give prune-aggressive a chance, it seems to be OK for now:

# run-restic.sh prune
repository f61db6ba opened successfully, password is correct
counting files in repo
building new index for repo
[4:01:26] 100.00%  2256424 / 2256424 packs
repository contains 2256424 packs (79874147 blobs) with 10.094 TiB
processed 79874147 blobs: 13891697 duplicate blobs, 1.760 TiB duplicate
load all snapshots
find data that is still in use for 17 snapshots
[1:55:21] 100.00%  17 / 17 snapshots
found 24423103 of 79874147 data blobs still in use, removing 55451044 blobs
will remove 0 invalid files
will delete 502926 packs and rewrite 1362261 packs, this frees 6.699 TiB
[2:22:46] 100.00%  502926 / 502926 packs deleted
[18:06:02] 21.51%  292961 / 1362261 packs rewritten
...

That said, I faced a problem due to the high memory usage of the prune command (both with regular version and prune-aggressive version), so that the restic prune has been oomkilled.

# cat /proc/139267/status
Name:   restic
State:  S (sleeping)
Tgid:   139267
Pid:    139267
PPid:   139266
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
Utrace: 0
FDSize: 256
Groups: 0
VmPeak: 40166480 kB
VmSize: 40166480 kB
VmLck:         0 kB
VmHWM:  30972256 kB
VmRSS:  24122140 kB
VmData: 40154560 kB
VmStk:        88 kB
VmExe:      6088 kB
VmLib:         0 kB
VmPTE:     78240 kB
VmSwap:  3290432 kB
Threads:        54
SigQ:   2/127888
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: fffffffe7fc1feff
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
Cpus_allowed:   ffff,ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list:      0-143
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list:      0-1
voluntary_ctxt_switches:        25294094
nonvoluntary_ctxt_switches:     448718

The server where restic is running on has 32 Gb of RAM.

BTW, this prune run has a lot of job to do because I missed to add the ‘–prune’ option with the ‘forget’ command.

–
Olivier