Restic not obeying --max-unused parameter on prune

Getting Help
1

Hi all,

Not really sure why but Restic doesn’t seem to be pruning my repo according to the max-unused parameter I am using.

Background

Repo has about 809Gb of used data and about 200GB of unused data. As a full max-unused 0 prune takes too long to complete (and causes me issues with temporary scratch space) - instead I slowly prune down 5GB at a time — it was going fine till I had about 105GB left and now Restic doesn’t seem to prune down anymore. Despite having the unused data in repo, when I give it the stepdown prune command, it just doesn’t do anything and I am in the same position as before running.

What is going on?

• The output of restic version

restic 0.17.1 compiled with go1.23.1 on linux/amd64

• The complete commands that you ran (leading up to the problem or to reproduce the problem).

for i in $(seq 105 -5 5); do
	echo Remaining $i GB
	prune -r $TARGET --max-unused $i\G
done

• The complete output of those commands (except any repeated output when obvious it’s not needed for debugging).

Remaining 105GB
repository 241d8bc5 opened (version 2, compression level auto)
loading indexes...
[0:12] 100.00%  18 / 18 index files loaded
loading all snapshots...
finding data that is still in use for 20 snapshots
[0:16] 100.00%  20 / 20 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:04] 100.00%  46722 / 46722 packs processed

to repack:             0 blobs / 0 B
this removes:          0 blobs / 0 B
to delete:             0 blobs / 0 B
total prune:           0 blobs / 0 B
remaining:        889547 blobs / 916.972 GiB
unused size after prune: 105.905 GiB (11.55% of remaining size)

done

(same output for --max-unused 100G and 95G, etc etc )

2

It always worked for me, so here are just some hint for further try and error:

  • You mean restic prune instead of prune in your for loop, don’t you?
  • --max-unused $i\G with a backslash may be correct but how about --max-unused ${i}G?
  • What does --max-unused 9% (in % rather than G) do?
  • You can also use --max-unused 1G --max-repack-size 5G to only prune 5 GiB towards the goal of having only 1 GiB of unused space.
  • Just speculating: could it be that you have very large files and you can’t just delete some packs. It would need to repack more than 5 GiB to reach a new consistent state?
  • Can you try with --dry-run? Like --max-unused 100M --dry-run.
3

The repo is my home photo/movie collection so a mix of small / big files but nothing close to 1GB in size.

The percentage approach doesn’t work either (I actually had started with that and when it stopped working went to the fixed GB appraoch)

Before I did both of these - I was actually using max-repack-size 2G on a loop to slowly reduce down the unused space — it was only when that stopped I relooked at the max-unused to force the issue but still not working

I experimented with dry-run a bit and found:

  • At 0 it would repack 367GB to remove the entire unused 105GB (but I am reluctant to do this coz it is almost half my entire report and means a really long (potentially multiday??) process)
  • Dropping down even further below 1G — when I feed max-unused 100M and successively lower inputs - still get nothing to do until 6M and then it gets interesting:

Input // Repack
6 // 364,1 GB
5 // 3.8MB (a small tiny bit!)
4 // 367.5GB
3 // 364.9GB
2 // 366.6GB
1 // 364.6GB
0 // 367.3GB

Its weird things don’t happen until input is 6M and below and then drop-then-bounce at 5M input and with all other inputs having slightly different amounts to repack (but ending at same end state - no unused space)

Anyway - I am going to try 5M and see if it mixes things up enough make restic prune work normally again

4

Hmm - looked like the 5M dry-run output was illusion - when I did run for real - suddenly it wanted to repack the ~365GB which is not workable for me and I terminated the process.

At a loss what to do now; I might start just deleting some less important snapshots to see if can force restic to reevaluate the prune

5

Are you using --repack-cacheable-only ?

Try a restic check ?

6

I think there is something similar going on my repo:

# restic prune --dry-run
repository 559ce728 opened (version 2, compression level auto)
loading indexes…
[0:10] 100.00% 62 / 62 index files loaded
loading all snapshots…
finding data that is still in use for 247 snapshots
[1:59] 100.00% 247 / 247 snapshots
searching used packs…
collecting packs for deletion and repacking
[0:03] 100.00% 200895 / 200895 packs processed

Would have made the following changes:

to repack: 513134 blobs / 8.339 GiB
this removes: 118942 blobs / 7.705 GiB
to delete: 0 blobs / 0 B
total prune: 118942 blobs / 7.705 GiB
remaining: 4768630 blobs / 3.226 TiB
unused size after prune: 913.420 GiB (27.65% of remaining size)

# restic prune --dry-run --max-unused 500G
repository 559ce728 opened (version 2, compression level auto)
loading indexes…
[0:11] 100.00% 62 / 62 index files loaded
loading all snapshots…
finding data that is still in use for 247 snapshots
[1:34] 100.00% 247 / 247 snapshots
searching used packs…
collecting packs for deletion and repacking
[0:01] 100.00% 200895 / 200895 packs processed

Would have made the following changes:

to repack: 507510 blobs / 1.295 GiB
this removes: 113635 blobs / 675.790 MiB
to delete: 0 blobs / 0 B
total prune: 113635 blobs / 675.790 MiB
remaining: 4773937 blobs / 3.232 TiB
unused size after prune: 920.465 GiB (27.81% of remaining size)

# restic prune --dry-run --max-unused 0G
repository 559ce728 opened (version 2, compression level auto)
loading indexes…
[0:10] 100.00% 62 / 62 index files loaded
loading all snapshots…
finding data that is still in use for 247 snapshots
[0:56] 100.00% 247 / 247 snapshots
searching used packs…
collecting packs for deletion and repacking
[0:00] 100.00% 200895 / 200895 packs processed

Would have made the following changes:

to repack: 2608254 blobs / 1.182 TiB
this removes: 1335976 blobs / 921.125 GiB
to delete: 0 blobs / 0 B
total prune: 1335976 blobs / 921.125 GiB
remaining: 3551596 blobs / 2.334 TiB
unused size after prune: 0 B (0.00% of remaining size)

I never had issues with pruning. I generally prune with --max-repack-size 100G and the --max-unused unspecified (so the default 5%) and it generally works, so I suspect there is something wrong with the pruning plan, triggered by something in the repository, perhaps how the data was deleted in the past forget/prune cycles.

The workaround I think is simply run a prune --max-unused 0G --max-repack-size 100G or whatever value you can handle in a single run.

7

I am kinda curious what is causing all of this; but unfortunately can’t be much help anymore since I bit the bullet and did the full prune/repack… but just to add - the max-repack-size doesn’t work as I was using that initially (5GB at a time) until got stuck and that is when I started trying the max unused as alternative (which didn’t work either)

PS - thats one hugggeee repo you got there with a lot of unused; biting the bullet and doing a full prune is going to be painful without a fast backend!!!

8

I still have the pruning issue with the repository, but I cannot understand the code. If someone is interested in troubleshooting the issue I can help with tests.

I suppose this is some kind of corner case (perhaps due to the data placement in the partially used blobs), so I understand developers could have something more useful to do. :smile:

Since this is costing 1.5TB of additional data i can wait for a bit if needed, otherwise I’ll just prune with max-unused 0G and forget about it… just let me know.

Radel

9

My guess would be that it is related to large amounts of duplicated blobs. But without digging further into the code, I can’t say anything specific. This might take a few days until I get around to taking a look.

10

Ok. I have an (almost) inexpensive copy of the repo with this defect, so I can wait and try a fix.

11

what does prune --dry-run --verbose=2 --max-unused $something (replace $something with a value that results in unexpected behavior) print? I’d expect the output to contain a line starting with duplicates:.

1 Like
12

I think I’ve found the problem: prune: correctly account for duplicates in max-unused check by MichaelEischer · Pull Request #5212 · restic/restic · GitHub

3 Likes
13

Bingo!

used:            3605012 blobs / 2.383 TiB
duplicates:       778241 blobs / 627.728 GiB
unused:           129952 blobs / 101.334 GiB
total:           4513205 blobs / 3.095 TiB
unused size: 23.00% of total size

to repack:          1374 blobs / 4.466 MiB
this removes:          0 blobs / 0 B
to delete:           447 blobs / 139.583 MiB
total prune:         447 blobs / 139.583 MiB
remaining:       4512758 blobs / 3.095 TiB
unused size after prune: 728.925 GiB (23.00% of remaining size)

totally used packs:      55460
partly used packs:      136243
unused packs:                9

to keep:          191682 packs
to repack:            21 packs
to delete:             9 packs
1 Like
14

Thanks for the confirmation. Feel free to get rid of the unused data, I think I have all the information I need to fix the issue.

Edit:
as a workaround for now to get rid of the duplicates, you can use --max-unused 0 --max-repack-size 20G (adjust the repack-size as necessary).