Very slow prune operation

I have a (large?) repository with about 2.1Tb of data; I add a dozen snapshots each work day, then wanted to forget/prune in the weekend. Unfortunately, the prune operation is so slow that at the current rate it would take over 300hours, so clearly I can not perform it without stopping the hourly backup for a long time.

Is there anything I could try to speed this up?

You should probably include the complete command and what repository backend you use with the information in the screenshot.

But assuming you are using restic 0.12.0, which it looks like, you are using the version that has dramatically improved prune performance and should work as fast as it can.

Are you perhaps on a slow Google Cloud backend? How long has it been since you pruned last time?

Your current prune operation is slow because it wants to repack many packs. This might be due to duplicates (maybe because of aborted backup or prune) which are not perfectly handled in 0.12.0. There is

but this still needs a review and should so far not be used in a production repository.
EDIT: If it is possible for you to copy your repo, I would be very much interested in you testing this PR :wink:

You can try to play around with the parameter --max-unused. If this is set to unlimited ,only duplicates and tree packs will be repacked.

Moreover, you can use --max-repack-size to limit the size which is actually repacked. This should also work with duplicates.

Always try with --dry-run to see what prune intends to do and run it without when you are satisfied with the choice of your parameters.

2 Likes

Sorry for keeping important info about the setup :smiley:

So, the file server and the restic server are on the same network, accessed via SFTP, Iā€™m currently using restic 0.12.0.

I managed to perform the prune in about 26 hours by running it directly in the server, my next automatic prune operation happens tomorrow, I hope this will be quicker :slight_smile:

Unfortunately the repo got pruned so I could not test the patch.

I can confirm the issue was probably caused by messing around with partial/not completed backups while I was testing things.

The last schedule prune operation completed smoothly

loading all snapshots...
finding data that is still in use for 68 snapshots
[0:06] 100.00%  68 / 68 snapshots

searching used packs...
collecting packs for deletion and repacking
[0:14] 100.00%  407865 / 407865 packs processed


to repack:        10401 blobs / 214.733 MiB
this removes        502 blobs / 91.212 MiB
to delete:        10859 blobs / 7.970 GiB
total prune:      11361 blobs / 8.059 GiB
remaining:      1800313 blobs / 1.919 TiB
unused size after prune: 5.930 GiB (0.30% of remaining size)

repacking packs
[0:01] 100.00%  63 / 63 packs repacked

rebuilding index
[0:04] 100.00%  405965 / 405965 packs processed

deleting obsolete index files
[0:00] 100.00%  90 / 90 files deleted

removing 1928 old packs
[0:00] 100.00%  1928 / 1928 files deleted

done