Prune speed with remote repository?

Getting Help
1

Hi, I’m currently evaluating the use of restic 0.9.4 from Debian buster repositories for my backup purposes. Basically, I am looking to copy the mail data from a small mail server (40 Gb) to a remote repository. Initial copy, after the first attempt was aborted over a connection loss, took 9 hours. A subsequent update was really fast, which made me very optimistic. As I am only looking for a single backup, I thought I’d test deleting old snapshots:

root@zeus:/usr/local/bin# restic forget --keep-last 1 --prune --password-file /root/.restic/mail.password -r sftp:root@truenas:/mnt/pool/backup/mail
repository c891f9ec opened successfully, password is correct
Applying Policy: keep the last 1 snapshots snapshots
snapshots for (host [zeus], paths [/mnt/zfs_opt/mail]):

keep 1 snapshots:
ID        Time                 Host        Tags        Reasons        Paths
---------------------------------------------------------------------------------------
7910921a  2020-11-09 18:06:52  zeus                    last snapshot  /mnt/zfs_opt/mail
---------------------------------------------------------------------------------------
1 snapshots

remove 2 snapshots:
ID        Time                 Host        Tags        Paths
------------------------------------------------------------------------
5b836bc0  2020-11-09 07:28:13  zeus                    /mnt/zfs_opt/mail
a44f8738  2020-11-09 07:40:35  zeus                    /mnt/zfs_opt/mail
------------------------------------------------------------------------
2 snapshots

2 snapshots have been removed, running prune
counting files in repo
building new index for repo
[16:24] 100.00%  15312 / 15312 packs
repository contains 15312 packs (234398 blobs) with 71.786 GiB
processed 234398 blobs: 114870 duplicate blobs, 35.241 GiB duplicate
load all snapshots
find data that is still in use for 1 snapshots
[0:01] 100.00%  1 / 1 snapshots
found 119300 of 234398 data blobs still in use, removing 115098 blobs
will remove 0 invalid files
will delete 4 packs and rewrite 15039 packs, this frees 35.329 GiB
[33:17] 0.64%  96 / 15039 packs rewritten

Ok, so after 33 minutes, it had 0.64% of the prune completed. Meanwhile, 3 hours have passed, and a bit more than 3.5% of the prune have been completed. At a bit more than 1% per hour, prune will not take more than 100 hours to complete…

Not sure what I’m doing wrong, but that is a tenth of the speed of deleting everything and doing a new backup?

Please help!

1 Like
2

Thanks a lot for a clear problem description, including a paste with both the command and all of its output!

Version 0.9.4 is quite old and it’s highly recommended to use a newer one. The current release is 0.11.0.

Prune is actually the feature that is being worked on right now (since 0.11.0 was released). There’s already been a couple of PRs merged to the restic master branch which has shown tremendous performance improvements when pruning, and there’s even more improvements to come before a release is made with the overall improved prune.

If you want to do so, you can test the latest master which includes the prune improvements I mentioned, by downloading the latest build from https://beta.restic.net/?sort=time&order=desc . Just download the binary for your operating system and run it in place of the current restic binary you have, it should work out of the box.

I’d be happy if you did so and then reported back your experience with it :slight_smile:

1 Like