Pruning multiple snapshots simultaneously — fasters?

I’ve got backups working nicely and I’m trying out the forget --prune incantation for the first time. It’s…slow:

[16:42] 1.44% 13249/914936

At that rate, this prune will take about 19 hours to complete. This is…unpleasant. (At least the CPU and RAM usage aren’t too crazy.)

For background about our data: It’s pretty incremental. Maybe 5% of it changes and the rest just grows. Things rarely get deleted. We have about 4TB of data, most of it fairly small PDFs.

I’m thinking about ways to speed things up:

  • Does pruning infrequently make things faster because you can prune 20 snapshots in about the same amount of time as you can prune one snapshot? Seems like scanning is the slowest thing here.

  • Should we back up our changing data (the database, mostly) in a different repo than the data that steadily grows and is almost never deleted? That way we could prune the changing data regularly, and the (mostly) static data rarely.

  • Are there other things to do to speed this up?

Thanks,

Mike

1 Like

Yes, scanning all the blobs is the slowest part.
I’m doing exactly point 1 & 2 for my backups.

1 Like

Prune needs optimization, badly. We started by implementing the safest, most straightforward algorithm, because this is really the critical piece in restic: prune removes data, so we need to be really really sure that the data should be removed. Optimization comes second, and I’m already working on that.

The scanning phase is slow because restic re-reads all headers of all files in the repo to find out what’s in each file. This is slow because:

  • it needs to access each file and download a tiny bit from the end of the file
  • it does that twice, at the start and the end of prune
  • it processes each file sequentially, no parallelism/concurrency involved, both at the scanning and the repacking phase

Only at the very very end, after the second scan has finished successfully, prune actually deletes data. That’s also a safety feature.

I’m already working on some internal restructure needed to make prune faster, the ideas I have are roughly as follows:

  • Use existing index files to find out which blobs are there instead of reading all headers, but make plausibility checks (file exists and has the right size)
  • Reduce the “find which data is there” phase to one at the start
  • Download/repack data concurrently instead of sequentially
  • Make the amount of unneeded data configurable: at the moment, if there’s a single blob of data not needed any more contained in a file in the repo, the file will be downloaded and repacked, that’s not very efficient. Instead, only consider files for repacking when e.g. more than 20% of the file size is unneeded data. When we make this limit configurable, we allow users to select between:
    • setting the limit to 0% -> smallest repo size
    • setting the limit to 100% -> largest repo size, will only remove files which contain only unneeded data, may be the cheapest option for e.g. B2, where downloads/transactions are much more expensive that just leaving the data on the server

So, that’s the medium-term plan for prune :slight_smile:

4 Likes

Gotcha. I think this means I just won’t prune until this is fixed. The prune I just did took longer than the backup itself and only deleted 62GB out of about 4TB:

repository e5fe6787 opened successfully, password is correct
removed snapshot f8e6997f
removed snapshot 165dd535
2 snapshots have been removed, running prune
counting files in repo
building new index for repo
[18:55:49] 100.00%  914936 / 914936 packs
repository contains 914936 packs (21545116 blobs) with 4.060 TiB
processed 21545116 blobs: 237085 duplicate blobs, 3.106 GiB duplicate
load all snapshots
find data that is still in use for 5 snapshots
[1:03:52] 100.00%  5 / 5 snapshots
found 21174866 of 21545116 data blobs still in use, removing 370250 blobs
will remove 0 invalid files
will delete 5397 packs and rewrite 18262 packs, this frees 62.103 GiB
[6:56:36] 100.00%  18262 / 18262 packs rewritten
counting files in repo
[16:31:30] 100.00%  901209 / 901209 packs
finding old index files
saved new indexes as [<-- 8<  SNIP - Lots of hashes here 8< -->]
remove 1667 old index files
[1:25:33] 100.00%  23659 / 23659 packs deleted
done

Thanks for all your hard work here. This is obviously a really difficult part of the code.

One thought: Is it worth mentioning in the docs that prune is so slow? If it’s really about as slow as a backup, that’s potentially a big deal that people should know up front. I’d be happy to do a PR for that if so.

Please do so! Small documentation improvements are always greatly appreciated! :slight_smile:

Please improve verbose output.

While you are improving prune, could you add an option to send more verbose data to stdout. I am currently running a prune & forget job with output diverted to a logfile. It has been going for 2 days now and I have no idea on it’s progress. The last lines of my log file read:

found 337264 of 499011 data blobs still in use, removing 161747 blobs
will remove 0 invalid files
will delete 1309 packs and rewrite 3922 packs, this frees 16777216.000 TiB

I know from the last time that I did a prune that it is probably running fine, but I won’t get any more output until it completes, and all I can do is be patent.

Could you add a command line option to all the long running tasks (backup, prune index rebuild) so that output is verbose and each bit of output on a fresh line, so that it is easy to tail a logfile and view progress. I would also suggest that you make that option the default when stdout is not a TTY, though that opinion might be controversial.

Yes, we’re in the process of reworking all output, starting with the backup operation in 0.9.0.

One more data point here re prune’s performance (though I don’t know that we need more data points). I set up daily streaming backups of my database and have been doing them for a week now, triggering my first prune operation. The backups take about an hour to do and consist of a single file. The repo has only ever had a single file. The file is typically about 50GB, and as of now we’ve got seven snapshots of it on backblaze B2 storage. So…how does the prune of a single file in a small repository go? Badly, very badly:

Applying Policy: keep the last 7 daily, 4 weekly, 2 monthly snapshots
snapshots for (host [[courtlistener.com](http://courtlistener.com)], paths [/courtlistener-db.sql]):

keep 7 snapshots:
ID        Date                 Host               Tags        Directory
----------------------------------------------------------------------
10b6567e  2018-07-24 15:55:07  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
85ae84a1  2018-07-25 20:00:01  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
8f2dd42b  2018-07-28 20:00:01  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
79dc2265  2018-08-01 20:00:01  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
89a38fc0  2018-08-04 20:00:02  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
2b5374ba  2018-08-08 20:00:01  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
6c82e12d  2018-08-11 20:00:01  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
----------------------------------------------------------------------
7 snapshots

remove 1 snapshots:
ID        Date                 Host               Tags        Directory
----------------------------------------------------------------------
5409fc32  2018-07-23 12:36:07  [courtlistener.com](http://courtlistener.com)              /courtlistener-db.sql
----------------------------------------------------------------------
1 snapshots

1 snapshots have been removed, running prune
counting files in repo
building new index for repo
[2:26:45] 100.00%  105787 / 105787 packs

repository contains 105787 packs (295380 blobs) with 568.349 GiB
processed 295380 blobs: 0 duplicate blobs, 0B duplicate
load all snapshots
find data that is still in use for 7 snapshots
[0:05] 100.00%  7 / 7 snapshots

found 277032 of 295380 data blobs still in use, removing 18348 blobs
will remove 0 invalid files
will delete 2828 packs and rewrite 7770 packs, this frees 36.471 GiB
[2:55:36] 100.00%  7770 / 7770 packs rewritten

counting files in repo
[2:21:40] 100.00%  99129 / 99129 packs

finding old index files
saved new indexes as [877b73ad 2ee8e2c3 03c99ffd d7d37c66 ce962e73 799fb443 bd35a27e 5a4dd14e 6a84a239 80cafd27 ba7590e0 a66c78fe aa5d50fd 1c577e89 188bffe2 e44ad8df b1d24a2d c08cd2b9 71a63277 f11a3116 811a5c39 31f6f546 3311af70 7bad7a07 5d984628 a51e51dc 50c997da 951b678b e6ed340c 4e9309b2 40c3e11c ab74431a e7003848 f70199e9]
remove 149 old index files
[33:38] 100.00%  10598 / 10598 packs deleted

done

In all, that’s about eight hours. Holy cow.

Err, shoot, sorry, I messed that up. The DELTA is about 50GB between each backup. The full file itself is about 500GB.

Your log shows that there’s a lot of data rewritten, which means it needs to be uploaded. Could this be limited by your bandwidth?

Good point, AFAIK B2 isn’t that fast.