Listing snapshots is slow

Getting Help
1

We are finding that listing snapshots in our repository is quite slow. We are using Restic to do daily backups on around 80 hosts to a single AWS S3 hosted repository. It takes anywhere from a minute and a half to over 5 minutes when listing snapshots for both a single host or all of them.

None of the backups are large (this is mostly configuration files and some small data files), but there are almost 12,000 snapshots now.

Frequently, but not always, messages like the ones below are printed:

Load(<lock/82e790f970>, 0, 0) returned error, retrying after 461.576101ms: The specified key does not exist.
Load(<lock/82e790f970>, 0, 0) returned error, retrying after 772.939287ms: The specified key does not exist.
Load(<lock/82e790f970>, 0, 0) returned error, retrying after 847.733063ms: The specified key does not exist.
Load(<lock/82e790f970>, 0, 0) returned error, retrying after 1.319761679s: The specified key does not exist.
Load(<lock/82e790f970>, 0, 0) returned error, retrying after 18.925881982s: The specified key does not exist.
Load(<lock/82e790f970>, 0, 0) returned error, retrying after 22.702309006s: The specified key does not exist.

Is there a way to speed these listings up? I’m afraid that all the other operations (like restores) are slow too. Perhaps:

  • Arrange the storage in a different way. I didn’t want to use multiple AWS buckets because managing a lot of them seems inconvenient. Can you have multiple Restic repositories in a single bucket, with an S3 object path like s3://my-backups/some-subfolder-name, perhaps with “some-subfolder-name” being the hostname or maybe a category, like “webservers”

  • Should we just accept the fact that we should prune old stuff? I doubt if last year’s /etc/ files are that valuable.

  • Is there some other maintenance operation we can do on the repository to optimize it that needs to be done a regular basis? This is restic 0.9.6 compiled with go1.13.4 on linux/amd64.

Thanks,
Chuck

2

Please upgrade to restic 0.12.1 (or at least 0.12.0 but if you do that there’s no reason to not go all the way to 0.12.1), then try again :slight_smile:

3

Nice,that did bring down snapshot list time to 20-30 seconds. For the long term, is the single S3 repository for this number of hosts a good way to go or should we consider a different arrangement?

4

It would be preferable to use different repositories either per client or per group of client. You can tell restic to store the repository in a subfolder of the bucket.

5

Ah ok, I think that might be a great way to go. Right now our restic repository is specified as:

export RESTIC_REPOSITORY=s3:https://s3.amazonaws.com/backup-bucket

Maybe this will work:

export RESTIC_REPOSITORY=s3:https://s3.amazonaws.com/backup-bucket/my-host.domain.com

One bucket, several repositories (one per host in this example). I’ll try that. It looks like the non-alphanumeric characters (dots and dashes in particular) are valid for use in AWS S3 object keys.