Non-existing keys errors after forget

Getting Help
1

Hi,

after having run restic --forget to remove some snapshots that had been committed more-or-less accidentally, I am seeing non-existing key errors:

bash-4.4$ restic -r $REPO -p ~/.restic/wasabi.pwd forget --host wandervogel-home --keep-last 1
repository 06114f99 opened successfully, password is correct
Applying Policy: keep the last 1 snapshots snapshots
snapshots for (host [wandervogel-home], paths [/Users/jdoe]):

keep 1 snapshots:
ID        Time                 Host              Tags        Reasons        Paths
------------------------------------------------------------------------------------------
25affdb2  2018-12-05 19:24:16  wandervogel-home              last snapshot  /Users/jdoe
------------------------------------------------------------------------------------------
1 snapshots

remove 11 snapshots:
ID        Time                 Host              Tags        Paths
---------------------------------------------------------------------------
dd878c2c  2018-12-02 20:36:56  wandervogel-home              /Users/jdoe
5ff8dddb  2018-12-03 05:14:04  wandervogel-home              /Users/jdoe
feae1428  2018-12-03 06:05:18  wandervogel-home              /Users/jdoe
1889b72b  2018-12-04 06:05:24  wandervogel-home              /Users/jdoe
b15b03ce  2018-12-04 18:17:34  wandervogel-home              /Users/jdoe
d029c664  2018-12-04 19:47:28  wandervogel-home              /Users/jdoe
650543da  2018-12-05 05:36:29  wandervogel-home              /Users/jdoe
20341e68  2018-12-05 06:32:50  wandervogel-home              /Users/jdoe
078b046f  2018-12-05 11:50:05  wandervogel-home              /Users/jdoe
5a07db9f  2018-12-05 14:55:18  wandervogel-home              /Users/jdoe
ceffd890  2018-12-05 19:09:07  wandervogel-home              /Users/jdoe
---------------------------------------------------------------------------
11 snapshots

bash-4.4$ restic -r $REPO -p ~/.restic/wasabi.pwd forget 25affdb2
repository 06114f99 opened successfully, password is correct
removed snapshot 25affdb2

bash-4.4$ restic -r $REPO -p ~/.restic/wasabi.pwd snapshots
repository 06114f99 opened successfully, password is correct
Load(<snapshot/b15b03ce5c>, 0, 0) returned error, retrying after 415.879788ms: Copy: The specified key does not exist.
Load(<snapshot/b15b03ce5c>, 0, 0) returned error, retrying after 818.024558ms: Copy: The specified key does not exist.

As well as index related errors:

bash-4.4$ restic -r $REPO -p ~/.restic/wasabi.pwd check
using temporary cache in /var/folders/7z/xjvww_7j1vs6n7zzfs3n7b9h0000gn/T/restic-check-cache-109749875
repository 06114f99 opened successfully, password is correct
created new cache in /var/folders/7z/xjvww_7j1vs6n7zzfs3n7b9h0000gn/T/restic-check-cache-109749875
create exclusive lock for repository
load indexes
Load(<index/2349849299>, 0, 0) returned error, retrying after 510.667103ms: Copy: The specified key does not exist.
Load(<index/f3fa8b0511>, 0, 0) returned error, retrying after 540.31219ms: Copy: The specified key does not exist.
Load(<index/f92ef98885>, 0, 0) returned error, retrying after 547.58134ms: Copy: The specified key does not exist.

A fairly recent restic check went through just fine (might even have been right before running restic --forget).

Any suggestions would be greatly appreciated as to (a) what is happening here, and more importantly (b) how to fix the issue (if even possible).

The repo is on Wasabi.

Thank you,
Clemens

2

Do you have versioning enabled on the Wasabi bucket? It is possible that a “list” operation against Wasabi is returning objects that do not actually exist. I have seen this behavior on S3 before; in particular, the “list” operation would return a different set of the deleted keys on every query, implying that there was some caching issue.

In my case, I saw the error with locks, which had the unfortunate effect of disabling restic access to the repository completely. (Restic checks for locks on startup, received a list of locks that were actually deleted, and then tried to fetch them.) I had to explicitly delete all past versions of all locks to remedy the situation. Since then, I have not seen the issue recur.

The problem with Wasabi is likely similar.

3

Thank you, @cdhowie, you are right - files appear in the listings, but are not actually downloadable/existing.

I have contacted Wasabi support to see what’s going on. At $5/TB/month, would be sad not to be able to use it …

4

Note that Wasabi does appear to have a 1TB overall storage minimum, and a 90-day minimum commitment for stored objects. Make sure you take this into account when running prune.

The best option I’ve found for restic backups is B2: $0.005 per GB-month ($0.0001 more than Wasabi) and $0.01 per GB egress. No minimums or object commitments.