Error on "Load(<snapshot/xxxx> ... conn.ObjectOpen: Object Not Found", backup stuck (openstack)

Hello, and thx in advance to take the time to read this.

I have several backup running on VPS to the “OVH Public Cloud” which use OpenStack Swift as backend.
For months, I have had no problems, but the 28/03/2020 I started to have some error during my daily backup (I did not change anything)

(I will hide the real path/hostname as this is customer production data…)

Restick version : restic_0.9.4_linux_amd64

Backup cmd : /root/restic -r swift:clientx:/ backup /home/www/site/clientx/ --password-file /path_to_restic_pass_file

Preview backup log, before error :


Files:          11 new,     0 changed,  3640 unmodified
Dirs:            0 new,     3 changed,     0 unmodified
Added to the repo: 506.202 KiB

processed 3651 files, 5.461 GiB in 0:03
snapshot 2ffe115c saved

Log after error :

Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 492.281519ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 575.963556ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.169576793s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 2.472413223s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 2.340672009s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 4.415271283s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 6.879400897s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 6.951407271s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 15.556540193s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 21.338218925s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 579.337318ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 855.502738ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 798.819413ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 914.563059ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.337963409s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 4.810099305s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 8.505349332s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 9.892090482s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 16.968228972s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 11.114048084s: conn.ObjectOpen: Object Not Found
Error loading snapshot 363e9a5b: conn.ObjectOpen: Object Not Found
github.com/restic/restic/internal/restic.FindLatestSnapshot.func1
	/restic/internal/restic/snapshot_find.go:39
github.com/restic/restic/internal/repository.(*Repository).List.func1
	/restic/internal/repository/repository.go:584
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1.1
	/restic/internal/backend/backend_retry.go:133
github.com/restic/restic/internal/backend/swift.(*beSwift).List.func1
	/restic/internal/backend/swift/swift.go:261
github.com/ncw/swift.(*Connection).ObjectsWalk
	/restic/vendor/github.com/ncw/swift/swift.go:1106
github.com/restic/restic/internal/backend/swift.(*beSwift).List
	/restic/internal/backend/swift/swift.go:237
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1
	/restic/internal/backend/backend_retry.go:127
github.com/cenkalti/backoff.RetryNotify
	/restic/vendor/github.com/cenkalti/backoff/retry.go:37
github.com/restic/restic/internal/backend.(*RetryBackend).retry
	/restic/internal/backend/backend_retry.go:36
github.com/restic/restic/internal/backend.(*RetryBackend).List
	/restic/internal/backend/backend_retry.go:126
github.com/restic/restic/internal/cache.(*Backend).List
	<autogenerated>:1
github.com/restic/restic/internal/repository.(*Repository).List
	/restic/internal/repository/repository.go:578
github.com/restic/restic/internal/restic.FindLatestSnapshot
	/restic/internal/restic/snapshot_find.go:36
main.findParentSnapshot
	/restic/cmd/restic/cmd_backup.go:366
main.runBackup
	/restic/cmd/restic/cmd_backup.go:450
main.glob..func2
	/restic/cmd/restic/cmd_backup.go:60
github.com/spf13/cobra.(*Command).execute
	/restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
	/restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
	/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
	/restic/cmd/restic/main.go:86
runtime.main
	/usr/local/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333

Every command blocked with that “Load(<snapshot/363e9a5bb7>...” error, and I have some other repository from the same VPS to the same openstack that continue to work, for this server, I have 2 errors on 5 repos, 2 errors on 3 for another VPS with the same kind of error

I have tried to “rebuild-index” (which work without error and have remove 2 indexes), but that did not help. I also tried to wipe the local restic cache, no help either.

When I try the “snapshots” cmd I have :

created new cache in /root/.cache/restic
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 462.318748ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 890.117305ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 636.341646ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.107876242s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.511079094s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 3.04096222s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 5.781953173s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 11.222385136s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 9.152900914s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 16.927710407s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 391.517075ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 594.826393ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.326470261s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.212558276s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.779941781s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 3.268621161s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 6.097818893s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 11.639721773s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 10.163325309s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 15.321265721s: conn.ObjectOpen: Object Not Found
could not load snapshot 363e9a5b: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 682.667507ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 897.539375ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 1.151797844s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 891.511453ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 1.666393453s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 4.204102903s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 8.401962033s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 4.950254196s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 14.029373456s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 10.747238066s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 596.012294ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 601.142011ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 757.424518ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 1.756856005s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 2.643018794s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 2.955896126s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 5.257640273s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 8.80426156s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 9.656209421s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 15.032910374s: conn.ObjectOpen: Object Not Found
could not load snapshot f1cd2116: conn.ObjectOpen: Object Not Found
ID        Time                 Host               Tags        Paths
-------------------------------------------------------------------------------------------
fda29a28  2019-04-14 14:35:04  vpsyyyyy                      /home/www/site/clientx
7609b00e  2019-05-31 05:45:02  vpsyyyyy                      /home/www/site/clientx
af4dfe20  2019-06-30 05:45:01  vpsyyyyy                      /home/www/site/clientx
27f02c68  2019-07-31 05:45:01  vpsyyyyy                      /home/www/site/clientx
2fdb9351  2019-08-31 05:45:01  vpsyyyyy                      /home/www/site/clientx
d430a5ae  2019-09-30 05:45:01  vpsyyyyy                      /home/www/site/clientx
b035d660  2019-10-31 05:45:01  vpsyyyyy                      /home/www/site/clientx
fc314023  2019-11-30 05:45:01  vpsyyyyy                      /home/www/site/clientx
b5daf1e3  2019-12-31 05:45:01  vpsyyyyy                      /home/www/site/clientx
34498eb4  2020-01-31 05:45:01  vpsyyyyy                      /home/www/site/clientx
413b250b  2020-02-29 05:45:01  vpsyyyyy                      /home/www/site/clientx
57e9bd00  2020-03-01 05:45:01  vpsyyyyy                      /home/www/site/clientx
622e4968  2020-03-08 05:45:01  vpsyyyyy                      /home/www/site/clientx
7a608516  2020-03-15 05:45:01  vpsyyyyy                      /home/www/site/clientx
d23805c4  2020-03-21 05:45:01  vpsyyyyy                      /home/www/site/clientx
d69ecca2  2020-03-22 05:45:01  vpsyyyyy                      /home/www/site/clientx
c0cbea77  2020-03-23 05:45:01  vpsyyyyy                      /home/www/site/clientx
f42cf964  2020-03-24 05:45:01  vpsyyyyy                      /home/www/site/clientx
dd5be703  2020-03-25 05:45:01  vpsyyyyy                      /home/www/site/clientx
ee8544ec  2020-03-26 05:45:01  vpsyyyyy                      /home/www/site/clientx
2ffe115c  2020-03-27 05:45:01  vpsyyyyy                      /home/www/site/clientx
-------------------------------------------------------------------------------------------
42 snapshots

On the openstack switch data, I can see the 2 snapshots files that are in error :

snapshots/363e9a5bb7ba2e997aea5b6028cf121b6bc1af3a915cc9d2d96d1c897f98071d
snapshots/f1cd2116db7de4093516ef99d99a4197f6d79a3d08376cd1067601d2ff4f0677

And I cannot “forget” the snapshots in error, that did not work with the same error, that ends with
Ignoring "363e9a5bb7ba2e997aea5b6028cf121b6bc1af3a915cc9d2d96d1c897f98071d", could not load snapshot: conn.ObjectOpen: Object Not Found

The “check” cmd also fail with the same error.

I did not try to recover data, but the “data/” files seem to be there.

So, any idea to solve the problem without wiping all the repository and restart my backup from scratch ?

Thanks a lot for your help

Hello, your error messages look very similar to OpenStack backend error: `conn.ObjectOpen: Object Not Found` . From what I can tell, OpenStack Swift seems to be unable to return the content of these two snapshots. You could try to access the file using the swift command line client, to make sure that these files are indeed inaccessible. If that is the case, the simplest way forward would be to delete the two damaged snapshots, this will just remove the corresponding snapshots but leaves everything else in place. In case the swift backend has a backup copy of the snapshots somewhere, you could also remove and then restore the snapshots.

Once these inaccessible files no longer show up in the snapshots ‘folder’, restic should be able to continue the backups. Please run a full restic check --read-data run to make sure that the remainder of the backup is intact (it’s probably ok as rebuild-index did work, but better be safe). That will also validate the full content of each pack file in addition to just checking the backup metadata.

You could also try the recover command which will search your backup for all existing directory trees in case you need some files included in the damaged snapshots.

You (or the backend provider) will definitely need to investigate why these snapshots can no longer be retrieved. It is a serious problem if the storage backend looses data; restic might become unable to restore some files or even whole directories if files in the data directory were to disappear.

P.S.: The anonymized paths/hostnames are fine.

Hello @MichaelEischer, and thx for your fast and complete response.

I have seen the topic that you have linked (and others) but I was stupidly focusing on the fact that it seems to every time occurred on the “lock” files, and not on the “snapshot”. But anyway, you are right, the 2 snapshots files were listed but inaccessible with Switch (I was dumb enough to not try to download them…)… And removing them have resolved the issue.

So thanks a lot for that, I will do a complete check with “read-data” on all the impacted repo (no errors have been found on this one).

But as you say, it’s a major failure from the OVH Public Cloud Swift offer, my confidence in their services has just dropped in the abyss… I will contact them, but clearly something happened on their side on the 28th, because all of the problems are on the same main “swift” frontend (GRA3 in this case).

As I wanted to add more backup to restic, I’m now looking for another provider… Any advice to choose a good “cloud” repository backend for less than 1To of data ? I don’t know if S3 or Azure are prone to failures like OVH just did…

Small follow up after contacting OVH Support and some investigation on my side, it appears that it’s not a data loss, but in fact it’s deleted data that “reappeared” on the swift listing, but with no data associated…

The snapshots files that blocked my backup on this example was first deleted 3 weeks before, and reappeared the 28th…

They seem to be aware of the problem and working on it but they do not give an estimated time to fix it.

I will try Backblaze B2…