Hello, and thx in advance to take the time to read this.
I have several backup running on VPS to the “OVH Public Cloud” which use OpenStack Swift as backend.
For months, I have had no problems, but the 28/03/2020 I started to have some error during my daily backup (I did not change anything)
(I will hide the real path/hostname as this is customer production data…)
Restick version : restic_0.9.4_linux_amd64
Backup cmd : /root/restic -r swift:clientx:/ backup /home/www/site/clientx/ --password-file /path_to_restic_pass_file
Preview backup log, before error :
Files: 11 new, 0 changed, 3640 unmodified
Dirs: 0 new, 3 changed, 0 unmodified
Added to the repo: 506.202 KiB
processed 3651 files, 5.461 GiB in 0:03
snapshot 2ffe115c saved
Log after error :
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 492.281519ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 575.963556ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.169576793s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 2.472413223s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 2.340672009s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 4.415271283s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 6.879400897s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 6.951407271s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 15.556540193s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 21.338218925s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 579.337318ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 855.502738ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 798.819413ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 914.563059ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.337963409s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 4.810099305s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 8.505349332s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 9.892090482s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 16.968228972s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 11.114048084s: conn.ObjectOpen: Object Not Found
Error loading snapshot 363e9a5b: conn.ObjectOpen: Object Not Found
github.com/restic/restic/internal/restic.FindLatestSnapshot.func1
/restic/internal/restic/snapshot_find.go:39
github.com/restic/restic/internal/repository.(*Repository).List.func1
/restic/internal/repository/repository.go:584
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1.1
/restic/internal/backend/backend_retry.go:133
github.com/restic/restic/internal/backend/swift.(*beSwift).List.func1
/restic/internal/backend/swift/swift.go:261
github.com/ncw/swift.(*Connection).ObjectsWalk
/restic/vendor/github.com/ncw/swift/swift.go:1106
github.com/restic/restic/internal/backend/swift.(*beSwift).List
/restic/internal/backend/swift/swift.go:237
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1
/restic/internal/backend/backend_retry.go:127
github.com/cenkalti/backoff.RetryNotify
/restic/vendor/github.com/cenkalti/backoff/retry.go:37
github.com/restic/restic/internal/backend.(*RetryBackend).retry
/restic/internal/backend/backend_retry.go:36
github.com/restic/restic/internal/backend.(*RetryBackend).List
/restic/internal/backend/backend_retry.go:126
github.com/restic/restic/internal/cache.(*Backend).List
<autogenerated>:1
github.com/restic/restic/internal/repository.(*Repository).List
/restic/internal/repository/repository.go:578
github.com/restic/restic/internal/restic.FindLatestSnapshot
/restic/internal/restic/snapshot_find.go:36
main.findParentSnapshot
/restic/cmd/restic/cmd_backup.go:366
main.runBackup
/restic/cmd/restic/cmd_backup.go:450
main.glob..func2
/restic/cmd/restic/cmd_backup.go:60
github.com/spf13/cobra.(*Command).execute
/restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
/restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
/restic/cmd/restic/main.go:86
runtime.main
/usr/local/go/src/runtime/proc.go:201
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333
Every command blocked with that “Load(<snapshot/363e9a5bb7>...
” error, and I have some other repository from the same VPS to the same openstack that continue to work, for this server, I have 2 errors on 5 repos, 2 errors on 3 for another VPS with the same kind of error
I have tried to “rebuild-index
” (which work without error and have remove 2 indexes), but that did not help. I also tried to wipe the local restic cache, no help either.
When I try the “snapshots
” cmd I have :
created new cache in /root/.cache/restic
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 462.318748ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 890.117305ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 636.341646ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.107876242s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.511079094s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 3.04096222s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 5.781953173s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 11.222385136s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 9.152900914s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 16.927710407s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 391.517075ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 594.826393ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.326470261s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.212558276s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 1.779941781s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 3.268621161s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 6.097818893s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 11.639721773s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 10.163325309s: conn.ObjectOpen: Object Not Found
Load(<snapshot/363e9a5bb7>, 0, 0) returned error, retrying after 15.321265721s: conn.ObjectOpen: Object Not Found
could not load snapshot 363e9a5b: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 682.667507ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 897.539375ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 1.151797844s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 891.511453ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 1.666393453s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 4.204102903s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 8.401962033s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 4.950254196s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 14.029373456s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 10.747238066s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 596.012294ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 601.142011ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 757.424518ms: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 1.756856005s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 2.643018794s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 2.955896126s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 5.257640273s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 8.80426156s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 9.656209421s: conn.ObjectOpen: Object Not Found
Load(<snapshot/f1cd2116db>, 0, 0) returned error, retrying after 15.032910374s: conn.ObjectOpen: Object Not Found
could not load snapshot f1cd2116: conn.ObjectOpen: Object Not Found
ID Time Host Tags Paths
-------------------------------------------------------------------------------------------
fda29a28 2019-04-14 14:35:04 vpsyyyyy /home/www/site/clientx
7609b00e 2019-05-31 05:45:02 vpsyyyyy /home/www/site/clientx
af4dfe20 2019-06-30 05:45:01 vpsyyyyy /home/www/site/clientx
27f02c68 2019-07-31 05:45:01 vpsyyyyy /home/www/site/clientx
2fdb9351 2019-08-31 05:45:01 vpsyyyyy /home/www/site/clientx
d430a5ae 2019-09-30 05:45:01 vpsyyyyy /home/www/site/clientx
b035d660 2019-10-31 05:45:01 vpsyyyyy /home/www/site/clientx
fc314023 2019-11-30 05:45:01 vpsyyyyy /home/www/site/clientx
b5daf1e3 2019-12-31 05:45:01 vpsyyyyy /home/www/site/clientx
34498eb4 2020-01-31 05:45:01 vpsyyyyy /home/www/site/clientx
413b250b 2020-02-29 05:45:01 vpsyyyyy /home/www/site/clientx
57e9bd00 2020-03-01 05:45:01 vpsyyyyy /home/www/site/clientx
622e4968 2020-03-08 05:45:01 vpsyyyyy /home/www/site/clientx
7a608516 2020-03-15 05:45:01 vpsyyyyy /home/www/site/clientx
d23805c4 2020-03-21 05:45:01 vpsyyyyy /home/www/site/clientx
d69ecca2 2020-03-22 05:45:01 vpsyyyyy /home/www/site/clientx
c0cbea77 2020-03-23 05:45:01 vpsyyyyy /home/www/site/clientx
f42cf964 2020-03-24 05:45:01 vpsyyyyy /home/www/site/clientx
dd5be703 2020-03-25 05:45:01 vpsyyyyy /home/www/site/clientx
ee8544ec 2020-03-26 05:45:01 vpsyyyyy /home/www/site/clientx
2ffe115c 2020-03-27 05:45:01 vpsyyyyy /home/www/site/clientx
-------------------------------------------------------------------------------------------
42 snapshots
On the openstack switch data, I can see the 2 snapshots files that are in error :
snapshots/363e9a5bb7ba2e997aea5b6028cf121b6bc1af3a915cc9d2d96d1c897f98071d
snapshots/f1cd2116db7de4093516ef99d99a4197f6d79a3d08376cd1067601d2ff4f0677
And I cannot “forget” the snapshots in error, that did not work with the same error, that ends with
“Ignoring "363e9a5bb7ba2e997aea5b6028cf121b6bc1af3a915cc9d2d96d1c897f98071d", could not load snapshot: conn.ObjectOpen: Object Not Found
”
The “check
” cmd also fail with the same error.
I did not try to recover data, but the “data/
” files seem to be there.
So, any idea to solve the problem without wiping all the repository and restart my backup from scratch ?
Thanks a lot for your help