Issue with Corrupted repo

Hi,
I was wondering if someone could shed some light on the issue im getting. currently running restic 0.9.5 and 2 of repositories. Both of them were getting

pack 55dbc6e7: not referenced in any index
pack 36f77f5f: not referenced in any index
pack 9f3e427b: not referenced in any index
60102 additional files were found in the repo, which likely contain duplicate data.

so on each repo i did the following

  • restic -r /path to repo rebuild-index
  • then restic -r /path to repo prune
  • then restic -r /path to repo check

The first repo worked fine and fixed it but the second repo when i run prune and leave it for a while i get this

> root@prometheus:/media/backupnas# restic -r backupzeus/ prune
enter password for repository:
repository 088f7bb8 opened successfully, password is correct
counting files in repo
building new index for repo
[47:45] 100.00%  120078 / 120078 packs
repository contains 120078 packs (1852706 blobs) with 508.855 GiB
processed 1852706 blobs: 1361386 duplicate blobs, 308.873 GiB duplicate
load all snapshots
find data that is still in use for 90 snapshots
[1:24] 100.00%  90 / 90 snapshots
found 452142 of 1852706 data blobs still in use, removing 1400564 blobs
will remove 0 invalid files
will delete 1511 packs and rewrite 80586 packs, this frees 342.057 GiB
Save(<data/27623663f8>) returned error, retrying after 552.330144ms: Sync: sync                                                                                                                                  backupzeus/data/27/27623663f8ccb00fede18a1168f954c3babcdb65e581a46909f9fc230b3db                                                                                                                                 fd9: resource temporarily unavailable
Save(<data/763d13078c>) returned error, retrying after 720.254544ms: Sync: sync                                                                                                                                  backupzeus/data/76/763d13078c84e89128ce95209601378821e04be62ceb3d278932c4a3e33b0                                                                                                                                 f94: resource temporarily unavailable
hash does not match id: want 3046d7bb2d7456b3f6040016cfd4de1cbd2b9ecb856e2b0b6af                                                                                                                                 e02a08923ce66, got dc6d2301a9fd3fe89f8bdb9704d18ac1c7f664b58f9f461b8fa7192d77cd9                                                                                                                                 0eb
github.com/restic/restic/internal/repository.Repack
        /restic/internal/repository/repack.go:34
main.pruneRepository
        /restic/cmd/restic/cmd_prune.go:278
main.runPrune
        /restic/cmd/restic/cmd_prune.go:85
main.glob..func18
        /restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:200
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337

Welcome back to the forum!

Uh, that does not sound good. When restic rewrites data (so the unreferenced data can be removed) it asks the filesystem and the OS to make sure the data has been saved (via the syscall fsync). The process blocks until the filesystem reports back that all data has been written successfully.

In your case here, the fsync syscall returned a (tepmorary) error. That’s not good, but outside of restic’s scope. Is the file system mounted over the network? If so, it looks like the implementation for the network file system (server and/or client) is not robust enough to handle restic’s workload.

Can you check the kernel log on that machine with journalctl and/or dmesg for any errors?

And then there’s this, which makes me wonder if there’s some hardware problem on the machine that runs restic… possibly bad RAM?

Hm, that’s compatible with my hypothesis that they use a network file system and something went wrong during prune… :slight_smile:

Thanks for the reply, currently have restic running on proxmox (debian 9) and i have CIFS mount the NAS which have the repo, whats odd is that on the same NAS i have 2 repo did the same one worked the other did not, whats also an issue is that its locking everytime i always have to unlock before running

update: have another server at another site very odd same issue alike i tried to prune and got this error

incomplete pack file (will be removed): ff5f952cce001f3257322c2b10cbd89d984b0fce 667ce380f88836c7bb479827
incomplete pack file (will be removed): ff85dfef4e6a266149031fb6e8cd0b0cbdcaa438 75ffddc1a980429506682d3f
incomplete pack file (will be removed): ff9aeff552d460b379ed7a5857c67f78afb5347b 39462dbcc885d561e498a795
incomplete pack file (will be removed): ffc2ef6dbf1f4076a6a83ed6fba00115dcf88a23 88043e9141b48d25c03b1b2c
incomplete pack file (will be removed): fff8c4675e139971e9fbc4cb205b4aa37b38ede8 3db3590764ff284495c1fd6a
repository contains 232695 packs (1682475 blobs) with 1009.106 GiB
processed 1682475 blobs: 22866 duplicate blobs, 11.653 GiB duplicate
load all snapshots
find data that is still in use for 90 snapshots
tree d0a5ad81f6e3bf54063dd583a7b3459bda5756aa56d4d47a5c84efe402eba8ad not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
/restic/internal/repository/repository.go:709
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:11
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
main.pruneRepository
/restic/cmd/restic/cmd_prune.go:191
main.runPrune
/restic/cmd/restic/cmd_prune.go:85
main.glob…func18
/restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
/restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
/restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
/restic/cmd/restic/main.go:86
runtime.main
/usr/local/go/src/runtime/proc.go:200
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1337

i tried checking

restic -r sql/ check --read-data
using temporary cache in /tmp/restic-check-cache-729972842
enter password for repository:
repository ff56417b opened successfully, password is correct
created new cache in /tmp/restic-check-cache-729972842
create exclusive lock for repository
load indexes
error: error loading index 2b9c709d: load <index/2b9c709dbd>: invalid data returned
error: error loading index 76c24252: load <index/76c242526b>: invalid data returned
error: error loading index c9519789: load <index/c9519789b7>: invalid data returned
error: error loading index cc92cb87: load <index/cc92cb87d2>: invalid data returned
error: error loading index d0c3fd34: load <index/d0c3fd344a>: invalid data returned
Fatal: LoadIndex returned errors

so i reran the rebuild index and found this

root@prometheus2:/media/nas/drobo# restic -r sql/ rebuild-index
enter password for repository:
repository ff56417b opened successfully, password is correct
counting files in repo
pack file cannot be listed 1b91e68ac20229fb980823ad3ba68680649c4e3fffd7483875ad241d30bbb34d: ciphertext ver ification failed
pack file cannot be listed 1ba4c1debcbf29600af511c2bfe2d73c84f8811cb94c9abd7ef055427d668893: ciphertext ver ification failed
pack file cannot be listed 1bb28534c90f810289fc35c69de9b6c4b4a643fe39e629ef84e6c7ced3d6c494: ciphertext ver ification failed
[4:36] 11.74% 7816 / 66594 packs