Rebuild Succeeds, but Check Fails – Restic Troubleshooting Request

Getting Help
1

Hello everyone!

I’m using restic 0.14.0 compiled with go1.19.8 on linux/amd64 on aDebian GNU/Linux 12 (bookworm), and I’m experiencing issues with the repositories.

Last Thursday, we found that the connection was interrupted during the backup process to the remote repository:

Save(<lock/4cefbcc797>) returned error, retrying after 552.330144ms: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 1.080381816s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 1.31013006s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 1.582392691s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 2.340488664s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 4.506218855s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 3.221479586s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 5.608623477s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 7.649837917s: Write: sftp: "Failure" (SSH_FX_FAILURE)
Save(<lock/4cefbcc797>) returned error, retrying after 15.394871241s: Write: sftp: "Failure" (SSH_FX_FAILURE)
sftp: "Failure" (SSH_FX_FAILURE)
Write
github.com/restic/restic/internal/backend/sftp.(*SFTP).Save
    github.com/restic/restic/internal/backend/sftp/sftp.go:367
github.com/restic/restic/internal/backend/limiter.rateLimitedBackend.Save
    github.com/restic/restic/internal/backend/limiter/limiter_backend.go:30
github.com/restic/restic/internal/backend.(*RetryBackend).Save.func1
    github.com/restic/restic/internal/backend/backend_retry.go:66
github.com/cenkalti/backoff.RetryNotifyWithTimer
    github.com/cenkalti/backoff/retry.go:55
github.com/cenkalti/backoff.RetryNotify
    github.com/cenkalti/backoff/retry.go:34
github.com/restic/restic/internal/backend.(*RetryBackend).retry
    github.com/restic/restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Save
    github.com/restic/restic/internal/backend/backend_retry.go:60
github.com/restic/restic/internal/cache.(*Backend).Save
    github.com/restic/restic/internal/cache/backend.go:59
github.com/restic/restic/internal/repository.(*Repository).SaveUnpacked
    github.com/restic/restic/internal/repository/repository.go:488
github.com/restic/restic/internal/restic.SaveJSONUnpacked
    github.com/restic/restic/internal/restic/json.go:31
github.com/restic/restic/internal/restic.(*Lock).createLock
    github.com/restic/restic/internal/restic/lock.go:161
github.com/restic/restic/internal/restic.newLock
    github.com/restic/restic/internal/restic/lock.go:105
github.com/restic/restic/internal/restic.NewLock
    github.com/restic/restic/internal/restic/lock.go:66
main.lockRepository
    github.com/restic/restic/cmd/restic/lock.go:42
main.lockRepo
    github.com/restic/restic/cmd/restic/lock.go:23
main.runBackup
    github.com/restic/restic/cmd/restic/cmd_backup.go:583
main.glob..func2
    github.com/restic/restic/cmd/restic/cmd_backup.go:74
github.com/spf13/cobra.(*Command).execute
    github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
    github.com/spf13/cobra/command.go:1044
github.com/spf13/cobra.(*Command).Execute
    github.com/spf13/cobra/command.go:968
main.main
    github.com/restic/restic/cmd/restic/main.go:98
runtime.main
    runtime/proc.go:250
runtime.goexit
    runtime/asm_amd64.s:1594
unable to create lock in backend

Since then, we’ve tried running rebuild-index, check, and prune (prune didn’t work either)

Currently, when we run rebuild-index, there don’t seem to be any errors:

root@AAAA:/usr/lib/nagios/plugins# restic -r sftp:$host_repo@AAAA-bkp.local:/data/$host_repo/instalacions \
--password-file "" rebuild-index
repository 4245cf01 opened (repository version 2) successfully, password is correct
loading indexes...
getting pack files to read...
adding pack file to index b95b40f666ba14e986dfcd526437c345c751f788021aed3dec24338d1bd0bc2b
adding pack file to index 6982e67bf510f3ce04d818a040878bd442567bbace2657b00e6780e525135509
reading pack files
[0:00] 100.00%  2 / 2 packs
rebuilding index
[0:02] 99.98%  8939 / 8941 packs processed
deleting obsolete index files
[0:00] 100.00%  40 / 40 files deleted
done

However, when we run check, the following issue occurs:

root@AAAA:/usr/lib/nagios/plugins# restic -r sftp:$host_repo@AAAA-bkp.local:/data/$host_repo/instalacions --password-file "AAAA" check
using temporary cache in /tmp/restic-check-cache-2297834724
repository 4245cf01 opened (repository version 2) successfully, password is correct
created new cache in /tmp/restic-check-cache-2297834724
create exclusive lock for repository
load indexes
check all packs
pack 6982e67bf510f3ce04d818a040878bd442567bbace2657b00e6780e525135509: not referenced in any index
pack b95b40f666ba14e986dfcd526437c345c751f788021aed3dec24338d1bd0bc2b: not referenced in any index
2 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree 094618fe:napshots
  tree 094618feab68be9ffe7c7f45cb8ec47287e6bde8e5d1d6a5f47d1745246dc697: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 094618feab68be9ffe7c7f45cb8ec47287e6bde8e5d1d6a5f47d1745246dc697: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 094618feab68be9ffe7c7f45cb8ec47287e6bde8e5d1d6a5f47d1745246dc697: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree dbf1ad16:napshots
  tree dbf1ad1649758822d05bd46a72a143d949d2e9cdae45b890203e020b206058d6: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree dbf1ad1649758822d05bd46a72a143d949d2e9cdae45b890203e020b206058d6: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree dbf1ad1649758822d05bd46a72a143d949d2e9cdae45b890203e020b206058d6: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree bdfe3b43:napshots
  tree bdfe3b43e267e9a4bfc90de824542f128218ecdd81d43caab820f9f67c50e126: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree bdfe3b43e267e9a4bfc90de824542f128218ecdd81d43caab820f9f67c50e126: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree bdfe3b43e267e9a4bfc90de824542f128218ecdd81d43caab820f9f67c50e126: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree a762bd38:napshots
  tree a762bd382a13a40b1d41041790249f9a6e0404f45e1f7d61d9f10b236626ab1a: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree a762bd382a13a40b1d41041790249f9a6e0404f45e1f7d61d9f10b236626ab1a: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree a762bd382a13a40b1d41041790249f9a6e0404f45e1f7d61d9f10b236626ab1a: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 3e0fda31:snapshots
  tree 3e0fda316a5cdf3b7c3c339d26f5ac4acb85494f262860f30013000f0e022a65: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 3e0fda316a5cdf3b7c3c339d26f5ac4acb85494f262860f30013000f0e022a65: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 3e0fda316a5cdf3b7c3c339d26f5ac4acb85494f262860f30013000f0e022a65: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 18169e24:snapshots
  tree 18169e24e9e652ff4b25120eb0df227b621f2f39dba866cb4167890931c83a0f: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 18169e24e9e652ff4b25120eb0df227b621f2f39dba866cb4167890931c83a0f: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 18169e24e9e652ff4b25120eb0df227b621f2f39dba866cb4167890931c83a0f: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 2903526a: snapshots
  tree 2903526a880ad4d1a49d5f98657410ba6bfbb91ecdd034345986c003967f453d: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 2903526a880ad4d1a49d5f98657410ba6bfbb91ecdd034345986c003967f453d: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 2903526a880ad4d1a49d5f98657410ba6bfbb91ecdd034345986c003967f453d: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree a3d23ba3: snapshots
  tree a3d23ba3d4ba01dc74f4a436fd3f7a11f480ed91760627e188f47b7c7cacafa2: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree a3d23ba3d4ba01dc74f4a436fd3f7a11f480ed91760627e188f47b7c7cacafa2: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree a3d23ba3d4ba01dc74f4a436fd3f7a11f480ed91760627e188f47b7c7cacafa2: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 88a5050f: snapshots
  tree 88a5050f23e4256f133838d10ced9315fec0bb2ad7d60a705a7ba8f2b40bea35: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 88a5050f23e4256f133838d10ced9315fec0bb2ad7d60a705a7ba8f2b40bea35: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 88a5050f23e4256f133838d10ced9315fec0bb2ad7d60a705a7ba8f2b40bea35: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 80486390: snapshots
  tree 8048639043256c01e0cd70e5b588513ac42b35a5022d8925ae967d2323d4bf76: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 8048639043256c01e0cd70e5b588513ac42b35a5022d8925ae967d2323d4bf76: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 8048639043256c01e0cd70e5b588513ac42b35a5022d8925ae967d2323d4bf76: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 75485b36: snapshots
  tree 75485b36491da2691e2b6b71686aa5887e8909eb9f3cf0c07a70d211f1cfdb66: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 75485b36491da2691e2b6b71686aa5887e8909eb9f3cf0c07a70d211f1cfdb66: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 75485b36491da2691e2b6b71686aa5887e8909eb9f3cf0c07a70d211f1cfdb66: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
  tree 75485b36491da2691e2b6b71686aa5887e8909eb9f3cf0c07a70d211f1cfdb66: file "BD_AAAA_2024-10-31.tar.gz" blob 253b5cce7b1137c430596343e6956c6e0be82e216ac13ee5811a42b91ea81f55 not found in index
error for tree a5ff7f42: snapshots
  tree a5ff7f4253f5184f3811f80b845d4a499b25363223b21bd6304350fa6de4ca75: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree a5ff7f4253f5184f3811f80b845d4a499b25363223b21bd6304350fa6de4ca75: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree a5ff7f4253f5184f3811f80b845d4a499b25363223b21bd6304350fa6de4ca75: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
error for tree 458b9ded: snapshots
  tree 458b9dedcb74d11b802873248224a6244ebdc15e2ebd193bfded451f06c3fc64: file "BD_AAAA_2024-08-28.tar.gz" blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a not found in index
  tree 458b9dedcb74d11b802873248224a6244ebdc15e2ebd193bfded451f06c3fc64: file "BD_AAAA_2024-08-28.tar.gz" blob b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 not found in index
  tree 458b9dedcb74d11b802873248224a6244ebdc15e2ebd193bfded451f06c3fc64: file "BD_AAAA_2024-08-28.tar.gz" blob 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8 not found in index
[0:06] 100.00%  538 / 538 snapshots
Fatal: repository contains errors

Could you please advise us on how to proceed?

We don’t think this is a hardware issue, as the physical hard drive is located in a remote data center and hosts other repositories that are functioning correctly.

Thanks in advance.

2

That error actually occurred before the backup could make any modifications to the repository. So it can’t be the source of the later described problems.

That looks like check was run before the rebuild-index execution shown above. Do the files at data/b9/b95b40f666ba14e986dfcd526437c345c751f788021aed3dec24338d1bd0bc2b and data/69/6982e67bf510f3ce04d818a040878bd442567bbace2657b00e6780e525135509 match their sha256 hash? (shasum -a256 ...). When were those files modified and created?

Which snapshots does restic find --blob 094618fe dbf1ad16 bdfe3b43 a762bd38 3e0fda31 18169e24 2903526a a3d23ba3 88a5050f 80486390 75485b36 a5ff7f42 458b9ded report as affected?

3

Hello,

I’m not sure because the script loops through an array of installations and makes backups of them in the /instalacions repository, so up until the moment the SSH connection was cut, it must have made changes to the repository (if I understood correctly).

Regarding the order of the commands, the script performs a check at the very end. Then, when the error occurred, I manually ran the rebuild-index, and it didn’t give an error. But when I ran the check again, it gave the one I attached in the previous message.

This is the result of the hash:

image
image1288×185 7.84 KB

About the modification…it seems that one exists and the other doesn’t. I’m a bit lost.

root@xxxxxxx:/data/xxxxxxxxxxx/instalacions/data/b9# ls -l b95b40f666ba14e986dfcd526437c345c751f788021aed3dec24338d1bd0bc2b
-rw------- 1 xxxx nogroup 16981922 ago 27  2024 b95b40f666ba14e986dfcd526437c345c751f788021aed3dec24338d1bd0bc2b

root@xxxxxxx:/data/xxxxxxxxxxx/instalacions/data/b9# ls -l 6982e67bf510f3ce04d818a040878bd442567bbace2657b00e6780e525135509
ls: no se puede acceder a '6982e67bf510f3ce04d818a040878bd442567bbace2657b00e6780e525135509': No existe el fichero o el directorio

The file BD_AAAA_2024-08-28.tar.gz is named after its creation date, so technically it was created on 08/28/2024.

Finally, regarding the search for the blobs, the command didn’t find anything.

I ran this in a loop with the different blobs:

restic -r sftp:$host_repo@xxxxxxx:/data/$host_repo/instalacions --password-file "xxxxxx" find --blob 094618fe

image
image736×698 24.8 KB

thank you very much

4

is located in data/69/6982e67bf510f3ce04d818a040878bd442567bbace2657b00e6780e525135509 (the directory name matches the first two characters of the filename). You’ve looked into the b9 folder.

Each restic command opens a new sftp connection. The sftp error message from the first post, states that restic failed to lock the repository. So that particular restic execution cannot have modified the repository.

I’ve noticed I’ve messed up the command, sorry! It should have been restic find --blob efc604b3e7827e37e530bdba574eaf5a7aaba521a05ca91d0a16c3aded58856a b98cb2c7ab7ddf0fcf2ff5e844f42ba83eda393d051accf99f2ad70851387447 4b0ea520c54c15b4a4349313b57d9999bb158fb5cab296adadd7ad2cd34126b8.

Do you still have the "BD_AAAA_2024-08-28.tar.gz" file? If yes, then just run a backup for that file. That should repair the repository.

Other than that you can follow the steps from Troubleshooting — restic 0.18.0 documentation to repair the repository. This will, however, completely remove the missing blobs from the repository, such that the damaged files reported by check will most likely be impossible to unpack.

Have you run prune or forget --prune recently? Those are the only commands that ever delete pack files. As the damage seems to affect some older snapshots, it’s either that something went wrong during that command execution or that the remote storage has swallowed some files. Although the latter would have caused some other error messages.