Ciphertext verification failed and other errors

Hello everyone,

I have a problem with the repositories on multiple servers.

Two days ago, while the backup was being done, on the destination server where the repositories are stored, there was a failure in the disks.

On each server a script was running (the same script on all of them).

This script consists of different restic commands: it starts by doing “backup”, then cleaning, “forget” and “prune”. Finally it does a “check”

The process stopped at different times on each server, in some it stopped during the cleanup and in others during the backup itself (for example).

Now, on each server it seems to have different errors/behavior

Server 67

Executed commands:

restic check --read-data -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

restic rebuild-index -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

However, when checking again:

restic check -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"


Server 73

Executed commands:

restic check --read-data -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

restic rebuild-index -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

And when checking again:

Server 71

Executed commands:

restic check  -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

restic rebuild-index -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

(I have no images of the rebuild)

And when checking again:

It has not finished after 5 hours

Server 05

On this last one the executed commands have been the same:

restic check --read-data -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

restic rebuild-index -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

restic check --read-data -r sftp:$host_repo@tesi-bkp.local:/data/$host_repo/instalacions -p "/root/restic/pass_file.txt"

On this server during one of these two commands the error has been different

error for pack b46af5679bc2e94f19d3878ea3d99a9446a66748ffda4a5e9dc259d5ab3cea22: header is larger than file

But only have an image of the second check:

Any help will be appreciated, thank you very much!

Judging from the screenshots, those repositories are significantly damaged, so it will require a bit of fiddling to fix them. As multiple index files were damaged in each repository, that means that the disk failure not just damaged files that were written at that point in time, but also older files that have been stored for a while.

The first step is to get restic 0.17.0 (older version won’t work). Then run restic check --read-data. Once the command completes it should show how to run restic repair packs ... in order to fix the damaged pack files.

Afterwards, follow the remaining steps from Troubleshooting — restic 0.17.0 documentation to fix the snapshots themselves.