I am using restic (version 0.12.0) for backups on two external hard drives (ext4 encrypted). At my last backup one of the repos throw an error. When I check the repo with the “check” command I get
restic -r /run/media/xxxxxx/BACKUP1/backup_restic/ check
using temporary cache in /tmp/restic-check-cache-300876377
enter password for repository:
repository 1ddea405 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-300876377
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree 6b414ec5:pshots
decrypting blob 6b414ec5152afcddb62e55d74de2f8c43877780c809bd8f65e7a597a365372dd failed: ciphertext verification failed
[0:00] 100.00% 4 / 4 snapshots
Fatal: repository contains errors
When I search for this specific blob, it seems to be a tree blob
restic -r /run/media/xxxxxx/BACKUP1/backup_restic/ list blobs | grep 6b414ec5152afcddb62e55d74de2f8c43877780c809bd8f65e7a597a365372dd
enter password for repository:
tree 6b414ec5152afcddb62e55d74de2f8c43877780c809bd8f65e7a597a365372dd
and there also are two unused data blobs
restic -r /run/media/xxxxxx/BACKUP1/backup_restic/ check --check-unused
using temporary cache in /tmp/restic-check-cache-223469466
enter password for repository:
repository 1ddea405 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-223469466
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree 6b414ec5:pshots
decrypting blob 6b414ec5152afcddb62e55d74de2f8c43877780c809bd8f65e7a597a365372dd failed: ciphertext verification failed
[0:00] 100.00% 4 / 4 snapshots
unused blob <data/5b71e606>
unused blob <data/d48d66d5>
Fatal: repository contains errors
I did a restore to another disk and run a full diff against the source and found no unexplainable differences.
So what could caused this ? What is “pshots” ? And what should I do with the repo now ? Fresh reinit ?
Not sure what it’s worth but my problem was on an encrypted ext4 drive too and kind of suddenly went away again.
One thing I understand now: the “fatal” error that --check-unused throws is not even an error but a standard condition. Since a while ago, restic doesn’t remove all unused blobs anymore to save time during prune.
along with your solution I tried to prune the unused blobs but the error message got worse:
restic -r /run/media/xxxxxx/BACKUP1/backup_restic/ prune --max-unused=0
enter password for repository:
repository 1ddea405 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 4 snapshots
[0:00] 25.00% 1 / 4 snapshots
decrypting blob 6b414ec5152afcddb62e55d74de2f8c43877780c809bd8f65e7a597a365372dd failed: ciphertext verification failed
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
github.com/restic/restic/internal/repository/repository.go:204
github.com/restic/restic/internal/repository.(*Repository).LoadTree
github.com/restic/restic/internal/repository/repository.go:728
github.com/restic/restic/internal/restic.loadTreeWorker
github.com/restic/restic/internal/restic/tree_stream.go:37
github.com/restic/restic/internal/restic.StreamTrees.func1
github.com/restic/restic/internal/restic/tree_stream.go:164
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57
runtime.goexit
runtime/asm_amd64.s:1374
This thread can help you I believe. There are multiple ideas in it. You can either find the snapshots which uses that blob or use mentioned recovery branch.
Please run check --read-data to verify that no further pack files are damaged. Then for recovery it is first necessary to remove the damaged tree from the repository before running any other repair steps. For that run restic find --show-pack-id --tree 6b414ec5152afcddb62e55d74de2f8c43877780c809bd8f65e7a597a365372dd and then locate the pack file whose id is reported like this:
Object belongs to pack de473f08368b0c18a82992c9643e198c40f38ff8e85bdea42a844e6b0a907f11
Move that pack file out of the data folder of the repository, but keep the file for now. Then run restic rebuild-index. Afterwards it’s a good idea to run restic backup again to let restic try to recover missing blobs. Then run restic check to check which blobs from the removed pack file are still missing. The way forward would then be to either delete affected snapshots, use the repair command or to follow the steps from Data blobs seem to be missing, aborting prune to prevent further data loss! · Issue #3023 · restic/restic · GitHub to extract the still valid blobs from the damaged pack file.
That problem is sometimes caused by bitflips in memory, so running a memcheck is advisable. Other causes would be harddisk problems or also software bugs.
“pshots” is just a slightly garbled remainder of the “4 / 4 snapshots” progress bar.
The “prune” command does not work for damaged repositories, as long as the damaged part is still required for a snapshot in the repository. That is the error is expected in this case. Oh, and prune failed during the initial integrity check of the repository, which means that there haven’t been any changes to the repository.
Removing the corrupted data file made it worse and produced more error which didn’t disappear after rebuilding the indexes.
Luckily your tips helped me to track down the specific snapshot and I could remove it. After creating a new one via “backup” and a intensive “check” with “–read-data” brings no errors any more, the repo seems fine now.
I added some calls of “check” to my backup script so I can keep an eye more closely if this will happen again.