I get the same issue after force exit the prune stage (cause i realized i made a mistake and too much would be pruned). The checksum on the (sftp) server of 4 index files is now invalid. Is it expected that killing prune will corrupt index files? I would think it uses temp files and rename them afterwards. Anyway, is there some way to fix this or should i start with a clean backup? Fortunately not a huge problem if not possible, but was wondering what will happen if somehow the prune stage gets killed under abnormal circumstances (eg. OOM or power outage), should be safe?
1 Like
No, it is not expected… Which version of restic are you using? 0.12.0 brought a lot of changes within the prune command…
Yes I’m using 0.12.0. I’ve managed to fix this by restoring the snapshots and index dir from a ZFS snapshot, which fortunately is accessible through SFTP and is much faster to down/upload than the whole restic data dir (stupid SFTP doesn’t support renaming/moving and this is the only way i can access this backup target). Prune and check worked afterwards and just lost a single day, so it was an interesting experiment as well. Wouldn’t have been a disaster if it didn’t work.
Still somewhat concerning it got corrupted by simply interrupting the prune stage.
@jvandenbroek Do you still have the index files from the interrupted prune at hand? Might be you discovered a bug and I would really appreciate if you could help to track this.
Prune in 0.12.0 works like this:
- create a lock file
- reads the index, scans all files and determines what to do (this is all read-only)
- remove unneded and unreferenced pack files (this only affects pack files)
- repack pack files (i.e. creating new pack files from the ones to repack; this is write-only and only writes pack files)
- save the new index from information which is all in memory (this is write-only)
- remove the old index files
- remove the now unneeded pack files
- remove the lock file
So, to possibly create damaged index files you have to abort pruning during step 5 (or maybe 6. but this is IMO much more unlikely).
If you still have the damaged repo, can you give me the error that is printed when running restic with these corrupt index files?
1 Like
This error occurred to me today. In my case, it was due to an interrupted backup/prune process. I sometimes CTR+C the process due to various reasons.
So the errors could occur due to interruption (incompletely written packs etc).
Restic rebuild-index still returns the same error. Deleting local cache doesn’t help. Deleting the index file in remote that doesn’t pass the sha256sum test does not help either (restic picks up the latest index file and prints the same error).
I therefore removed all index files in remote, cleared local cache, and ran backup again. Hoping for a newly generated index (I will report back).
But I am not sure why rebuild-index did not work though. Restic should ignore current index files and create a new one. It keeps reading old index files.
Update
I confirm that deleting all index files on remote, clearing local cache, running backup, rebuild-index, and backup-prune solved the problem. A new index file was generated.
On another separate note: Yesterday I mistakenly deleted 50 pages of a work related file. Restic saved me one month of work: I simply grabbed the latest version from a previous snapshot.
Happy Resticing!
2 Likes
I can confirm this…just stumbled upon this error too and for me it seemed several index files were wrong. Both on server as in cache gave wrong sha back. Anyway removed all the index files (on server), removed the local cache completely and did a rebuild. After 3 hours or so the repo was back in shape again.
It would be worth a try to only remove the index files which don’t match their sha256 hash. That way rebuild-index can reuse the intact part of the index, which will be much faster.
The next restic release (probably 0.12.2) will include a few fixes which should prevent most instances of this error from occuring (Make cache concurrency-safe by greatroar · Pull Request #2838 · restic/restic · GitHub and Save files under temporary name in local backend by greatroar · Pull Request #3436 · restic/restic · GitHub ) and rebuild-index should be able to fix the problem (Fix `rebuild-index` for damaged index by MichaelEischer · Pull Request #3488 · restic/restic · GitHub).
@Eli6 rebuild-index uses the existing index to quickly rebuild the index. To start from scratch you have to run rebuild-index --read-all-packs.
1 Like