Hi, I started making backup with restic and I have some questions about the corrupted repo.
Suppose I have 2 backup repo backuping the same folder, one of them (backup_repoA) is corrupted and the other (backup_repoB) is fine. Is it possible to repair the corrupted repo using the other repo.
For example, if a file or some files from /backup_repoA/data/xxxx is lost, restoring using backup_repoA shows the following error:
Load(<data/ea7f1272fd>, 8388640, 0) returned error, retrying after 15.394871241s: open /tmp/backup_test/corrupt_test/data/ea/ea7f1272fd6474b0880d383c18b58bc8fd7eac08ce279f0515995adf72d48761: no such file or directory
ignoring error for /tmp/my_folder/data1.rar: open /tmp/backup_test/corrupt_test/data/ea/ea7f1272fd6474b0880d383c18b58bc8fd7eac08ce279f0515995adf72d48761: no such file or directory
Fatal: There were 1 errors
I checked other files and they are fine except the file data1.rar. Does it mean that restoring data1.rar from backup_repoB and replace data1.rar restored from backup_repoA with data1.rar restored from backup_repoB can generate a complete backup? Also, can I know which file is problematic before running the restore command?
In another case, suppose both backup_repoA and backup_repoB are corrupted, and there is different error message for each repo:
backup_repoA: ignoring error for /tmp/my_folder/data1.rar: open /tmp/backup_test/corrupt_test/data/xx/xxxxxx: no such file or directory
Fatal: There were 1 errors
backup_repoB: ignoring error for /tmp/my_folder/data3.rar: open /tmp/backup_test/corrupt_test/data/yy/yyyyyy: no such file or directory
Fatal: There were 1 errors
In this case, does it mean I can still get a complete backup if a use the data1.rar restored from backup_repoB and data3.rar restored from backup_repoA?
Also, if backup_repoA is corrupted and backup_repoB is fine, and then I get the following error when I run the check command:
load indexes
error: error loading index de30f323: load <index/de30f3231c>: invalid data returned
Fatal: LoadIndex returned errors
Suppose the file /backup_repo/index/xxxx is modified by someone, can I repair the backup_repoA? Or do I need to clear the backup_repoA and start a new backup?
A broken repository index can be repaired by running rebuild-index. Just make sure to use a recent restic version.
That entirely depends on the relation of the backups to each other the exact lost version of a file is contained in the other repository or not. Are those separate repositories? Or do the use the same chunker parameters (only the case if specified when running init)? Or were the two repos copies of each other?
The exact steps to repair a repository depend on the exact kind of damage. In your case apparently the storage backend (a local disk?) lost files. In that case you probably have to run rebuild-index and then afterwards either delete the damaged snapshots or if you still have the original data of the lost file somewhere then just run backup missing-file, which will recover the missing file chunks.
To verify a repository run the check command from time to time. In case of a repository damage please try to first find out what caused the problem before trying to repair it. e.g. if the storage backend looses data then that has to be fixed first before the backup can work reliably.
Thanks! I still have some question about the corrupted repo.
For repairing the corrupted repo using other repo, they are two separate repositories, and the chunker parameters is not specified when running init. In this case, is it possible to repair the corrupted repo using the other repo?
Also, there is one more issue. Suppose my original data is lost, and I have 2 corrupted repo (they are 2 separate repositories and the chunker parameters is not specified when running init). Before running the rebuild-index, I can restore the data with some error messages. However, after running the rebuild-index, the data cannot be restored. In this case, assume I restore the data from the 2 repos first, and then run the rebuild-index command, and then use the check --read data command and get the following error:
repo1:
tree xxxx: file “file_1” blob xxxx not found in index.
tree xxxx: file “file_2” blob xxxx not found in index.
repo2:
tree yyyy: file “file_3” blob yyyy not found in index.
tree yyyy: file “file_4” blob yyyy not found in index.
In this case, does it means only file_1 and file_2 are missing in backup_repo1 and only file_3 and file_4 are missing in backup_repo2? If I copy file_1 and file_2 from the data restored from backup_repo2 and copy them to replace the file_1 and file_2 restored from backup_repo1, can I say that I have the full data recover?
For restic these two repositories are independent of each other. If a file in one repo is damaged and the other repo has exactly this file (with identical content!) then it should be possible to repair the repository by first restoring the file from the intact repo and then running backup on the missing files (after a prior index rebuild).
If these are the only complaints, then only these four files are damaged in the snapshots.
You will end up having all four files, but depending on when you created the snapshots a part of the folder could be newer than the other files, which can cause problems. If the files haven’t changed in between the snapshots, then you should be able to repair both repositories by first running rebuild-index and then backup restored-data.
For deleting the damaged snapshot, suppose I get the following errors after running the check command:
error for tree 6668fde8:pshots…
tree 6668fde8: file “file_1” blob 30e10e1dfb997083cd902f10fa6936ef8c9c666622f6290cb6b5c572d15e6fbd not found in index
error for tree 70d5c1ad:
tree 70d5c1ad: file “file_2” blob 461a766778642bb11a513319552c70ec847af8e825c5786df427141bf83382bf not found in index
tree 70d5c1ad: file “file_2” blob 461a766778642bb11a513319552c70ec847af8e825c5786df427141bf83382bf not found in index
Suppose I used restic find --tree 6668fde8 and restic find --tree 70d5c1ad to find out the snapshot(s) and delete them, do I need to also run restic find --blob and check for the damaged snapshot?
The preferred solution is to repair the repo without the need to modify any snapshot. In the case of missing data blobs, this can be done (as @MichaelEischer pointed out) by backing up up the missing data again (just run a backup command to backup any dir which contains the missing data. This creates a new snapshot and “heals” all existing snapshots).
Only if you are missing data which you cannot get anymore AND still want to keep the sane parts of some snapshot(s), you can use the linked repair command.