When I run this:
restic forget \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 3 \
--prune \
-r $TARGET
I am getting error saying data is missing:
loading all snapshots...
finding data that is still in use for 9 snapshots
[0:02] 100.00% 9 / 9 snapshots
searching used packs...
{<data/4ac89084> <data/4e3fd177> <data/65c364ef> <data/a61dca20> <data/b56a120e> <data/e8c54897>} not found in the index
Integrity check failed: Data seems to be missing.
Will not start prune to prevent (additional) data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/chooseFatal: index is not complete
Backups seem to run fine and I am just wondering how to re-create the data files to avoid this error going forward.
Addendum, when I run restic check then I get another fatal error…
using temporary cache in /tmp/restic-check-cache-180987829
repository 2527be8c opened successfully, password is correct
created new cache in /tmp/restic-check-cache-180987829
create exclusive lock for repository
load indexes
check all packs
pack af40fb6d: not referenced in any index
pack eedcf3db: not referenced in any index
2 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree f927ac79:pshots
tree f927ac79: file "filename.xls" blob b56a120eba883f6c0e3a0904f1b3e1a0c05b66996fdedbc780aefd339940cc06 not found in index
tree f927ac79: file "filename.xls" blob 65c364eff6349283c93aa06d82cb0db54dc6bd4d2a36d40fbcf727324134abb9 not found in index
tree f927ac79: file "filename.xls" blob 4e3fd1770ca3211aaaa5f3cbd4fd552bfe0cb2d42b41160ca9a025d3fa88ae59 not found in index
tree f927ac79: file "filename.xls" blob 4ac890842be844fd7b97fd0dd7ffedbf99b4b795336c3af774783015445eaf63 not found in index
tree f927ac79: file "filename.xls" blob e8c54897bafa7bce3a4dedc30be1faf4d2ccad2580e51b8615baa5a7dd5fce3f not found in index
tree f927ac79: file "filename.xls" blob a61dca2091cc3c1963305e87f08dff855829276cc00489b55ef2c6651114926a not found in index
[2:59] 100.00% 9 / 9 snapshots
Fatal: repository contains errors
Mhh - you have blobs that are missing in the index. Did you run a rebuild-index in the past? If yes, this could mean you had lost pack files. If no, you lost index files.
In any case you should try to find the root cause for this loss. If this occurs again, you might lose the data in your backup that you desperately need…
It might be also an undiscovered bug in restic, in this case tracking this may not only save you but also other users!
To bring back your repository in a good state: As you use 0.12.0, you just need to run a backup that contains the file "filename.xls". This will automatically heal your repo. After that, prune should run and clean everything up.
the weekly backup task ran and after a prune; I am no longer getting this problem any more. Will keep an eye out and if see again then figure out to repro so can hunt down bugs
help! Similar odd problems - Errors on various commands detailed below. Please help with suggestions on how to recover as the repo is kinda large and would take a week to reupload so I am keen to reuse if possible. Tried check/rebuild index as per below. Any other suggestions on commands to do and/or manual edits to repo?
output from backup
open repository
lock repository
load index files
Fatal: load <index/25459c4d29>: invalid data returned
output from rebuild-index
loading indexes...
Fatal: load <index/25459c4d29>: invalid data returned
output from check
using temporary cache in /tmp/restic-check-cache-825257802
created new cache in /tmp/restic-check-cache-825257802
create exclusive lock for repository
load indexes
pack ff4df95a contained in several indexes: {a938407e c1a73b83}
pack b56b17fc contained in several indexes: {16a752a3 b91129b9}
pack 4341e016 contained in several indexes: {418bcaef bd9ff572}
pack 259d9f34 contained in several indexes: {3cd484a5 cf0f08d1}
[[[ 70000+ rows of the above]]]
error: error loading index 25459c4d: load <index/25459c4d29>: invalid data returned
error: error loading index 52d4d7ee: load <index/52d4d7eed6>: invalid data returned
[[[ the above appears 4 times ]]]
This is non-critical, you can run `restic rebuild-index' to correct this
output from prune (no errors)
Applying Policy: keep 7 daily, 4 weekly, 3 monthly snapshots
keep 1 snapshots:
ID Time Host Tags Reasons Paths
----------------------------------------------------------------------------------------
7c5f3c86 2021-05-19 23:24:23 CORE-PC daily snapshot /mnt/z/UserData
weekly snapshot
monthly snapshot
----------------------------------------------------------------------------------------
1 snapshots
@kellytrinh You should first try to find out why your data got corrupted.
Can you give more information on the storage and storage backend you are using?
The problem you seem to have is corrupt index files.
Can you download these files and check the SHA256 sum? It should be equal to the filename but most likely isn’t. Or are these files even of zero length?
To repair these index files you can remove them, followed by a rebuild-index run. But be aware that you might have other corrupt files which you have not detected yet.
You should run a check --read-data and try to find out what happened to the now-corrupted files!
Hi Alex
on the how data corrupted - I am using a rclone backend to my cloud storage provider (pcloud). There may be corruption because around the time; I was doing some heavy network copy and my wifi router crashed (altho I would have hoped that restic/rclone would not have such a severe failure in the face of hardware issues).
checking out the index files - there is 4 of them corrupted; all pretty big size; 2-4.5MB or so. Given corruption, deleted and then ran rebuild index and things seem to work now!
Restic and rclone already provide the expected upload file size to pCloud. rclone even sets the parameter that incomplete uploads should not be saved (rclone/pcloud.go at 22e86ce3359a34f299aedd9d11a869ec0fecda74 · rclone/rclone · GitHub) That should allow pCloud to detect interrupted uploads. AFAICT the uploads also use HTTPS, which basically prevent data corruption during transmission.
So this rather looks like a pcloud problem to me.