Hi I am not sure if this is related but my issue looks very similar.
I have attempted to run prune and I run into the following errors. I have copy pasted an excerpt of the message I see on screen
incomplete pack file (will be removed): fe74838f41ec8aed7777b95a4f043218a22fed6098be633797482128f4d8c21f
repository contains 54947 packs (432865 blobs) with 261.044 GiB
processed 432865 blobs: 4171 duplicate blobs, 4.262 GiB duplicate
load all snapshots*
find data that is still in use for 8 snapshots
Load(<data/0027d9a403>, 2410, 1409604) returned error, retrying after 745.465671ms: open /media/sf_restic/data/00/0027d9a403747b221936ca78274c673c087b09210b2a2fac987baf6faf0d162c: no such file or directory
Load(<data/0027d9a403>, 2410, 1409604) returned error, retrying after 829.067344ms: open /media/sf_restic/data/00/0027d9a403747b221936ca78274c673c087b09210b2a2fac987baf6faf0d162c: no such file or directory
Fatal: unable to load a tree from the repo: open /media/sf_restic/data/00/0027d9a403747b221936ca78274c673c087b09210b2a2fac987baf6faf0d162c: no such file or directory
I am not sure if this is related, but I absentmindedly deleted some of the data from the repository without without being able to react fast enough to stop the process. Ever since I have made new snapshots and have deleted the snapshots created before I damaged the repository. The backups do not throw any error and I am able to restore some files and browse using FUSE but when I perform a check or prune it throws errors but nothing gets removed.
@zavedf Iâm pretty sure I know what happened here.
While you did delete the snapshots made before you accidentally deleted the files, that didnât actually fix anything. Restic is operating with bad indexes because they reference data objects that arenât in the repository anymore, but newer backups still think they are. Hence, these objects are getting deduplicated, even though the original is gone.
This means that most likely all of the new snapshots are also damaged because they reference an object that doesnât exist.
The correct fix is to run restic rebuild-index as soon as possible, so that the missing objects are removed from the index. After doing this, subsequent backups will be undamaged.
Future backups have the potential to restore the objects missing from the older backups, so I would not run prune until a week or two of new snapshots has had a chance to bring in objects that the older snapshots depended on. Then, you can re-run prune. Youâll likely still get some errors at that time, but we can fix those by finding the damaged snapshots and specifically dropping them.
I did run a restic rebuild-index soon after I posted my comment. Thank you for explaining how the application works after the files in the repository are deleted, I will remember to rebuild the index should I face this issue.
Do you recon it would be a good idea to rebuild index periodically? Say once a month or so? Is there a way to check integrity of the backups in comparison with the source files? Does ârestic checkâ verify the backup integrity? As in will it check if the files themselves will be recoverable or will it only verify the availability of the index files?
If I would like to write a script to automate checks what exit code should I look out for?
This is not necessary unless you are deleting repository files directly instead of using restic to do it, or if the data files become damaged.
Not against the source files, because the source files presumably change so this wouldnât be very useful. HoweverâŚ
A bare restic check only verifies connectivity of objects â that is, every object that should be in the repository is actually there, but it doesnât check that objects are intact. restic check --read-data will, however, fully read all objects in the repository and make sure that their hash matches their object ID (which is their hash at the time they were entered into the repository).
The tl;dr:
restic check â All the objects that should be there are there.
restic check --read-data â All the objects that should be there are there and not damaged.
I believe both operations implicitly rebuild the index.
IIRC any non-zero output of restic check means there might be problems.
Awesome! Thank you for the in-depth response.
I can see myself using restic check --read-data from time to time, thank you so much for highlighting this option.
Thatâs not the case, restic check only reports but does not change the repo (except adding a lock file).
Exactly, if it exits non-zero, thereâs something you need to do.
Thereâs also a way to regularly read all data back from the repo, but nothing at once: pass --read-data-subset 1/10 to restic check, for that particular run restic will read the first tenth of all data files. On the next run, pass --read-data-subset 2/10 and itâll read the next tenth. After running with --read-data-subset 10/10, restic will have read almost all data in the repo.
This way, you can split reading all the files in up to 256 runs.
I had just had a chance to perform a cleanup and I get errors when I run the restic prune.
Basically I have done rebuild-index, backed up latest changes, performed a prune (in that order) and everything seemed ok until prune threw the following errors:
========================================================
PS C:\Users\User> restic -r W:\restic\ rebuild-index
enter password for repository:
repository a6a81653 opened successfully, password is correct
counting files in repo
[29:52] 100.00% 55042 / 55042 packs
finding old index files
saved new indexes as [32292f56 93dbd91d 69861b50 7319c3b0 6857e992 637a3587 bfa489fd ea9d3638 450a1111 150e8c4b 02646ac5 cc2b3eeb 884d9cca 2c737c9e c846d7f9 5b8138e8 356e99dd 6515333a c72f3ad4]
remove 19 old index files
PS C:\Users\User> restic --exclude "D:\*RECYCLE*" --exclude "D:\Config.Msi*" --exclude "C:\Users\User\Documents\My Music*" --exclude "C:\Users\User\Documents\My Pictures*" --exclude "C:\Users\User\Documents\My Videos*" --exclude "D:\System Volume Information*" -r w:\restic\ backup C:\Users\User\Pictures C:\Users\User\Downloads C:\Users\User\Documents
D:\
open repository
enter password for repository:
repository a6a81653 opened successfully, password is correct
Files: 15 new, 2 changed, 265890 unmodified
Dirs: 0 new, 3 changed, 0 unmodified
Added to the repo: 11.484 MiB
processed 265907 files, 240.022 GiB in 1:46
snapshot e4a084b3 saved
PS C:\Users\User> restic -r W:\restic\ prune
enter password for repository:
repository a6a81653 opened successfully, password is correct
counting files in repo
building new index for repo
[30:24] 100.00% 55045 / 55045 packs
incomplete pack file (will be removed): 00b95d3dbe2837cea815ea7a581b67076beffaaa38086a61242080aed2ed98c9
incomplete pack file (will be removed): 01adbbb734c69f44edc28a6eb9ae32af12066c88a051c0d1373fb0b6e31394b5
incomplete pack file (will be removed): 05f8b33c3cf34b810674ef1ba74aed7b1a8bac124b7ac66c68acb317668698ae
incomplete pack file (will be removed): 08471b76322a1eec5ac4a449b2a85268acf0ad9fa87710ce513b2273d67a6a77
repository contains 54950 packs (432899 blobs) with 261.057 GiB
processed 432899 blobs: 4171 duplicate blobs, 4.262 GiB duplicate
load all snapshots
find data that is still in use for 9 snapshots
tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
/restic/internal/repository/repository.go:653
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:11
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:31
main.pruneRepository
/restic/cmd/restic/cmd_prune.go:191
main.runPrune
/restic/cmd/restic/cmd_prune.go:85
main.glob..func17
/restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
/restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
/restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
/restic/cmd/restic/main.go:86
runtime.main
/usr/local/go/src/runtime/proc.go:201
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333
PS C:\Users\User>
========================================================
I am not sure what to make of this.
Restic version is: restic 0.9.4 compiled with go1.11.4 on windows/amd64
Windows version is Win 10 Home
*Note: some sections of the output has been removed to keep the post reasonably small.
So a specific tree (directory) object is still missing. You can figure out what snapshot(s) reference it with restic find --tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476.
@zavedf My suspicion is that the snapshot that references the missing tree is a snapshot that was created before you manually deleted some data files from the repository. As mentioned previously, running rebuild-index should ensure that future backups donât try to deduplicate with missing objects, so any snapshots taken since you ran that command should be intact. We can verify this after this find bug is fixed.
@cdhowie thank you for looking into this and noted on the future backups not referencing the missing index files.
I will look forward to this fix when available. In the mean time please let me know if prune would work with other forgotten repositories? I am unable to verify the results since I mostly see the same error on screen.
At the moment I am still open to deleting the original repository and creating a fresh install if this is the only way since the amount of data I am backing up is not massive.
I assume by ârepositoriesâ you meant âsnapshots.â If so, the answer is probably no, not until we can figure out which snapshot(s) reference the missing object(s).
This should not be necessary⌠basically you have a repository that canât be pruned or checked at the moment, but it should still correctly store new backups.
If you want to verify this, you can try restoring your most recent backup into a new directory temporarily, and make sure that the operation completes successfully.
I have a PR that fixes restic find aborting when a tree canât be loaded. Instead it will show you the tree ID and the snapshot itâs looking in, the implication being that if you forget the snapshot, the tree isnât needed anymore.