Repository damaged after accidentally deleting data files

#1

Hi I am not sure if this is related but my issue looks very similar.

I have attempted to run prune and I run into the following errors. I have copy pasted an excerpt of the message I see on screen

incomplete pack file (will be removed): fe74838f41ec8aed7777b95a4f043218a22fed6098be633797482128f4d8c21f
repository contains 54947 packs (432865 blobs) with 261.044 GiB
processed 432865 blobs: 4171 duplicate blobs, 4.262 GiB duplicate
load all snapshots*
find data that is still in use for 8 snapshots
Load(<data/0027d9a403>, 2410, 1409604) returned error, retrying after 745.465671ms: open /media/sf_restic/data/00/0027d9a403747b221936ca78274c673c087b09210b2a2fac987baf6faf0d162c: no such file or directory
Load(<data/0027d9a403>, 2410, 1409604) returned error, retrying after 829.067344ms: open /media/sf_restic/data/00/0027d9a403747b221936ca78274c673c087b09210b2a2fac987baf6faf0d162c: no such file or directory
Fatal: unable to load a tree from the repo: open /media/sf_restic/data/00/0027d9a403747b221936ca78274c673c087b09210b2a2fac987baf6faf0d162c: no such file or directory

I am not sure if this is related, but I absentmindedly deleted some of the data from the repository without without being able to react fast enough to stop the process. Ever since I have made new snapshots and have deleted the snapshots created before I damaged the repository. The backups do not throw any error and I am able to restore some files and browse using FUSE but when I perform a check or prune it throws errors but nothing gets removed.

0 Likes

Unable to Check or Prune
#2

@zavedf I’m pretty sure I know what happened here.

While you did delete the snapshots made before you accidentally deleted the files, that didn’t actually fix anything. Restic is operating with bad indexes because they reference data objects that aren’t in the repository anymore, but newer backups still think they are. Hence, these objects are getting deduplicated, even though the original is gone.

This means that most likely all of the new snapshots are also damaged because they reference an object that doesn’t exist.

The correct fix is to run restic rebuild-index as soon as possible, so that the missing objects are removed from the index. After doing this, subsequent backups will be undamaged.

Future backups have the potential to restore the objects missing from the older backups, so I would not run prune until a week or two of new snapshots has had a chance to bring in objects that the older snapshots depended on. Then, you can re-run prune. You’ll likely still get some errors at that time, but we can fix those by finding the damaged snapshots and specifically dropping them.

1 Like

#3

Hi @cdhowie thank you for your response.

I did run a restic rebuild-index soon after I posted my comment. Thank you for explaining how the application works after the files in the repository are deleted, I will remember to rebuild the index should I face this issue.

Do you recon it would be a good idea to rebuild index periodically? Say once a month or so? Is there a way to check integrity of the backups in comparison with the source files? Does “restic check” verify the backup integrity? As in will it check if the files themselves will be recoverable or will it only verify the availability of the index files?

If I would like to write a script to automate checks what exit code should I look out for?

Thank you

1 Like

#4

This is not necessary unless you are deleting repository files directly instead of using restic to do it, or if the data files become damaged.

Not against the source files, because the source files presumably change so this wouldn’t be very useful. However…

A bare restic check only verifies connectivity of objects – that is, every object that should be in the repository is actually there, but it doesn’t check that objects are intact. restic check --read-data will, however, fully read all objects in the repository and make sure that their hash matches their object ID (which is their hash at the time they were entered into the repository).

The tl;dr:

  • restic check – All the objects that should be there are there.
  • restic check --read-data – All the objects that should be there are there and not damaged.

I believe both operations implicitly rebuild the index.

IIRC any non-zero output of restic check means there might be problems.

1 Like

#5

Awesome! Thank you for the in-depth response.
I can see myself using restic check --read-data from time to time, thank you so much for highlighting this option.

Noted on the exit code options. Thank you.

1 Like

#6

That’s not the case, restic check only reports but does not change the repo (except adding a lock file).

Exactly, if it exits non-zero, there’s something you need to do.

There’s also a way to regularly read all data back from the repo, but nothing at once: pass --read-data-subset 1/10 to restic check, for that particular run restic will read the first tenth of all data files. On the next run, pass --read-data-subset 2/10 and it’ll read the next tenth. After running with --read-data-subset 10/10, restic will have read almost all data in the repo.

This way, you can split reading all the files in up to 256 runs.

0 Likes

#7

Thank you so much for your response, the --read-data-subset option is well though of! Very useful for large repositories!

0 Likes

#8

Hi,

I had just had a chance to perform a cleanup and I get errors when I run the restic prune.

Basically I have done rebuild-index, backed up latest changes, performed a prune (in that order) and everything seemed ok until prune threw the following errors:

========================================================
PS C:\Users\User> restic -r W:\restic\ rebuild-index
enter password for repository:
repository a6a81653 opened successfully, password is correct
counting files in repo
[29:52] 100.00%  55042 / 55042 packs
finding old index files
saved new indexes as [32292f56 93dbd91d 69861b50 7319c3b0 6857e992 637a3587 bfa489fd ea9d3638 450a1111 150e8c4b 02646ac5 cc2b3eeb 884d9cca 2c737c9e c846d7f9 5b8138e8 356e99dd 6515333a c72f3ad4]
remove 19 old index files
PS C:\Users\User> restic --exclude "D:\*RECYCLE*" --exclude "D:\Config.Msi*" --exclude "C:\Users\User\Documents\My Music*" --exclude "C:\Users\User\Documents\My Pictures*" --exclude "C:\Users\User\Documents\My Videos*" --exclude "D:\System Volume Information*" -r w:\restic\ backup C:\Users\User\Pictures C:\Users\User\Downloads C:\Users\User\Documents
D:\
open repository
enter password for repository:
repository a6a81653 opened successfully, password is correct

Files:          15 new,     2 changed, 265890 unmodified
Dirs:            0 new,     3 changed,     0 unmodified
Added to the repo: 11.484 MiB

processed 265907 files, 240.022 GiB in 1:46
snapshot e4a084b3 saved
PS C:\Users\User> restic -r W:\restic\ prune
enter password for repository:
repository a6a81653 opened successfully, password is correct
counting files in repo
building new index for repo
[30:24] 100.00%  55045 / 55045 packs
incomplete pack file (will be removed): 00b95d3dbe2837cea815ea7a581b67076beffaaa38086a61242080aed2ed98c9
incomplete pack file (will be removed): 01adbbb734c69f44edc28a6eb9ae32af12066c88a051c0d1373fb0b6e31394b5
incomplete pack file (will be removed): 05f8b33c3cf34b810674ef1ba74aed7b1a8bac124b7ac66c68acb317668698ae
incomplete pack file (will be removed): 08471b76322a1eec5ac4a449b2a85268acf0ad9fa87710ce513b2273d67a6a77
repository contains 54950 packs (432899 blobs) with 261.057 GiB
processed 432899 blobs: 4171 duplicate blobs, 4.262 GiB duplicate
load all snapshots
find data that is still in use for 9 snapshots
tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        /restic/internal/repository/repository.go:653
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:11
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
main.pruneRepository
        /restic/cmd/restic/cmd_prune.go:191
main.runPrune
        /restic/cmd/restic/cmd_prune.go:85
main.glob..func17
        /restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333
PS C:\Users\User>
========================================================

I am not sure what to make of this.

Restic version is: restic 0.9.4 compiled with go1.11.4 on windows/amd64
Windows version is Win 10 Home

*Note: some sections of the output has been removed to keep the post reasonably small.

Thank you.

0 Likes

#9

So a specific tree (directory) object is still missing. You can figure out what snapshot(s) reference it with restic find --tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476.

0 Likes

#10

Hi,
Thank you for your response. I have tried finding that tree but it throws a similar error.

PS C:\Users\User> restic -r W:\restic\ find --tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476
enter password for repository:
repository a6a81653 opened successfully, password is correct
tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        /restic/internal/repository/repository.go:653
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:107
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.Walk
        /restic/internal/walker/walker.go:56
main.(*Finder).findIDs
        /restic/cmd/restic/cmd_find.go:341
main.runFind
        /restic/cmd/restic/cmd_find.go:549
main.glob..func9
        /restic/cmd/restic/cmd_find.go:33
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333

Regards

0 Likes

#11

Seems you’ve hit this bug in the find command:

@fd0, any word on the fix for this? Is it something I could help with?

0 Likes

#12

Ah, no progress made here. It would help if you could add a GitHub for it, so we don’t forget. Thanks!

0 Likes

#13

I’ve submitted the issue for tracking.

@zavedf My suspicion is that the snapshot that references the missing tree is a snapshot that was created before you manually deleted some data files from the repository. As mentioned previously, running rebuild-index should ensure that future backups don’t try to deduplicate with missing objects, so any snapshots taken since you ran that command should be intact. We can verify this after this find bug is fixed.

1 Like

#14

@cdhowie thank you for looking into this and noted on the future backups not referencing the missing index files.

I will look forward to this fix when available. In the mean time please let me know if prune would work with other forgotten repositories? I am unable to verify the results since I mostly see the same error on screen.

At the moment I am still open to deleting the original repository and creating a fresh install if this is the only way since the amount of data I am backing up is not massive.

Thanks for all your help.

0 Likes

#15

I assume by “repositories” you meant “snapshots.” If so, the answer is probably no, not until we can figure out which snapshot(s) reference the missing object(s).

This should not be necessary… basically you have a repository that can’t be pruned or checked at the moment, but it should still correctly store new backups.

If you want to verify this, you can try restoring your most recent backup into a new directory temporarily, and make sure that the operation completes successfully.

0 Likes

#16

Hi yes I meant snapshots sorry.

Noted on this, I will try to restore a few random files to see if everything is ok.

Thank you so much for your assistance!

1 Like

#17

I have a PR that fixes restic find aborting when a tree can’t be loaded. Instead it will show you the tree ID and the snapshot it’s looking in, the implication being that if you forget the snapshot, the tree isn’t needed anymore.

If you want to test it to try to fix your repository, you can build this branch: https://github.com/cdhowie/restic/tree/issue-2224

1 Like

#18

Thank you for your response and for the fix. I will try to download this version on my VM to test and update you how it goes by end of this week.

1 Like

#19

This fix is now in the newly-released 0.9.5.

1 Like

#20

That is awesome! Thank you for the update, I will go ahead and update my version in the soonest.

0 Likes