Repository damaged after accidentally deleting data files

[fd0]

@cdhowie

I believe both operations implicitly rebuild the index.

That’s not the case, restic check only reports but does not change the repo (except adding a lock file).

@cdhowie

IIRC any non-zero output of restic check means there might be problems.

Exactly, if it exits non-zero, there’s something you need to do.

@zavedf

I can see myself using restic check --read-data from time to time, thank you so much for highlighting this option.

There’s also a way to regularly read all data back from the repo, but nothing at once: pass --read-data-subset 1/10 to restic check, for that particular run restic will read the first tenth of all data files. On the next run, pass --read-data-subset 2/10 and it’ll read the next tenth. After running with --read-data-subset 10/10, restic will have read almost all data in the repo.

This way, you can split reading all the files in up to 256 runs.

[zavedf]

Thank you so much for your response, the --read-data-subset option is well though of! Very useful for large repositories!

[zavedf]

Hi,

I had just had a chance to perform a cleanup and I get errors when I run the restic prune.

Basically I have done rebuild-index, backed up latest changes, performed a prune (in that order) and everything seemed ok until prune threw the following errors:

========================================================
PS C:\Users\User> restic -r W:\restic\ rebuild-index
enter password for repository:
repository a6a81653 opened successfully, password is correct
counting files in repo
[29:52] 100.00%  55042 / 55042 packs
finding old index files
saved new indexes as [32292f56 93dbd91d 69861b50 7319c3b0 6857e992 637a3587 bfa489fd ea9d3638 450a1111 150e8c4b 02646ac5 cc2b3eeb 884d9cca 2c737c9e c846d7f9 5b8138e8 356e99dd 6515333a c72f3ad4]
remove 19 old index files
PS C:\Users\User> restic --exclude "D:\*RECYCLE*" --exclude "D:\Config.Msi*" --exclude "C:\Users\User\Documents\My Music*" --exclude "C:\Users\User\Documents\My Pictures*" --exclude "C:\Users\User\Documents\My Videos*" --exclude "D:\System Volume Information*" -r w:\restic\ backup C:\Users\User\Pictures C:\Users\User\Downloads C:\Users\User\Documents
D:\
open repository
enter password for repository:
repository a6a81653 opened successfully, password is correct

Files:          15 new,     2 changed, 265890 unmodified
Dirs:            0 new,     3 changed,     0 unmodified
Added to the repo: 11.484 MiB

processed 265907 files, 240.022 GiB in 1:46
snapshot e4a084b3 saved
PS C:\Users\User> restic -r W:\restic\ prune
enter password for repository:
repository a6a81653 opened successfully, password is correct
counting files in repo
building new index for repo
[30:24] 100.00%  55045 / 55045 packs
incomplete pack file (will be removed): 00b95d3dbe2837cea815ea7a581b67076beffaaa38086a61242080aed2ed98c9
incomplete pack file (will be removed): 01adbbb734c69f44edc28a6eb9ae32af12066c88a051c0d1373fb0b6e31394b5
incomplete pack file (will be removed): 05f8b33c3cf34b810674ef1ba74aed7b1a8bac124b7ac66c68acb317668698ae
incomplete pack file (will be removed): 08471b76322a1eec5ac4a449b2a85268acf0ad9fa87710ce513b2273d67a6a77
repository contains 54950 packs (432899 blobs) with 261.057 GiB
processed 432899 blobs: 4171 duplicate blobs, 4.262 GiB duplicate
load all snapshots
find data that is still in use for 9 snapshots
tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        /restic/internal/repository/repository.go:653
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:11
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /restic/internal/restic/find.go:31
main.pruneRepository
        /restic/cmd/restic/cmd_prune.go:191
main.runPrune
        /restic/cmd/restic/cmd_prune.go:85
main.glob..func17
        /restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333
PS C:\Users\User>
========================================================

I am not sure what to make of this.

Restic version is: restic 0.9.4 compiled with go1.11.4 on windows/amd64
Windows version is Win 10 Home

*Note: some sections of the output has been removed to keep the post reasonably small.

Thank you.

[cdhowie]

@zavedf

tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476 not found in repository

So a specific tree (directory) object is still missing. You can figure out what snapshot(s) reference it with restic find --tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476.

[zavedf]

Hi,
Thank you for your response. I have tried finding that tree but it throws a similar error.

PS C:\Users\User> restic -r W:\restic\ find --tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476
enter password for repository:
repository a6a81653 opened successfully, password is correct
tree cee759a90802904dcdbe733403dd452f8d709dc637d8fc97b966fc7c07838476 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        /restic/internal/repository/repository.go:653
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:107
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.walk
        /restic/internal/walker/walker.go:127
github.com/restic/restic/internal/walker.Walk
        /restic/internal/walker/walker.go:56
main.(*Finder).findIDs
        /restic/cmd/restic/cmd_find.go:341
main.runFind
        /restic/cmd/restic/cmd_find.go:549
main.glob..func9
        /restic/cmd/restic/cmd_find.go:33
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333

Regards

[cdhowie]

Seems you’ve hit this bug in the find command:

Unable to Check or Prune

Hmm, it appears that restic find gives up when it encounters an error. @fd0 maybe it would make sense to have a flag that allows it to skip objects that it can’t read? However, if this pack is damaged and the tree that contains tree 7bfdff4d is in the same pack, then we might be SOL…

@fd0, any word on the fix for this? Is it something I could help with?

[fd0]

Ah, no progress made here. It would help if you could add a GitHub for it, so we don’t forget. Thanks!

[cdhowie]

I’ve submitted the issue for tracking.

The find command should not fail when objects are missing from the repository

@zavedf My suspicion is that the snapshot that references the missing tree is a snapshot that was created before you manually deleted some data files from the repository. As mentioned previously, running rebuild-index should ensure that future backups don’t try to deduplicate with missing objects, so any snapshots taken since you ran that command should be intact. We can verify this after this find bug is fixed.

[zavedf]

@cdhowie thank you for looking into this and noted on the future backups not referencing the missing index files.

I will look forward to this fix when available. In the mean time please let me know if prune would work with other forgotten repositories? I am unable to verify the results since I mostly see the same error on screen.

At the moment I am still open to deleting the original repository and creating a fresh install if this is the only way since the amount of data I am backing up is not massive.

Thanks for all your help.

[cdhowie]

@zavedf

In the mean time please let me know if prune would work with other forgotten repositories?

I assume by “repositories” you meant “snapshots.” If so, the answer is probably no, not until we can figure out which snapshot(s) reference the missing object(s).

@zavedf

At the moment I am still open to deleting the original repository and creating a fresh install if this is the only way since the amount of data I am backing up is not massive.

This should not be necessary… basically you have a repository that can’t be pruned or checked at the moment, but it should still correctly store new backups.

If you want to verify this, you can try restoring your most recent backup into a new directory temporarily, and make sure that the operation completes successfully.

[zavedf]

@cdhowie

I assume by “repositories” you meant “snapshots.” If so, the answer is probably no, not until we can figure out which snapshot(s) reference the missing object(s).

Hi yes I meant snapshots sorry.

@cdhowie

This should not be necessary… basically you have a repository that can’t be pruned or checked at the moment, but it should still correctly store new backups.

If you want to verify this, you can try restoring your most recent backup into a new directory temporarily, and make sure that the operation completes successfully.

Noted on this, I will try to restore a few random files to see if everything is ok.

Thank you so much for your assistance!

[cdhowie]

I have a PR that fixes restic find aborting when a tree can’t be loaded. Instead it will show you the tree ID and the snapshot it’s looking in, the implication being that if you forget the snapshot, the tree isn’t needed anymore.

If you want to test it to try to fix your repository, you can build this branch: https://github.com/cdhowie/restic/tree/issue-2224

find command: don't abort on tree load errors

[zavedf]

Thank you for your response and for the fix. I will try to download this version on my VM to test and update you how it goes by end of this week.

[cdhowie]

This fix is now in the newly-released 0.9.5.

[zavedf]

That is awesome! Thank you for the update, I will go ahead and update my version in the soonest.

[zavedf]

Hi, I would like to check if there is instructions to updating restic in Windows Powershell installed via Scoop?
Scoop Update did not update Restic and restic self-update gets an error message
Fatal: unable to update restic: unable to remove target file: remove C:\Users\User\scoop\apps\restic\current\restic.exe: Access is denied.

Thanks

[cdhowie]

This issue is documented on the forum and on github.

Self-update fails: unable to remove target file

Trying to get the new version via self-update fails on me: C:\Temp>restic self-update writing restic to C:\_h\bin\restic.exe find latest release of restic at GitHub latest version is 0.9.5 download SHA256SUMS download SHA256SUMS.asc GPG signature verification succeeded download restic_0.9.5_windows_amd64.zip downloaded restic_0.9.5_windows_amd64.zip Fatal: unable to update restic: unable to remove target file: remove C:\_h\bin\restic.exe: Access is denied. Current version is restic 0.9.4 compi…

self-update doesn't work on Windows

[zavedf]

Thank you for your response. I have noticed the same error when I tried doing the self-update in my Linux VM. Restic was installed using the APT package manager.

[cdhowie]

@zavedf

I have noticed the same error when I tried doing the self-update in my Linux VM.

Were you running the command as root?

@zavedf

Restic was installed using the APT package manager.

It would be wise not to self-update a binary installed using your package manager.

[zavedf]

@cdhowie

Were you running the command as root?

Hi yes, I tried it as root and got the error.

@cdhowie

It would be wise not to self-update a binary installed using your package manager.

Noted on this. I will wait for the update to come via the package managers then. Thank you.