Unfixable not found in index (fixed) / not found in repo issues on prune

Getting Help
1

Before i start: Restic is a cool tool and i am already using it since the early days! :slight_smile:

I have two issues each on independent repos (and independent source/target):

1. not found in index (fixed!):

This is from a pretty big backup (1,7TB). I can backup without problems, but when i run prune i get this:

# restic prune
repository b08be969 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 2 snapshots
[0:31] 100.00%  2 / 2 snapshots
searching used packs...
{<data/2e2cf236> <data/39cbdcbd> <data/3db90903> <data/6e0b9d19> <data/799c304d>} not found in the index

Integrity check failed: Data seems to be missing.
Will not start prune to prevent (additional) data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/choose
Fatal: index is not complete

I tried “restic rebuild-index” already, but that did not solve the issue.

Fixed: Another backup run fixed it, now all is green and restic prune/check is happy… Don’t know why, but i like it :slight_smile:

2. not found in repository

The data to backup is about 180GB and over a slow connection (16mbit), however the backups usually are succesfull.
When i run prune there, i get:

# restic prune
repository 60ae9cc1 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 14 snapshots
[0:20] 78.57%  11 / 14 snapshots
id f07257addde62d4b0d3cad7ec14bc67b4f90cf04bfe5953c7e7e7992fa21f0c2 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
	/restic/internal/repository/repository.go:162
github.com/restic/restic/internal/repository.(*Repository).LoadTree
	/restic/internal/repository/repository.go:728
github.com/restic/restic/internal/restic.loadTreeWorker
	/restic/internal/restic/tree_stream.go:37
github.com/restic/restic/internal/restic.StreamTrees.func1
	/restic/internal/restic/tree_stream.go:164
golang.org/x/sync/errgroup.(*Group).Go.func1
	/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
	/usr/local/go/src/runtime/asm_arm.s:841

additional info

Both setups use up-to-date restic clients (0.12.1). I can not completely exclude it, but both targets are raided linux filesystem based storages. File loss is not known there so far, at least that would very much surprise me.
What indeed happened at least to the backups in 2. is that the backup itself got killed while doing the backup, either to a bad connection or by the backing up device being switched off. According to the restic docs an incomplete backup run is not a big deal, it will rerun from where it left until the cancellation. Is that wrong?
Both backups use the restic-rest-server docker container at the target by the way and speak to it via HTTPS (if that helps in any way).

Can you help me? Thanks in advance! :slight_smile:

2

Addition: On 2. i also tried rebuild-index, but that did not change anything :frowning:

3 
id f07257addde62d4b0d3cad7ec14bc67b4f90cf04bfe5953c7e7e7992fa21f0c2 not found in repository

This looks like a file on repository vanished somehow…

What indeed happened at least to the backups in 2. is that the backup itself got killed while doing the backup, either to a bad connection or by the backing up device being switched off. According to the restic docs an incomplete backup run is not a big deal, it will rerun from where it left until the cancellation. Is that wrong?

That should not be an issue.

I could suggest maybe running a backup --force to see if the file will appear on the repo again (that’s just a theory though, if things change fast, low chance).

Worst case I can suggest the repair branch, but I don’t want to give bad/untested subcommand advice. Maybe someone else will have an idea.

4

The backup command checks during backup that all necessary blobs for a file are in the repository (by looking at the index, which is why rebuild-index has helped here). If blobs are missing then it will automatically add those back to the repository. Note if that was the case then restic printed a warning and exited with a non-zero exit code. This of course only works if the missing data still exists on disk.

That probably won’t help for a missing tree blob, but it’s worth a try a running rebuild-index first. The alternative is to remove the affected snapshot. You can use the command restic find --tree f07257addde62d4b0d3cad7ec14bc67b4f90cf04bfe5953c7e7e7992fa21f0c2 to identify the snapshot and then delete it by passing the snapshot id to restic forget <snapshot-id>.

@gurkan The currently recommended steps to repair a repository are posted at Recover from broken pack file · Issue #828 · restic/restic · GitHub .
@timdeluxe Please run restic check or preferably restic check --read-data afterwards to make sure the repository is intact. Note that the latter command will have to download the whole repository.

1 Like
5

@MichaelEischer
The restic find --tree together with the forget of the affected snapshot was really helpful - it needed another rebuild-index and another backup, but now the prune worked and also check is not complaining anymore.
Thank you very much!!
@gurkan backup --force did not help, as MichaelEischer already assumed. However thank you for trying to help anyway! Really appreciate it.

Thanks guys, i am happy to have to clean / working backups again and again happy to use restic. :slight_smile: