Repository broken - id not found in repository

Hi there,

I know there’s other topics covering this (similar) case but I didn’t found the exact situation like mine so here I go with this thread.

restic prune is not working as it’s returning an id not found in the repository.

repository 2461be36 opened (version 1)
loading indexes...
loading all snapshots...
finding data that is still in use for 30 snapshots
[0:00] 0.00%  0 / 30 snapshots
id 01687380b40c74ec0dcebc50acaffec730ac0c4a335265272e9a1b6eb542baa1 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
	/restic/internal/repository/repository.go:274
github.com/restic/restic/internal/restic.LoadTree
	/restic/internal/restic/tree.go:113
github.com/restic/restic/internal/restic.loadTreeWorker
	/restic/internal/restic/tree_stream.go:36
github.com/restic/restic/internal/restic.StreamTrees.func1
	/restic/internal/restic/tree_stream.go:176
golang.org/x/sync/errgroup.(*Group).Go.func1
	/home/build/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1598

We tried the following commands (sorted by time)

restic check returns lots of file or directory not found.
restic rebuild-index didn’t help
restic prune again to check, but did nothing
restic backup --force did nothing new
restic prune to see if backup force helped, but nope…
restic rebuid-index --read-all-packls didn’t help neither
restic prune again, nothing worked.

the server is a 24x7 online server in a CPD that mades daily backups, one of them copied with restic from a remote server (using restic-server for the repo access via -r)
We’ve checked server, dmesg, smartctl, and we see no errors at all. Doesn’t look like a problem in t he fs or hardware.

anyone has any clue what’s going on? how can I Get rid of those issues?

also, in case there’s nothing to do, how can I remove all the snapshots in one comand to start with the same repository from scratch?

Thanks in advance.

When reporting a problem, please include the relevant info such as the exact command you run including any env vars it uses, and in this case I think the output from the check command would be good to look at. Also include obvious things like what restic and rest-server versions you run :slight_smile:

Hi,

My apologies, replying inline:

Those commands are ran on the restic’s server so it’s basically

restic -r /path/to/repo --password-file /path/to/password/file prune
restic -r /path/to/repo --password-file /path/to/password/file rebuild-index
restic -r /path/to/repo --password-file /path/to/password/file rebuild-index read-all-packs
…etc, the same list of commands I’ve added before. I’m not adding anything else.

lots of:

   id xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx not found in repository

and lots of

error for tree xxxx
  tree  xxxxxxxxxxx file yyyy not found in in dex

and then

Fatal: repository contains errors
restic version
restic 0.15.2 compiled with go1.20.3 on linux/amd64

rest-server running on docker restic/rest-server:0.12.0

Did you notice any errors reported in prior restic runs (before the failure) or did something unexpected happen? Maybe an interrupted network connection?

Do you have logs of the backup and prune before the first failed prune? Did restic report any errors there?

How often do you run backup, check and/or prune?

Is the repository used by multiple hosts or just a single host?

Do you use the unlock command or is it called automatically from your backup scripts?

What did the first rebuild-index run print?

The problem is that some tree blobs are missing, so prune cannot determine what data is still needed and what not.

Please try to locate the affected snapshot(s) by running restic find --tree 01687380b40c74ec0dcebc50acaffec730ac0c4a335265272e9a1b6eb542baa1.

Judging from the amount of missing tree blobs, at least one file in the data folder of the repository was lost. It could affect either a single or multiple snapshots.

To repair the repository, just either have to delete the broken snapshots (which means finding out which ones are affected first, see Recover from broken pack file · Issue #828 · restic/restic · GitHub Route 2), or you can use a beta version of restic (restic beta releases (/)) which can automatically repair damaged snapshots by removing the broken parts (see Troubleshooting — restic 0.15.2 documentation)
Recover from broken pack file · Issue #828 · restic/restic · GitHub