So I did mess it up... How can I fix it? :( [very sad face]

Hello everyone!

I am receiving a lot o messages from my personal backup/repository:

restic stats

repository 6d9f087d opened successfully, password is correct
scanning…
walking tree 02c2b5841d2202398c02535dd549fa17291f10a22eb0ac3ee9fc6a435dd216c6: tree 02c2b5841d2202398c02535dd549fa17291f10a22eb0ac3ee9fc6a435dd216c6 not found in repository

restic find

repository 6d9f087d opened successfully, password is correct
Unable to load tree a3d42dfe1be29734192ca3865d3c7b28ac2118f5fde86a9576d4e411da4141be
… which belongs to snapshot e187bb0ba0d142deb715d19dbac80f164479c21493c99d6970b3409a09f7f8e7.
Unable to load tree 2646df869fb727d3a12215c8d8d921443bbad00e3b3bd589065e5fe3c16be7f8
… which belongs to snapshot e3d2f3f25426ce96cd568adf70893a6d2248a2008f08a67c2754c0f797902106.
Unable to load tree d017dfebfde1b7f1e472127394e072c0756ba81e14e28490c201e5586372f10c
… which belongs to snapshot e7dd95f98846dc947b63b2d5edba1f83b8628204ce6db117eb39a19dc48c1d0d.
Unable to load tree b3c1933de9bf26ab4ac1a429dd1cd0c1e70e9ed4259c128a10d0a72e170ad50d
… which belongs to snapshot ebcbdfa386d8ad4ce254a53346d39cc11466f84ceb3b5e72b01eb6d1c7b73e6c.
Unable to load tree 40e31ba3eeda5dfb26fecba20b53b785bdb6fe5dd4a9497af74f28a66a1b2096
… which belongs to snapshot f4021a24a4d2cd672fca025872165ddbdae71131b159c5a78c3f8352066e2ae9.

… and so on

restic check

error for tree 742d8ed6:
tree 742d8ed6: file “README” blob 0 size could not be found
tree 742d8ed6: file “init.py” blob 0 size could not be found
tree 742d8ed6: file “do_example.py” blob 0 size could not be found
tree 742d8ed6: file “do_exchain.py” blob 0 size could not be found
tree 742d8ed6: file “do_exploitable.py” blob 0 size could not be found
tree 742d8ed6: file “do_symfix.py” blob 0 size could not be found
tree 742d8ed6, blob 8efd4ce2: not found in index
tree 742d8ed6, blob 85961b36: not found in index
tree 742d8ed6, blob 422a93fd: not found in index
tree 742d8ed6, blob ea4f6931: not found in index
tree 742d8ed6, blob 735ff9b5: not found in index
tree 742d8ed6, blob c7552590: not found in index
error for tree e738a765:
tree e738a765: file “init.py” blob 0 size could not be found
tree e738a765: file “pydevd_helpers.py” blob 0 size could not be found
tree e738a765: file “pydevd_plugin_numpy_types.py” blob 0 size could not be found
tree e738a765: file “pydevd_plugins_django_form_str.py” blob 0 size could not be found
tree e738a765, blob 037628a5: not found in index
tree e738a765, blob 366f60d7: not found in index
tree e738a765, blob 4662789d: not found in index
tree e738a765, blob 8d03d5a2: not found in index
error for tree 7db25783:
tree 7db25783: file “README” blob 0 size could not be found
tree 7db25783: file “init.py” blob 0 size could not be found
tree 7db25783: file “do_example.py” blob 0 size could not be found
tree 7db25783: file “do_exchain.py” blob 0 size could not be found
tree 7db25783: file “do_exploitable.py” blob 0 size could not be found
tree 7db25783: file “do_symfix.py” blob 0 size could not be found
tree 7db25783, blob 8efd4ce2: not found in index
tree 7db25783, blob 85961b36: not found in index
tree 7db25783, blob 422a93fd: not found in index
tree 7db25783, blob ea4f6931: not found in index
tree 7db25783, blob 735ff9b5: not found in index
tree 7db25783, blob c7552590: not found in index
… and so on

restic prune

counting files in repo
building new index for repo
[5:08] 100.00% 18058 / 18058 packs

tree 02c2b5841d2202398c02535dd549fa17291f10a22eb0ac3ee9fc6a435dd216c6 not found in repository
…github.com/restic/restic/internal/repository.(*Repository).LoadTree
/restic/internal/repository/repository.go:713
…github.com/restic/restic/internal/restic.FindUsedBlobs
/restic/internal/restic/find.go:11
main.pruneRepository
/restic/cmd/restic/cmd_prune.go:191
main.runPrune
/restic/cmd/restic/cmd_prune.go:85
main.glob…func18
/restic/cmd/restic/cmd_prune.go:25
…github.com/spf13/cobra.(*Command).execute
/restic/vendor/github.com/spf13/cobra/command.go:762
…github.com/spf13/cobra.(*Command).ExecuteC
/restic/vendor/github.com/spf13/cobra/command.go:852
…github.com/spf13/cobra.(*Command).Execute
/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
/restic/cmd/restic/main.go:86
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
repository contains 18058 packs (531155 blobs) with 83.220 GiB
processed 531155 blobs: 36506 duplicate blobs, 716.422 MiB duplicate
load all snapshots
find data that is still in use for 59 snapshots
… and so on

restic rebuild-index

counting files in repo
[0:05] 100.00% 18058 / 18058 packs

finding old index files
saved new indexes as [4fecbd8d fa3c4282 c86b7b4e 384ba6dc a2045f82 b887f2c9 38d1891a]
remove 7 old index files
… and the same each time

I am sure that what I did was unplug the external drive while pruning. Actually I unplugged the USB3 HUB and didn’t even notice.

Is there a way for me to fix it?

Cheers!
[Edit]
I forgot to mention that I can add to the repository with no errors thrown.

Files: 3474 new, 1908 changed, 122746 unmodified
Dirs: 0 new, 7 changed, 5 unmodified
Data Blobs: 1246 new
Tree Blobs: 7 new
Added to the repo: 142.538 MiB

processed 128128 files, 27.028 GiB in 1:13
snapshot 2cb85d11 saved

Hello @msmafra, I’m sorry to hear that your backup repository got damaged. The prune command (hopefully) shouldn’t damage the repository, actually it goes to great lengths to try to ensure that the repository is not damaged no matter when the command crashes. There are a few commands you can try to repair your repository, but I also hope we can find out what went wrong. I currently suspect that some packs which restic assumed to be stored safely, were not actually written to the external disk.

Which version of restic are you using, what is the full backup command line? Which operating system do you use? Which filesystem does your external drive use? Which mount options are used for the drive? Please run a filesystem check on your external drive if you haven’t already done so.

Did you run the rebuild-index command after all the other commands or some time before? In case the repository index is damaged, restic won’t be able to find all tree/data blobs. If the index is correct and the errors persist, then some pack files seem to be missing. Can you run find /path/to/repo/data -type f | wc -l and compare that number with the pack count reported by rebuild-index?

restic currently hasn’t that many options to recover damaged repositories. The usual steps to recover are the following: First run restic rebuild-index (which you already have done) to ensure that the index only contains blobs which actually exist. Then run restic backup --force to have restic reread your backup folder. This can help fix the “file […] could not be found” errors if the corresponding files still exist. Afterwards you can run restic check to see whether this helped. You also might want to run restic check --read-data to ensure that none of the existing packs are corrupted.

Regarding the “tree […] not found in repository” errors are more problematic. If these are not fixed by the backup run, then your only option is to remove all affected snapshots, unless the missing packs files somehow show up again. I’d suggest to wait a bit with that option, at least until we had a chance to analyze the problem.

The backup command probably works always no matter how damaged the repository is. Please run the restic backup --force command as soon as possible to ensure that newly created snapshots do not refer to missing data.

3 Likes

Hello @MichaelEischer!

Thanks for the answer. The prune command is the last thing I remember doing when I noticed the problems.
I am using 0.9.6 was 0.9.5, from Fedora’s 31 repo, at the time. The full command is ran from inside a script I am making. The drive was in NTFS (found no fs problems) because there are some Acronis backups from my Windows 10 install there and I tested restic inside Windows to a different repository, that doesn’t even exist anymore. I am actually moving everything BTRFS on my drives and copied the entire backup/repository folder to another drive.

...
RESTIC_REPOSITORY="/run/media/marcelo/TheGrey/BACKUPS/tars_linux/"
RESTIC_PASSWORD_COMMAND="\gpg --quiet --decrypt /home/${USER:-}/Projects
/pwd.restic.txt.gpg"
RESTIC_EXCLUDED="restic-exclusions.txt"
RESTIC_INCLUDED="restic-inclusions.txt"
RESTIC_SOURCE="${HOME}/"
RESTIC_USER_HOME="${HOME}/"
...
\restic backup --verbose --exclude-file=${RESTIC_USER_HOME}Projects/${RESTIC_EXCLUDED:-} --files-from ${RESTIC_USER_HOME}Projects/${RESTIC_INCLUDED} ${RESTIC_USER_HOME:-}
...

Sometime I run it with the addition of —cleanup-cache.

And it also runs:

\restic stats --json
\restic stats --json
\restic check
\restic check --check-unused

The prune part I did manually:

\restic --repo “${RESTIC_REPOSITORY}” prune --cleanup-cache --verbose

The find and the rebuild-index results

❯ find ${RESTIC_REPOSITORY} -type f | wc -l
18174
❯ restic --repo ${RESTIC_REPOSITORY} rebuild-index
repository 6d9f087d opened successfully, password is correct
counting files in repo
[4:08] 100.00% 18099 / 18099 packs
finding old index files
saved new indexes as [1577a57d 2315f2ff 4976813d 302b765d e436301a b6e5cc94 74591f63]
remove 10 old index files

Ran now the backup with --force:

❯ restic --repo ${RESTIC_REPOSITORY} backup --force
lock repository
load index files
start scan on [/home/marcelo/ /home/marcelo/Games/steamapps/compatdata/275850/pfx/drive_c/users/steamuser/Application Data/HelloGames/NMS]
start backup on [/home/marcelo/ /home/marcelo/Games/steamapps/compatdata/275850/pfx/drive_c/users/steamuser/Application Data/HelloGames/NMS]
scan finished in 41.060s: 126708 files, 27.039 GiB

Files: 126708 new, 0 changed, 0 unmodified
Dirs: 12 new, 0 changed, 0 unmodified
Data Blobs: 126 new
Tree Blobs: 3 new
Added to the repo: 29.103 MiB

Checking my history here I also tried (should not have probably)

❯ restic check --no-cache --read-data
❯ restic check --verbose --read-data --check-unused --cleanup-cache --no-lock

I think I left nothing out.

Cheers!

[Edit]
The external deriver was monted with those parameters:
/dev/sdc2 on /run/media/marcelo/SHIROHIGE type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2)

Oh, I meant to run the find ${RESTIC_REPOSITORY}/data -type f | wc -l command for just the data folder, sorry for hiding the data folder part too well in the command. Can you give that command a try? It should report the same number of files as rebuild-index.

Did backup --force help with some of the check errors? The check command only ever reads data from the repository. It never modifies anything, so you can’t break your repository by running the check command.

Regarding the used filesystem: btrfs should be a safe choice as for it restic’s create,write,fsync,close cycle is enough to guarantee that files are safely stored in the repository. However, I also took a look at how NTFS-3G implements fsync (which is the main ingredient to safely store data files in the repository) and it just flushes the whole filesystem, which should also prevent the data loss you’ve experienced. So I’m a bit confused right now, as this seems to rule out my hypothesis on what went wrong…

1 Like

Sorry, I just ran on the repo did not think about the ‘data’ part of your instruction. He it is:

❯ find ${RESTIC_REPOSITORY}/data -type f | wc -l
18106

I did not run check yet. I will run in a bit.

❯ restic check --read-data

tree 4b067af0, blob be00b8ad: not found in index
tree 4b067af0, blob 8704775c: not found in index
tree 4b067af0, blob d814961d: not found in index
tree 4b067af0, blob c317982b: not found in index
tree 4b067af0, blob a4c75057: not found in index
tree 4b067af0, blob 7d87ff5b: not found in index
error for tree 2b561b8c:
tree 2b561b8c: file “package.json” blob 0 size could not be found
tree 2b561b8c, blob 9f6889e0: not found in index
error for tree a264690d:
tree a264690d: file “package.json” blob 0 size could not be found
tree a264690d, blob c47cb48b: not found in index
error for tree 49b234d8:
tree 49b234d8: file “PowerShellEditorServices.VSCode.cat” blob 0 size could not be found
tree 49b234d8: file “PowerShellEditorServices.VSCode.psd1” blob 0 size could not be found
tree 49b234d8: file “PowerShellEditorServices.VSCode.psm1” blob 0 size could not be found
tree 49b234d8, blob 599f770f: not found in index
tree 49b234d8, blob 2d55f388: not found in index
tree 49b234d8, blob 994c3def: not found in index
error for tree 832175d7:
tree 832175d7: file “PowerShellEditorServices.cat” blob 0 size could not be found
tree 832175d7: file “PowerShellEditorServices.psd1” blob 0 size could not be found
tree 832175d7: file “PowerShellEditorServices.psm1” blob 0 size could not be found
tree 832175d7: file “Start-EditorServices.ps1” blob 0 size could not be found
tree 832175d7, blob 73f3d24a: not found in index
tree 832175d7, blob 492223be: not found in index
tree 832175d7, blob a8684a2c: not found in index
tree 832175d7, blob bc9d7221: not found in index
error for tree 04aac374:
tree 04aac374: file “package.json” blob 0 size could not be found
tree 04aac374, blob 20433049: not found in index
error for tree c10fa686:
tree c10fa686: file “package.json” blob 0 size could not be found
tree c10fa686, blob 8ab55c4f: not found in index
error for tree f3c651f3:
tree f3c651f3: file “debugAdapter.js” blob 0 size could not be found
tree f3c651f3: file “debugAdapter.js.map” blob 0 size could not be found
tree f3c651f3: file “feature.js” blob 0 size could not be found
tree f3c651f3: file “feature.js.map” blob 0 size could not be found
tree f3c651f3: file “logging.js” blob 0 size could not be found
tree f3c651f3: file “logging.js.map” blob 0 size could not be found
tree f3c651f3: file “main.js” blob 0 size could not be found
tree f3c651f3: file “main.js.map” blob 0 size could not be found
tree f3c651f3: file “platform.js” blob 0 size could not be found
tree f3c651f3: file “platform.js.map” blob 0 size could not be found
tree f3c651f3: file “process.js” blob 0 size could not be found
tree f3c651f3: file “process.js.map” blob 0 size could not be found
tree f3c651f3: file “session.js” blob 0 size could not be found
tree f3c651f3: file “session.js.map” blob 0 size could not be found
tree f3c651f3: file “settings.js” blob 0 size could not be found

All those errors occur before the check process start.

❯ restic --repo ${RESTIC_REPOSITORY} check --read-data --check-unused --cleanup-cache
using temporary cache in /tmp/restic-check-cache-689558897
enter password for repository:
repository 6d9f087d opened successfully, password is correct
created new cache in /tmp/restic-check-cache-689558897
create exclusive lock for repository
load indexes
check all packs

unused blob 42d292c0
unused blob a8b81cab
unused blob e7992406
unused blob ea5750e9
unused blob 51782eab
unused blob 1e924330

unused blob 11f88603
unused blob afbcae2c
unused blob 52207f73
unused blob a1693542
unused blob 65639ae6
unused blob 9570a95d
unused blob 75e32cee
unused blob 6e5cb180
unused blob c39a4f3f
unused blob bf92824a
unused blob 35602cc4
unused blob 6050a7c2
unused blob dcc7984a
unused blob 674e7b70
unused blob 0682c4b8
unused blob 140b82f9
unused blob 103a2dba
unused blob 7d836ac0

unused blob 1d41fcdb
unused blob d6f0bc5b
unused blob cca1164e
unused blob be4d9ff9
read all data
[1:10:51] 100.00% 18106 / 18106 items
duration: 1:10:51
Fatal: repository contains errors

Also I mounted the repo and it gives me various I/O errors:

ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2020-01-22T16:35:40-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-07-04T16:10:16-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-28T15:22:53-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-25T18:42:19-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-11-25T09:30:39-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-25T12:36:57-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-11-07T18:33:50-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-08T20:35:14-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-06-05T13:18:16-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-08-14T18:03:57-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-08-30T18:23:58-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2020-01-22T16:58:44-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-09-13T11:12:56-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-09-08T20:51:35-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-06-19T09:09:38-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-06-06T12:27:10-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2020-02-01T16:56:47-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-14T16:40:57-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-22T22:16:01-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-06-17T21:56:46-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-06-06T15:20:40-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-11-04T10:53:54-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-09-28T11:32:40-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-08T20:14:56-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-04-27T16:27:26-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2020-02-05T14:41:27-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-19T14:15:03-03:00’: Erro de entrada/saída
ls: não foi possível acessar ‘/home/marcelo/rest-o/snapshots/2019-05-19T14:23:17-03:00’: Erro de entrada/saída
2019-04-27T16:27:26-03:00 2019-11-26T11:33:03-03:00
2019-05-02T12:37:57-03:00 2019-11-26T20:34:47-03:00
2019-05-08T20:14:56-03:00 2020-01-22T16:35:40-03:00
2019-05-08T20:35:14-03:00 2020-01-22T16:58:44-03:00
2019-05-14T16:40:57-03:00 2020-02-01T16:56:47-03:00
2019-05-14T16:44:10-03:00 2020-02-05T14:41:27-03:00
2019-05-19T14:15:03-03:00 2020-02-11T13:12:37-03:00
2019-05-19T14:23:17-03:00 2020-02-15T13:09:53-03:00
2019-05-22T22:16:01-03:00 2020-02-16T19:51:37-03:00
2019-05-25T12:36:57-03:00 2020-02-16T21:34:54-03:00
2019-05-25T18:42:19-03:00 2020-02-16T21:45:24-03:00
2019-05-28T15:22:53-03:00 2020-02-17T09:13:41-03:00
2019-05-28T15:37:58-03:00 2020-02-17T09:21:21-03:00
2019-06-05T13:18:16-03:00 2020-02-18T14:25:14-03:00
2019-06-06T12:27:10-03:00 2020-02-18T15:37:22-03:00
2019-06-06T15:20:40-03:00 2020-02-18T18:18:30-03:00
2019-06-17T21:49:22-03:00 2020-02-18T20:30:35-03:00
2019-06-17T21:56:46-03:00 2020-02-18T20:37:56-03:00
2019-06-19T09:09:38-03:00 2020-02-18T20:44:35-03:00
2019-07-04T16:10:16-03:00 2020-02-18T23:51:24-03:00
2019-07-24T18:57:14-03:00 2020-02-20T09:39:36-03:00
2019-08-14T18:03:57-03:00 2020-02-21T17:48:37-03:00
2019-08-21T13:45:56-03:00 2020-02-22T15:04:10-03:00
2019-08-30T18:23:58-03:00 2020-02-22T16:43:59-03:00
2019-09-08T20:51:35-03:00 2020-02-22T19:07:36-03:00
2019-09-13T11:12:56-03:00 2020-02-23T02:55:11-03:00
2019-09-28T11:32:40-03:00 2020-02-26T10:24:28-03:00
2019-11-04T10:53:54-03:00 2020-03-03T14:10:28-03:00
2019-11-04T11:33:22-03:00 2020-03-03T14:41:57-03:00
2019-11-07T18:33:50-03:00 2020-03-03T15:30:10-03:00
2019-11-25T09:30:39-03:00 2020-03-05T11:16:00-03:00
2019-11-25T19:12:43-03:00 latest

Hmm, that looks rather bad. I wonder how many pack files were lost. In order to repair the repository you will have to forget every damaged snapshot, in case you have enough free space you can keep a copy of the damaged repository in case you might need data from a damaged snapshot. The recover command might come in handy in that case as it searches the repository for every tree blob it can find and then adds them into a new snapshot.

You can use the stats command to find damaged snapshots: It will complain about missing trees/blobs in a snapshot which you can forget then. Afterwards check and prune should be able to complete again.

1 Like

I actually end up running some commands starting with restic forget. It shows no more errors, but as I was a bit “radical” with the forget range, almost oblivion :), I don’t know how much I lost, if any, or file that were only on the repo.

$ restic forget -m 1 --prune
$ restic check --read-data --verbose
$ restic check --read-data --verbose --check-unused
$ restic stats
$ restic snapshots
$ restic rebuild-index
$ restic mount ~/restic

Thanks very much for your help.

Cheers!

1 Like