Fatal: number of used blobs is larger than number available blobs!

Getting Help
1

I get this message when running prune. I’m running restic 0.9.3, the host storing the repo is Debian and I’m backing up an Ubuntu machine. This is my output:

repository 75ed913a opened successfully, password is correct
counting files in repo
building new index for repo
[0:32] 100.00%  24401 / 24401 packs
incomplete pack file (will be removed): a4211ba254b88471e9bd64596f7e48082fd30cf097a743ab7b40f864f4cf7330
incomplete pack file (will be removed): ddf2d259ffcd45d434f00806e3730f10f6f1f964c981ed180075d4237e79bb0f
incomplete pack file (will be removed): 5f3c5666bbaedbe175da089925e1ea35f26d3bce38f57a88e6ffa65f159463c5
incomplete pack file (will be removed): 4193897d4943c903ac10ca44a253a0ccb2c3900f34db2b5611c90181d83fb678
incomplete pack file (will be removed): faaa65c6100f48890934cdcefec6bc33d277f75574e55983f3420aafe34741ab
incomplete pack file (will be removed): 4cda4347e13636d9eb71fb7edc29f706cf0b97906c3b691f6f5deb6956191d40
incomplete pack file (will be removed): 152b1481a59d8273c0028b5aaa8726ef85066236995015645109960b2033fcbf
repository contains 24394 packs (254518 blobs) with 108.574 GiB
processed 254518 blobs: 0 duplicate blobs, 0B duplicate
load all snapshots
find data that is still in use for 51 snapshots
[0:07] 100.00%  51 / 51 snapshots
Fatal: number of used blobs is larger than number of available blobs!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new

Still, running backups works and it doesn’t complain. I ran rebuild-index, check and it says:

using temporary cache in /tmp/restic-check-cache-776913306
repository 75ed913a opened successfully, password is correct
created new cache in /tmp/restic-check-cache-776913306
create exclusive lock for repository
load indexes
check all packs
pack 4cda4347: not referenced in any index
pack 4193897d: not referenced in any index
pack faaa65c6: not referenced in any index
pack ddf2d259: not referenced in any index
pack a4211ba2: not referenced in any index
pack 5f3c5666: not referenced in any index
pack 152b1481: not referenced in any index
7 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree ac08e4f0:
  tree ac08e4f0: file "d8cbe80e7ffd44acbea278507ecd65d8-device-volumes.tdb" blob 0 size could not be found
  tree ac08e4f0, blob eb4c8c4c: not found in index
error for tree fd2c7606:
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 191 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 193 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 194 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 201 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 202 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 204 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 205 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 220 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 221 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 224 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 227 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 228 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 229 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 231 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 232 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 286 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 287 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 290 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 295 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 298 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 300 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 302 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 330 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 337 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 339 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 341 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 343 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 346 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 347 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 349 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 351 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 352 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 353 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 354 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 358 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 359 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 360 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 363 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 364 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 365 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 369 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 370 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 371 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3872 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 7643 size could not be found
  tree fd2c7606: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 8896 size could not be found
  tree fd2c7606, blob f5e26eac: not found in index
  tree fd2c7606, blob f1ef845f: not found in index
  tree fd2c7606, blob 658910ec: not found in index
  tree fd2c7606, blob fd05611a: not found in index
  tree fd2c7606, blob 34fd46d3: not found in index
  tree fd2c7606, blob 8498c836: not found in index
  tree fd2c7606, blob f4cf51e2: not found in index
  tree fd2c7606, blob 25ab5818: not found in index
  tree fd2c7606, blob 1819b147: not found in index
  tree fd2c7606, blob e9abece4: not found in index
  tree fd2c7606, blob 8846b4ed: not found in index
  tree fd2c7606, blob 9a61985a: not found in index
  tree fd2c7606, blob 1c6a1c49: not found in index
  tree fd2c7606, blob 5f0f0cae: not found in index
  tree fd2c7606, blob 5680a992: not found in index
  tree fd2c7606, blob a916d525: not found in index
  tree fd2c7606, blob 89b72aa1: not found in index
  tree fd2c7606, blob c6418b93: not found in index
  tree fd2c7606, blob 5b68bc8f: not found in index
  tree fd2c7606, blob f6be3b3d: not found in index
  tree fd2c7606, blob 01fdd840: not found in index
  tree fd2c7606, blob 40ae55ee: not found in index
  tree fd2c7606, blob 7e88171b: not found in index
  tree fd2c7606, blob 798b7f1e: not found in index
  tree fd2c7606, blob 3ba22ccf: not found in index
  tree fd2c7606, blob 4c0849fb: not found in index
  tree fd2c7606, blob a6a7daf7: not found in index
  tree fd2c7606, blob 9dcd1d31: not found in index
  tree fd2c7606, blob b962369b: not found in index
  tree fd2c7606, blob 2e116b27: not found in index
  tree fd2c7606, blob 5cf76ddd: not found in index
  tree fd2c7606, blob 7be16ddf: not found in index
  tree fd2c7606, blob 8b0bfd2b: not found in index
  tree fd2c7606, blob e9926f0f: not found in index
  tree fd2c7606, blob 7b28ed57: not found in index
  tree fd2c7606, blob df684f97: not found in index
  tree fd2c7606, blob 4d54d4c8: not found in index
  tree fd2c7606, blob 796c2294: not found in index
  tree fd2c7606, blob 033192c0: not found in index
  tree fd2c7606, blob 3b3b0ab2: not found in index
  tree fd2c7606, blob 780a44b7: not found in index
  tree fd2c7606, blob 07142166: not found in index
  tree fd2c7606, blob 3611aa2a: not found in index
  tree fd2c7606, blob fb2e726f: not found in index
  tree fd2c7606, blob 13aa5ee5: not found in index
  tree fd2c7606, blob 84d47d3f: not found in index
error for tree c80b0385:
  tree c80b0385: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3872 size could not be found
  tree c80b0385: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 7641 size could not be found
  tree c80b0385: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 8895 size could not be found
  tree c80b0385, blob fb2e726f: not found in index
  tree c80b0385, blob 13aa5ee5: not found in index
  tree c80b0385, blob 84d47d3f: not found in index
error for tree 4430a525:
  tree 4430a525: file "d8cbe80e7ffd44acbea278507ecd65d8-device-volumes.tdb" blob 0 size could not be found
  tree 4430a525, blob eb4c8c4c: not found in index
error for tree 45b1a9fb:
  tree 45b1a9fb: file "local-log_2018-11-06-19:00" blob 0 size could not be found
  tree 45b1a9fb: file "local-log_2018-11-06-21:00" blob 0 size could not be found
  tree 45b1a9fb: file "rsync-log_2018-11-06-20:10" blob 0 size could not be found
  tree 45b1a9fb, blob 32c4fe85: not found in index
  tree 45b1a9fb, blob 7fbf7a8b: not found in index
  tree 45b1a9fb, blob 8a15704b: not found in index
error for tree e5954881:
  tree e5954881: file "rhythmdb.xml" blob 0 size could not be found
  tree e5954881: file "rhythmdb.xml" blob 1 size could not be found
  tree e5954881: file "rhythmdb.xml" blob 5 size could not be found
  tree e5954881: file "rhythmdb.xml" blob 6 size could not be found
  tree e5954881, blob db5808b1: not found in index
  tree e5954881, blob 47fba7c6: not found in index
  tree e5954881, blob e5028949: not found in index
  tree e5954881, blob e8df5c77: not found in index
error for tree 55249162:
  tree 55249162: file "4c134e0dedda492393414cc1ecc47171.manager" blob 6 size could not be found
  tree 55249162: file "4c134e0dedda492393414cc1ecc47171.manager" blob 48 size could not be found
  tree 55249162, blob 92ef75c2: not found in index
  tree 55249162, blob 9df4d307: not found in index
error for tree 17dd982c:
  tree 17dd982c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3881 size could not be found
  tree 17dd982c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3884 size could not be found
  tree 17dd982c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 7599 size could not be found
  tree 17dd982c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 8850 size could not be found
  tree 17dd982c, blob fb2e726f: not found in index
  tree 17dd982c, blob 2b15a7d2: not found in index
  tree 17dd982c, blob 13aa5ee5: not found in index
  tree 17dd982c, blob 84d47d3f: not found in index
error for tree ed307211:
  tree ed307211: file "local-log_2018-11-06-19:00" blob 0 size could not be found
  tree ed307211: file "rsync-log_2018-11-06-20:10" blob 0 size could not be found
  tree ed307211, blob 32c4fe85: not found in index
  tree ed307211, blob 8a15704b: not found in index
error for tree 58b4abfb:
  tree 58b4abfb: file "000282.log" blob 0 size could not be found
  tree 58b4abfb, blob fcfad676: not found in index
error for tree 19ea73f0:
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 0 size could not be found
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 6 size could not be found
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 12 size could not be found
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 17 size could not be found
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 33 size could not be found
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 48 size could not be found
  tree 19ea73f0: file "4c134e0dedda492393414cc1ecc47171.manager" blob 59 size could not be found
  tree 19ea73f0, blob 31f22e2e: not found in index
  tree 19ea73f0, blob 92ef75c2: not found in index
  tree 19ea73f0, blob 5075ee5f: not found in index
  tree 19ea73f0, blob b497b70a: not found in index
  tree 19ea73f0, blob b2bf9429: not found in index
  tree 19ea73f0, blob 9df4d307: not found in index
  tree 19ea73f0, blob 138c8a18: not found in index
error for tree 3d21062a:
  tree 3d21062a: file "file" blob 0 size could not be found
  tree 3d21062a: file "file" blob 6 size could not be found
  tree 3d21062a: file "file" blob 12 size could not be found
  tree 3d21062a: file "file" blob 17 size could not be found
  tree 3d21062a: file "file" blob 33 size could not be found
  tree 3d21062a: file "file" blob 48 size could not be found
  tree 3d21062a: file "file" blob 59 size could not be found
  tree 3d21062a: file "file" blob 6 size could not be found
  tree 3d21062a: file "file" blob 48 size could not be found
  tree 3d21062a, blob 31f22e2e: not found in index
  tree 3d21062a, blob 92ef75c2: not found in index
  tree 3d21062a, blob 5075ee5f: not found in index
  tree 3d21062a, blob b497b70a: not found in index
  tree 3d21062a, blob b2bf9429: not found in index
  tree 3d21062a, blob 9df4d307: not found in index
  tree 3d21062a, blob 138c8a18: not found in index
  tree 3d21062a, blob 92ef75c2: not found in index
  tree 3d21062a, blob 9df4d307: not found in index
error for tree f02fc452:
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3879 size could not be found
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3880 size could not be found
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3882 size could not be found
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3883 size could not be found
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3884 size could not be found
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 7587 size could not be found
  tree f02fc452: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 8838 size could not be found
  tree f02fc452, blob fb2e726f: not found in index
  tree f02fc452, blob 80ebbc47: not found in index
  tree f02fc452, blob 2b15a7d2: not found in index
  tree f02fc452, blob b7d74a00: not found in index
  tree f02fc452, blob 6c83cead: not found in index
  tree f02fc452, blob 13aa5ee5: not found in index
  tree f02fc452, blob 84d47d3f: not found in index
error for tree 06be86f1:
  tree 06be86f1: file "http_127.0.0.1_55555.localstorage-shm" blob 0 size could not be found
  tree 06be86f1: file "http_127.0.0.1_55555.localstorage-wal" blob 0 size could not be found
  tree 06be86f1, blob 38b1969c: not found in index
  tree 06be86f1, blob 30f31282: not found in index
error for tree 5d861a1d:
  tree 5d861a1d: file "file" blob 0 size could not be found
  tree 5d861a1d: file "file" blob 6 size could not be found
  tree 5d861a1d: file "file" blob 12 size could not be found
  tree 5d861a1d: file "file" blob 17 size could not be found
  tree 5d861a1d: file "file" blob 33 size could not be found
  tree 5d861a1d: file "file" blob 48 size could not be found
  tree 5d861a1d: file "file" blob 59 size could not be found
  tree 5d861a1d, blob 31f22e2e: not found in index
  tree 5d861a1d, blob 92ef75c2: not found in index
  tree 5d861a1d, blob 5075ee5f: not found in index
  tree 5d861a1d, blob b497b70a: not found in index
  tree 5d861a1d, blob b2bf9429: not found in index
  tree 5d861a1d, blob 9df4d307: not found in index
  tree 5d861a1d, blob 138c8a18: not found in index
error for tree 8e23e83c:
  tree 8e23e83c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3878 size could not be found
  tree 8e23e83c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3881 size could not be found
  tree 8e23e83c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 3882 size could not be found
  tree 8e23e83c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 7561 size could not be found
  tree 8e23e83c: file "{8636e43b-5952-41ea-a09c-8679ff84055a}.vdi" blob 8811 size could not be found
  tree 8e23e83c, blob fb2e726f: not found in index
  tree 8e23e83c, blob 2b15a7d2: not found in index
  tree 8e23e83c, blob b7d74a00: not found in index
  tree 8e23e83c, blob 13aa5ee5: not found in index
  tree 8e23e83c, blob 84d47d3f: not found in index
Fatal: repository contains errors

Before this I had a minor problem with the disk I am storing the repo. I repair it and then move the repo again to the disk.

How could I solve this? Is this a bug?

Edit:
I think this was my fault. Initially my problem was that I had some data with ???? permissions. So I searched and found that maybe was something wrong with my disk. At first I thought it was the repo. So, the day before that I made a copy of my repo (usually I do make a copy of my repo every week or so). So I used fsck and the problem was gone. I transferred my repo again to my main disk and then this “fatal” error happened. What I think the problem was that maybe while I was copying the repo from one disk to another there was some machine doing a backup at the same time, so rsync didn’t notice that there was some data being transferred at the same time so the repo at the end is incomplete, missing data; leaving me no option than to start a new one.

I was going to delete this but I read recently a question about copying the repo to another location, which I think cause this problem. It seems like it is not a good practice to copy the repo while others processes are running. If I’m wrong and there is a way to fix this, please let me know. I may be wrong, I don’t know.

2

Hey, I’m sorry that you had a problem with restic, we don’t really handle this workflow well. What restic is telling you (during prune and also during check) that snapshots in the repo reference data that just is not there. I may have happened when moving the repo around, or fsck on the disc did not discover a file or so.

It should not be a problem, since the snapshot (which references the data) is created as the very last step. So if you make a copy of a repo while a backup is running and then inspect this repo, the new snapshot won’t be in there, and it’ll just have some data that isn’t referenced anywhere. All snapshots should be complete and consistent.

In order to be able to run prune again, you need to remove the snapshots which reference the missing data. That’s not optimal, but works for now.

The restic check tells you which trees reference the missing data, e.g. c80b0385. You can find out which snapshots reference this tree by running:

$ restic find --tree c80b0385

It will (among others) tell you which snapshot references this tree. You can then use restic forget to remove that snapshot, then run restic check again to check if there are more references for the missing data.

3 Likes
3

Well that was easy. Sorry if this was something basic but I get scared when the Please report this error was displayed. I did the restic find --tree with all trees from error and just found one snapshot.

repository 75ed913a opened successfully, password is correct
Found tree 45b1a9fb97c353f22dd86cd8130ff782817fa99d8e0f0ed72d2bc6d52e821ca2
 ... path /home/admin/.logs
 ... in snapshot cd4e9031 (2018-11-06 21:00:04)

After restic forget cd4e9031, check found no errors and prune is successful. However, I still think this had something to do with the backup running while the copying the repo process was running too because the only snapshot listed is the one taken at the same time that the copying process was running. I started the sync proccess at 20:30 approximately. This always take between 30-50 minutes (time vary because I’m copying the repo and other folders besides the repo). This specific time, it started at 20:30 and it lasted about 30-35 minutes. I use rsync to sync the repo and other to another HDD and looking at the output, every time it runs it will be syncing file by file, so it will copy/sync config file, then data folder and subfolders starting with 00 and ending with ff. So what I think is that maybe while the backup was running, the sync process was already finishing (backup time was 1 minute) and maybe the backup was writing in some of the files that were already synced, so it may copy the index and snapshots directories correctly but there were missing files in data directory. Could that be possible? Based on what you said, then the repo may be listing a snapshot that was missing data, and maybe that was the problem. Does that makes sense?

This is the only explanation I have because the problem with the disk was the next day, when the repo was already in the extra HDD. The next day I used fsck because the problem I was having was that some file permissions in the data directory of the repository was ????????? ? ? for some reason. I tried to correct that using chown but didn’t work so I had to repair that via fsck; so fsck deleted those files and then I synced again the repo from the spare disk to my main machine. At that point was when the fatal error happened.

Again, I know I may be wrong but this is my best guest. Thank you very much for your time to answer me and help me with my problem. You are the man! Restic is awesome, thank you for everything.

4

What program were you using to perform the copy while the backup was running?

Tools like rclone/rsync will scan the source directory before copying anything, so they would never copy the snapshot for an in-progress backup since the snapshot didn’t exist when the source directory was scanned.1

However, tools like cp/scp may perform a depth-first search and copy at the same time; there is no separate scanning phase. So they might miss data files that were created later (particularly in the earlier directories searched) but if the backup completes by the time the copy tool makes it to the snapshots folder, it could have copied the snapshot. Then you wind up in the situation you found yourself in: a snapshot referencing data that wasn’t copied.

1I lied. This may not be exactly true if the backup completes during the rsync/rclone scanning phase; the scan could miss data files that were just created but still catch the snapshot. This situation should be rare with rsync/rclone but is still possible because recursively scanning a directory is not an atomic operation. Note that it also depends on the order of the data and snapshot directories; if snapshot is scanned first then this situation is impossible, but if data is scanned first then it is possible.

1 Like
5

Exactly, that’s my guess. I’m using rsync and by looking at the output at the moment, it copy the config file first, then the data directory, etc. The snapshot did finished before the sync process finish. The sync process took about 30-35 minutes and the snapshot took about 1 minute when the sync process was almost finishing.

6

Hm, interesting. That may indeed have happened, I think it’s realistic. :slight_smile:

1 Like
7

I think so because I have been doing this copy thing for quite some time and never had any problem but most of the time there was no restic process running. These things just happen to me, when I need the backup of the backup, guess what, I run into problems, lol. Anyways, I’ll make sure from now on to sync just when there are no other processes running and will always make a check of that copy just in case. Thanks again for your time @fd0.

8

FWIW, you don’t necessarily have to change anything about your process. Think about it. At the end of the day, the result of both situations was the same: you either didn’t have the snapshot from a currently-running backup, or you had the snapshot but not all of its data. Either way, you didn’t have the currently-running backup fully synced. That’s not really a failure case; that’s precisely what’s expected.

The only difference between the two in practice is that, until the next sync can bring in the data for that snapshot, you have an effectively invalid snapshot that check/prune will complain about.

Alternatively, you can break the sync into two operations: sync snapshots and then sync everything else.

1 Like
9

You’re totally right and the problem (after explained) was pretty easy to fix.

Weird thing that happens after that was that it happen again the error but now (thanks to find --pack ID command) I found the file giving me problems. It was a Virtualbox log file. Deleted the snapshot again and now I’m excluding this logs because I don’t know why it keeps giving me problems. Today I tested like three times deleting the new snapshot and backing up again and the same logs were giving me the same error after a new snapshot. Don’t know why but since I don’t need them now I’m excluding those logs from my backups.

Thanks for your answers! It helped a lot, I really appreciate it.

10

Huh, that’s really odd. Did the log files change? If not, maybe restic thought that the content was already stored and didn’t store it (deduplication) but it wasn’t actually stored. Perhaps the indexes need to be regenerated?

At any rate, if you can reliably reproduce this issue, particularly when no backup is running, that seems like it warrants a bug report.

1 Like
11

Yes, I think the logs files were changed. I don’t really dig into Virtualbox log files but I’ll test it tomorrow again to see if it gives me the same errors to post it here so you can see if is a bug or not.

12

restic will not be able to save a consistent copy of a file when it changes during backup, but the errors described here are on a much lower level in the repository data…

1 Like
13

When I asked if the file changed, I meant since the prior backup, not during the backup. What I’m getting at is: if it did not change since the last backup, and the data is still missing after the following backup, that could mean that restic thinks that the data is stored (due to out-of-date indexes maybe?) and therefore doesn’t store it again.

1 Like
14

Ah, yes, that’s correct. When the data is still in the index (although the actual files are missing), restic will assume the data is there and not store it again.

1 Like
15

False alarm… I think I forgot about rebuild-index in the process and after removing the problematic snapshots, rebuild-index and prune everything is fine now. Did a new backup including the VirtualBox logs and then did a check --read-data and no errors were found. Sorry for the confusion. Thanks again to both for the answers, it really helped me a lot!

16

No worries, happy to help!

And FWIW, I’m pretty sure indexes are rebuilt during prune (and prune doesn’t use the indexes, either), so you shouldn’t have to do rebuild-index before or after a prune.

1 Like