Help in recovery a corrupted repository

Hi all,

I’m new to repository errors and I’m afraid I have no idea about the correct sequence of actions in order to fix it.

My idea is to rclone the repo and start working on the copy, so I can start again if I screw up.

pack 7dc10f2c: unexpected file size: got 4777388, expected 1879850 I confirm the size is 4.56mb in the repo.

error for tree 833ca51f:
id 833ca51f317f7ba031942b0eb17c30efb0374c55e38b597d484e1895d579684a not found in repository All the “not found” files are really missing.

Can someone please advise?

On this server I have memory constraints, and I had to forcefully restart it a few times. Moreover the backup duration suddenly spiked, so I got multiple backups running at the same time (now I use flock), but I never thought some daily backups could last more than a day instead of an hour or less. I also had another directory backed up 4 times a day to the same repo, and of course I had a lot of instances of this job too.

I’m not familiar with restic internals, but I understand some of the indexing issues could be related to interrupted backups and multiple backups of the same data at the same time, but I’m puzzled by files not found and the file size difference.

Sorry for the lengthy post, but I really want to understand what is happening, because I’m a little scared and I want to sleep at night well with restic looking after my data. :slight_smile:

below the full log of the latest check:

create exclusive lock for repository
load indexes
pack 7847c86a contained in several indexes: {a9ad7aa1 eec1e41a}
pack c6c85157 contained in several indexes: {19132904 e2c1c9ae}
pack 05bf2858 contained in several indexes: {005c6f15 1c5ad87d}
pack 38e5aa9d contained in several indexes: {519de78c 917fcc22}
pack ebaed3ee contained in several indexes: {4be298f2 f6c558e0}
pack 5b1c6f24 contained in several indexes: {464acfd2 cd7e3d90}
pack 6531df69 contained in several indexes: {1fcaa3c1 8f0245c8}
pack 63b92fd1 contained in several indexes: {0d2114b5 fdce2dd7}
pack ef0a97d9 contained in several indexes: {1ca31e67 7aba678d}
pack 711baf37 contained in several indexes: {16851985 66aef055}
pack 76901163 contained in several indexes: {55a63574 98d02e01}
pack d0ed34c0 contained in several indexes: {5f0867f1 eec1e41a}
pack d4791837 contained in several indexes: {a6e21caa f3ecf298}
pack 38abe413 contained in several indexes: {0e1091b1 3ec4be9b}
pack 69586d45 contained in several indexes: {47458d84 503bccc8}
pack 6c17ae4a contained in several indexes: {34393c80 d0552fed}
pack 82c29025 contained in several indexes: {8164d9f8 a22e3b62}
pack c4e5589f contained in several indexes: {11721c95 4a37937a}
pack 6c3c3d55 contained in several indexes: {06e4ff11 91c2d035}
pack ad748c5a contained in several indexes: {31e3dd07 eec1e41a}
pack 229e4b99 contained in several indexes: {6e2475f2 fa00590c}
pack f39fd3ab contained in several indexes: {1e8e8971 c43ddb75}
pack e71d32ef contained in several indexes: {07022b49 a6a4d2d0}
pack 9600aaea contained in several indexes: {1d0f5464 8cda8c55}
pack 7e9f3e6e contained in several indexes: {ec4cb63e f69a56e2}
pack b745498d contained in several indexes: {ed4bea49 f4329a65}
pack c1668d2a contained in several indexes: {7a0deaf0 eec1e41a}
pack 2c93e5fe contained in several indexes: {1cf66a5d 7e958f38}
pack 49f65ee1 contained in several indexes: {c68661fe fd22a741}
pack 3a79fe0b contained in several indexes: {4fbe6415 65c6aacf}
pack 588d3bce contained in several indexes: {309a5de9 5cfecb6d}
pack 6f1f2ae0 contained in several indexes: {5f7493a6 eec1e41a}
pack 08bd22f4 contained in several indexes: {4ee18435 d94f846a}
pack 9fd176f7 contained in several indexes: {0e1faf16 5ac8ec7c}
pack 2eed4fb3 contained in several indexes: {0cfed863 d21ca9e2}
pack 6334781b contained in several indexes: {6575ec01 a07b244f}
pack 20ee25fd contained in several indexes: {8ed127b5 aef86b6d}
pack 32504448 contained in several indexes: {01fdd9e5 b027a9a0}
pack 8489970f contained in several indexes: {abd8bd82 dce2b1ac}
pack 6d01a04d contained in several indexes: {19fdc38c e98a2f56}
pack da52198e contained in several indexes: {26efbaa5 4bbfdd07}
pack 026124e7 contained in several indexes: {8ba3eff5 8c33ae22}
pack 2281bb8b contained in several indexes: {470160ab 9e393685}
pack 1bd629fc contained in several indexes: {1cf2817b 53fd3609}
pack b1dec0e0 contained in several indexes: {393393ee 3fdca105}
pack b450f015 contained in several indexes: {07022b49 4b0d1bb0}
pack 7aad3cdb contained in several indexes: {b14c2815 d1d55f69}
pack ad56786f contained in several indexes: {b64cdae7 c68661fe}
pack fbb068cc contained in several indexes: {5831fd8d f716acae}
pack bc4a6068 contained in several indexes: {189eb866 e4ce123f}
pack a6978b4d contained in several indexes: {4fc62c54 eec1e41a}
pack 897c8e26 contained in several indexes: {e2136ef5 f50eda8c}
pack 0c0bd230 contained in several indexes: {55b16d33 770cb754}
pack 9d7d19ab contained in several indexes: {0cfed863 55cfc0b7}
pack 3697c923 contained in several indexes: {14e5cb0e 20c02df5}
pack 47fe33f9 contained in several indexes: {5d1ef9d9 9e18b651}
pack 81f35101 contained in several indexes: {98d02e01 f15b1da4}
pack 06533c0d contained in several indexes: {260e873f fa1050bc}
pack 968bbb72 contained in several indexes: {01fdd9e5 8cda8c55}
pack c2f7943f contained in several indexes: {58fd8781 b8548a21}
pack ed0bb954 contained in several indexes: {0ec2db89 66aef055}
pack ac110ed6 contained in several indexes: {0ba7b010 e73ad6f5}
pack ea8fb8be contained in several indexes: {9e0a50ab a266b78f}
pack 170a75c9 contained in several indexes: {830c33c3 8c62531a}
pack 2d78a285 contained in several indexes: {614691b4 67107a82}
pack d4ec1225 contained in several indexes: {b7969312 f2c617cb}
pack e1aaf70b contained in several indexes: {64ace629 8d672b88}
pack 57168282 contained in several indexes: {b5f02c47 cd72797d}
pack 390fc272 contained in several indexes: {2af2e61f 697f559d}
pack d739d01a contained in several indexes: {2405e2cd 7e00354f}
pack ef4ccb95 contained in several indexes: {5c66c78d a9af73ab}
pack 6a3f16cf contained in several indexes: {b26da722 b7969312}
pack 8395a310 contained in several indexes: {23d3e8a3 b7969312}
pack 11a77493 contained in several indexes: {4c47bff1 dcc44ede}
pack 90b8caa8 contained in several indexes: {d16f01ff f892b229}
pack 3760ebe2 contained in several indexes: {94434923 e0997cd3}
pack 77eac1fa contained in several indexes: {281a5f37 bc5941c5}
pack 8fd16a8e contained in several indexes: {a41a24f5 e33ed426}
pack 5fda3dab contained in several indexes: {37d69098 61d824fa}
pack db7313ec contained in several indexes: {4bc9bd5d a51d15ab}
pack 4588e81e contained in several indexes: {046bfc22 396bd6fb}
pack 7dd417c2 contained in several indexes: {85af1f4b eec1e41a}
pack 0d6ce8ad contained in several indexes: {4c7889cf f0baba37}
pack c4e7ee12 contained in several indexes: {81ecfb9e d783cdb7}
pack 7b8aa79d contained in several indexes: {5ffb2d0b db046242}
pack bc8b44c9 contained in several indexes: {17255de9 3d59e455}
pack 1ccc5c2b contained in several indexes: {1cb5c623 f93a1b72}
pack 819eb149 contained in several indexes: {425295bb c9e10315}
pack 87befbc9 contained in several indexes: {14e5cb0e d049d483}
pack 6e330de0 contained in several indexes: {4267bbf0 b16a867b}
pack 694393a2 contained in several indexes: {7755dae3 99b9d9a7}
pack 5bd8c22f contained in several indexes: {1b67322b df41ad21}
pack 465c6c4d contained in several indexes: {6e2475f2 c98d81d5}
pack 16c446b6 contained in several indexes: {9db0229c fa1a48fe}
pack 91c967e5 contained in several indexes: {72782925 f3ccdba4}
pack 4969f933 contained in several indexes: {3329f6cd d9708e02}
pack 04bea3b4 contained in several indexes: {86227128 f4283a15}
pack f72d3665 contained in several indexes: {ab4232fc eec1e41a}
pack 40b823fc contained in several indexes: {3354ffe7 699b285d}
pack 71a92ae5 contained in several indexes: {6e2475f2 b31bf396}
pack 1dc0dede contained in several indexes: {dd71623c eec1e41a}
pack 9f897a89 contained in several indexes: {840fcc7e 8c0ebdea}
pack 01fe35f1 contained in several indexes: {a9391800 f8e5b136}
pack 29bb21a4 contained in several indexes: {1a1e56b4 c86a7198}
pack 9c423fa2 contained in several indexes: {5b21cd5f 99688264}
pack e7852a14 contained in several indexes: {3315eb47 ca81db80}
pack 4519c530 contained in several indexes: {88dcd6fa d94f846a}
pack 0b9e7718 contained in several indexes: {6574aa8a eec1e41a}
pack 68a362ec contained in several indexes: {833f3460 b1653f3c}
pack 2f544337 contained in several indexes: {4df3e582 8e37be80}
pack 8f55b7df contained in several indexes: {5928c0e3 d6b3edbc}
pack 3705473a contained in several indexes: {11d5de25 30c75821}
pack a8729a7d contained in several indexes: {33e15c74 3a6b5e46}
pack 9858119d contained in several indexes: {26a4d837 40594b42}
pack c6999de6 contained in several indexes: {14e5cb0e ef13306a}
pack e6ee3abc contained in several indexes: {0c01dcb2 cc20d11f}
pack 1af81335 contained in several indexes: {0dc9b9a9 b757564c}
pack 184422c8 contained in several indexes: {33dde731 d083cba3}
pack 2d5cbfb3 contained in several indexes: {39c84791 fb902b39}
pack 1259c9e4 contained in several indexes: {2331ed74 5a9fb906}
This is non-critical, you can run `restic rebuild-index' to correct this
check all packs
pack 7dc10f2c: unexpected file size: got 4777388, expected 1879850
pack 54e3aa52: not referenced in any index
pack cf0febed: not referenced in any index
pack 76ca2642: not referenced in any index
pack 2d202d12: not referenced in any index
pack e1caba0a: not referenced in any index
pack 504b4d50: not referenced in any index
pack ec31d283: not referenced in any index
pack b889e764: not referenced in any index
pack 00e8ed83: not referenced in any index
pack 02e69d97: not referenced in any index
pack 543ce9d0: not referenced in any index
pack 5fb13be5: not referenced in any index
pack a6fbb175: not referenced in any index
pack 97679c37: not referenced in any index
pack 3384a5e3: not referenced in any index
pack e092b2e1: not referenced in any index
pack a99bf803: not referenced in any index
pack f4366bdd: not referenced in any index
pack fb53e713: not referenced in any index
pack f1de3a2e: not referenced in any index
pack 7e21fed1: not referenced in any index
pack 0954e43f: not referenced in any index
pack 2f2a786b: not referenced in any index
pack 3cbee540: not referenced in any index
pack 97c40a04: not referenced in any index
pack c7d0e5e4: not referenced in any index
pack 99859924: not referenced in any index
pack 38cf2a22: not referenced in any index
pack 9a147d19: not referenced in any index
pack 1892c9d6: not referenced in any index
pack fa7e4dea: not referenced in any index
pack 90b63475: not referenced in any index
pack 1140d841: not referenced in any index
pack 1bda34ee: not referenced in any index
pack ebe0baac: not referenced in any index
pack 5b60b5f3: not referenced in any index
pack d6af88ba: not referenced in any index
pack 8df34f7d: not referenced in any index
pack 2059559b: not referenced in any index
pack 942c5f20: not referenced in any index
pack 3e64128e: not referenced in any index
pack 814259a5: not referenced in any index
pack 21b359d4: not referenced in any index
pack a73fa4c6: not referenced in any index
pack d0b096da: not referenced in any index
pack fbde42bf: not referenced in any index
pack e8313983: not referenced in any index
pack 65deb881: not referenced in any index
pack dc6d776d: not referenced in any index
pack 8e2eee3c: not referenced in any index
pack 48493ee3: not referenced in any index
pack 5c7119c7: not referenced in any index
pack a708446f: not referenced in any index
pack ecc25096: not referenced in any index
pack cb0c98ae: not referenced in any index
pack fea17084: not referenced in any index
pack ed6d433b: not referenced in any index
pack 30fb60a6: not referenced in any index
pack 0e4591db: not referenced in any index
pack f86bc585: not referenced in any index
pack 7f1d4937: not referenced in any index
pack 2f3b8397: not referenced in any index
pack 0dab67fc: not referenced in any index
pack 4cc82d3a: not referenced in any index
pack 57fb0ca6: not referenced in any index
pack 45f7eca8: not referenced in any index
pack 3357e82f: not referenced in any index
pack 5efb64fd: not referenced in any index
pack 6ea2f47c: not referenced in any index
pack c0b2ada6: not referenced in any index
pack bd8c21b2: not referenced in any index
pack 6a372360: not referenced in any index
pack 252e9100: not referenced in any index
pack 1724ca6a: not referenced in any index
pack 17374214: not referenced in any index
pack 2a1587a4: not referenced in any index
pack 1fded26b: not referenced in any index
pack 198c6fb6: not referenced in any index
pack d3c8b6d6: not referenced in any index
pack 583b7f29: not referenced in any index
pack b92f4ae6: not referenced in any index
pack 45c0dd89: not referenced in any index
pack b6349420: not referenced in any index
pack 8e81cbea: not referenced in any index
pack 6218ed4e: not referenced in any index
pack b12a16e3: not referenced in any index
pack a9673bbc: not referenced in any index
pack a77a4070: not referenced in any index
pack 225a3655: not referenced in any index
pack 7623f815: not referenced in any index
pack 530c17ee: not referenced in any index
pack 800e0699: not referenced in any index
pack 480e7757: not referenced in any index
pack 37d48480: not referenced in any index
pack d76813d1: not referenced in any index
pack eff30c1a: not referenced in any index
pack 3079b570: not referenced in any index
pack c8d5f175: not referenced in any index
pack 99588f84: not referenced in any index
pack 42b919b5: not referenced in any index
pack 59a23543: not referenced in any index
pack 82434870: not referenced in any index
pack b86c50bc: not referenced in any index
pack 7d36c56f: not referenced in any index
pack 4c315749: not referenced in any index
pack 448a189f: not referenced in any index
pack 6db42778: not referenced in any index
pack c6558bfc: not referenced in any index
pack 6643e8d8: not referenced in any index
pack 3946742a: not referenced in any index
pack 59e336bf: not referenced in any index
pack f3fa4a10: not referenced in any index
pack ad27968d: not referenced in any index
113 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree 833ca51f:
  id 833ca51f317f7ba031942b0eb17c30efb0374c55e38b597d484e1895d579684a not found in repository
error for tree 9215158f:
  id 9215158f7ddfdaefa38b0df2ac634be4789e109f2f5d6dc44034fd36be5633e7 not found in repository
error for tree d4d6959b:
  id d4d6959b83f001f2a5d4eb1cb271289a8ae20708b4b78af9054d8ed6bde8ade6 not found in repository
error for tree 3d4aff38:
  id 3d4aff3891dc8bed7acc74a71a128f141e0598c2d038078f1a161610a49f9785 not found in repository
error for tree c6ef8a94:
  id c6ef8a94542d01b93cc6f8374021be782a9a1676b29fc1829b41b8fe350273b1 not found in repository
error for tree 5bbd62b8:
  tree 5bbd62b8: file "auth.log" blob 311f1d1fc5804a724281ecc5c650eb5bb3ff3a8b31a8273b266b0614a9b0b091 not found in index
  tree 5bbd62b8: file "clamav-unofficial-sigs.log" blob 058402ab8e5c8552646436993bd62747140db8e92bbedf874fd607b503080290 not found in index
  tree 5bbd62b8: file "daemon.log" blob f387080d24e767cb3918e6a067da963e7dfdc869b266e7af32ef563564d69965 not found in index
  tree 5bbd62b8: file "fail2ban.log" blob 4502b95d23972236547a4b6846a7f1efa22bca9a02999bc57f947b7d6744aecd not found in index
  tree 5bbd62b8: file "kern.log" blob d13dc67437da941c1a062ca30f1e2fab7ce96f1dc6a6edecfb4e2c09efd9269e not found in index
error for tree e32818bf:
  id e32818bf7d2d0d19ab8543264ee5d751ff1fd716b209d3520a7c35749b5ce8fd not found in repository
error for tree 919d682d:
  id 919d682d2b66d88a67bab763fa4e850810422116b5c499dc582d8b654f682136 not found in repository
[10:30:32] 100.00%  4797 / 4797 snapshots

Which restic version are you using? I guess the repository is the same as in Migration from ancient version ?

If yes, then it’s probably best to start with running rebuild-index (just create a copy of the index/ folder first). It could simply be the case that the not referenced packs contain the missing blobs. Afterwards please run check again to learn which problems were fixed and which are remaining.

Tree and blob ids are different from pack ids. Only the latter are visible in the form of filenames, whereas the former are only visible by inspecting pack files (files in the data/ folder).

I’m using “restic 0.13.1 compiled with go1.18 on linux/amd64”, and yes: It is the same repository.

You ask, we do it! :slight_smile: (It was running since this morning, but happened to finish just after you requested it! :smiley: )

root@lingo:~/script/backup_mail# restic -v rebuild-index
repository 9478621b opened successfully, password is correct
counting files in repo
[15:54] 100.00%  45733 / 45733 packs
finding old index files
saved new indexes as [2a257c4e b22b0510 3ecafc64 4c369785 d0caa37c 51d3a3aa dd288376 32615df8 60b9386d 8da6498a 6fa80a50 fd25f0c6 4d3f676c 0996f6f7 9d41bd02 cbb8cb78]
remove 8506 old index files
[0:58] 100.00%  8506 / 8506 files deleted

root@lingo:~/script/backup_mail# /home/radel/Downloads/restic_0.13.1_linux_amd64 check --cache-dir=$RESTIC_CACHE_DIR
using temporary cache in /home/restic/.cache/restic-check-cache-548707364
repository 9478621b opened successfully, password is correct
created new cache in /home/restic/.cache/restic-check-cache-548707364
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
Load(<data/6024621abd>, 8549480, 1302782) returned error, retrying after 552.330144ms: unexpected EOF
Load(<data/41cd03f10d>, 6275699, 247190) returned error, retrying after 720.254544ms: unexpected EOF
Load(<data/6024621abd>, 8549480, 1302782) returned error, retrying after 873.42004ms: unexpected EOF
Load(<data/41cd03f10d>, 6275699, 247190) returned error, retrying after 703.28564ms: unexpected EOF
Load(<data/6024621abd>, 8549480, 1302782) returned error, retrying after 1.040217184s: unexpected EOF
Load(<data/41cd03f10d>, 6275699, 247190) returned error, retrying after 1.335175957s: unexpected EOF
[9:21:41] 100.00%  4797 / 4797 snapshots...
no errors were found

It seems everything is fine. Wonderful!

From this experience I take home:

  • If I understand correctly the last errors were only transient transfer errors which restic automatically retried.
  • My old 8506 indexes were replaced by only 16 indexes.
  • I need to forget and prune even in this small repo, not just because to reclaim space, but because everything becomes clogged by thousands of old indexes which needs a lot of memory and cache space
  • Even with the bad things this repo saw (tens of concurrent jobs killed, server hangs, and so on) the repository seems rock solid and no data was lost

I also found two strange things, perhaps they’re not bugs, but this behavior surprised me.

  1. It seems restic forget accepts the snapshot id before the command if '-n" is used, but it doesn’t if the option is not used.
root@lingo:~/script/backup_server# /home/radel/Downloads/restic_0.13.1_linux_amd64 -n e0ca63c6 forget
repository 559ce728 opened successfully, password is correct
Would have removed the following snapshots:
{e0ca63c6}
root@lingo:~/script/backup_server# /home/radel/Downloads/restic_0.13.1_linux_amd64  e0ca63c6 forget
unknown command "e0ca63c6" for "restic"
root@lingo:~/script/backup_server# /home/radel/Downloads/restic_0.13.1_linux_amd64 forget e0ca63c6
repository 559ce728 opened successfully, password is correct
[0:00] 100.00%  1 / 1 files deleted...

That’s an artifact of how command line parsing works. When it encounters an unknown option, -n in this case, then cobra assumes that the option takes one parameter. And thus it correctly guesses which command you meant. And restic e0ca63c6 forget obviously calls the command e0ca63c6.

Yes these were retried automatically, although I’m not sure what caused those.

The performance of several commands also will suffer if there are too many snapshots. A few thousand snapshots are definitely a lot.

The repository format is designed to allow commands to be interrupted at arbitrary points in their execution. (There have been quite a few corner cases that were fixed over time, and I’m pretty sure we’ve caught most of them by now.)

1 Like

Sorry to bother again,

I’m always recovering the same repository. I performed the first check / rebuid-index / check on a copy of the repo, and by mistake I used a 0.11.0 binary in the first rebuild-index. Everything was good, so I was ready for the “production” repo.

This time I made sure to use the latest (0.13.1) version every time, but it seems I cannot get a clean repo.

Each time I perform a check (with or without read-data) it complains about packs in several indexes and suggests a rebuild-index. Effectively the index are rewritten, but it still complains about the packs in multiple indexes.

I tried on different machines, with and without caching, even copying the repo to a local filesysem… Same result.

I also noted that on the test run it created only 16 indexes, but now it seems to need 113 indexes for the same data.

Something has changed in indexing between 0.11.0 and 0.13.1 that could explain this behavior?

I also noted that using caching for read-data creates an almost complete copy of the filesystem: is this expected? As soon as I fix this repo I need to start checking a 4Tb repo, and I don’t have so much space for the cache!

@MichaelEischer I would try a prune, which anyway I need to do, to see if it changes something, but I’ll wait a little bit in case this could be interesting for you to troubleshoot.

Are these the only errors reported by check? How old are the files in the index/ folder? My best guess right now is that some old files are not removed for whatever reason. rebuild-index should replace all of them, so if there are files that are older than the rest, then you can remove the files and run rebuild-index again.

prune and rebuild-index were rewritten in 0.12.0. To fully rebuild the index from scratch it’s now necessary to run rebuild-index --read-all-packs. But the pack in multiple indexes error should be impossible in newly written indexes (We’ve made sure of that somewhere around restic 0.10 or 0.11).

Your repositories predate the repository cache. Modern restic versions separate metadata and data, such that only pack files containing metadata are stored in the cache. Earlier versions just mixed up everything together such that everything has to be cached. So you might have to disable the cache using --no-cache.

These mixed pack files will also cause prune to completely rewrite every mixed pack file. So it might be necessary to limit how much data is rewritten at once by using prune --max-repack-size <size>. Or if you have enough storage space you could just use copy to move everything to a new repository, which will use the modern repository layout.

We should first find out what caused the index problems. If this is caused by some old indexes not being deleted, then that could result in further problems.

All the old errors were fixed some days ago after the rebuild index with the 0.13.1 binary, I suppose. I’m not 100% sure because I worked on a cloned repo and I wasn’t expecting any more issues, so I didn’t took notes about that run.
Anyway now check complains only about the packs in multiple indexes.
Every index file is new, and all the old one are deleted.

The only thing I little off is that rebuild-index from 0.13.1 scans more packs that it thinks would be necessary. The old rebuild-index doesn’t do that.


#restic rebuild-index --no-cache --read-all-packs outputs:

adding pack file to index 6507f0000f41c9778ac73d2a9eea445e5badca77bd14efd4bf1e58ccc5d39312
adding pack file to index fa7810bd039f77e51e51ffae838970619aea08c334e01026570c45b13a370b96
... other similar lines omitted ...
adding pack file to index 5566280c0a54f6aac5fc54ac4e3ba3461429e12f05e0918c4ded924d91b52509
adding pack file to index b64823d0f0e5ed589df84aeedd56cca504280781e4605983a3aa086970b9c591
adding pack file to index 2c6fb0e3df19d169afcac95c8b9e9510022c1eda801064a4cb534375af0a7c6f
reading pack files
[4:41] 100.00%  45733 / 45733 packs...
rebuilding index
[0:55] 100.00%  71813 / 45733 packs processed...
deleting obsolete index files
[0:00] 100.00%  113 / 113 files deleted...
done

moreover a ls |wc -l in data folder reports 45733 files!

Somehow I realized I had to use the --no-cache, but now I know why. :slight_smile:

Is there a way to tell if a repo is in the old or in the new layout? I have others repo and I’ll start to use 0.13.1 binary soon, but first I would like to understand what repo format are using.

this explains the difference. I wonder if the additional packs (71813 / 45733 packs processed) could be related to the source pack in multiple index issue.

I’m pretty sure every index file is new, rebuilt each time I use “rebuild-index --read-all-packs” .

Due to the caching issue I now moved the repo on a local filesystem, and I work on a third copy, so I can revert any change quickly and experimenting without fears.

I think for some reasons 0.13.1 can still produce fresh indexes not optimized, so I think could be valuable for restic itself trying to troubleshoot this issue. If you think it is not just let me know.

I’m not a Go programmer, but I’m starting to become interested in understanding how restic works under the hood :slight_smile:

I can’t think of an easy way. The most useful sidechannel is probably that the cache no longer tries to download the whole repository or that prune doesn’t try to rewrite the whole repository. Let `check` warn about legacy variants of the repo format version 1 by MichaelEischer · Pull Request #3730 · restic/restic · GitHub makes it a bit easier as that will change the check command to complain about repositories still using mixed pack files. Essentially, once prune using any restic version since 0.8.0 has finished for a repository, then there are no more mixed pack files.

I’ll take a look at the code and try to find out what is going on, right now my guess is that it is related to the mixed pack files.

1 Like

I’ve found the problem:

The index is currently iterated first by type and then by pack id. And for packs that contain data and tree blobs, that will result in adding them twice…

3 Likes

The fix is now available at rebuild-index: correctly rebuild index for mixed packs by MichaelEischer · Pull Request #3772 · restic/restic · GitHub

1 Like