rebuild-index actually did help on one repository, the most damaged one. It looks like it stopped with the retries on missing file, and I was able to rsync out of it most data. Luckily all the lost files were in ~/.cache.
Wish me good luck for when I restore my postgres db 
I also read this comment, I’ll make sure I mirror my backups this way.