Repo corrupted with Box Drive, now attempting to repair/recover

I back up my computer to a restic repo on Box.com via rclone’s restic support. This can take me a long time though, so I thought that maybe backing up to the ~/Box folder on my computer might be faster, or at least let restic finish faster while Box Drive continues uploading. This was a disaster and Box Drive could not keep up with restic. Now my data directory is full of duplicate folders like this:

data
├── 00
├── 00\ (username@email.com\ 2)
├── 00\ (username@email.com\ 3)
├── 00\ (username@email.com\ 4)
├── 00\ (username@email.com\ 5)
├── 00\ (username@email.com\ 6)
├── 00\ (username@email.com)
...

I’m thinking that if I can de-duplicate and unify the folders, maybe I can fix the repo. Failing that, is there a way to get the repo back in a consistent state? Or do you think it’s completely hosed?

I guess first step is to keep accessing the repo via restic and rclone and not through that other stuff.

That said, make sure you run restic 0.12.1 and then do a restic check on the repository, it will tell you how things are with it. Please pastebin the complete command and output, then we have something to go by.

In general, an interrupted backup run isn’t a problem, it will just result in some unused files, so unless you did something else than backing up you might be fine :slight_smile:

Thanks, I’ll try running restic check overnight and report back.

Ok, it finished earlier than I expected. Discourse did not like how big the output was, so here it is as a Gist:

It appears to show that a bunch of packs are missing.

So, a backup run with restic will only add files to the repository. So if there are pack files missing, that presumably is because you moved the repository to your Box folder, and the Box client never fully synced that complete repo in the first place (never mind that it didn’t sync the changes you made when you backed up after moving the repository to the Box folder).

Don’t you still have the complete repository in that Box folder locally on your computer though? I mean, if you moved the entire repository there, and even if you did additional backups to that repository in your local Box folder, shouldn’t all files be in that folder still?

You could run a restic check on your computer, using the local path to the repository in that Box folder, to see if that way you have all the files. Or am I missing something here?

In general I’m curious if any part of the missing files is due to the refusal from Box to let you/restic list the files (the GOAWAY errors). I don’t know.

The big problem is that restic is unable to properly list all pack files. As long as that operation fails as in that log, the big list of missing pack files is not reliable in any way (and probably way too long).

Maybe rclone has an option to not use HTTP2 when connecting to Box Drive?

I ran restic check again using the local path to my restic repo and it looks more or less the same.

I’ve been reading the restic design document and given what I’ve read so far, I’m reasonably confident that if I take all the loose packs in the duplicate data/[0-9a-f]{2} directories I might have a chance of fixing this. Otherwise, I’ll probably just remove all the bad directories and delete the last snapshot.

@MichaelEischer or someone similar is probably best suited to answer what your best chances are. Seems like Box caused a real mess in that it hasn’t synced the repository properly at all.

As long as restic is not able to properly list all files and read them, there’s not much that can be done here. The best bet might be to move all files to a new location and then try to put the pieces back together.

Seems like a big boo-boo on Box’s behalf.

I have been backing up to a local Dropbox folder, with Dropbox then responsible to syncing to the cloud. It’s not my only recovery level, but helpful for accidental deletions or modifications of files since I then have the backup history locally on my laptop as well. No problems at all with the Dropbox sync, for close to six months now. I’ve done forgets, prunes, and checks, and Dropbox has been playing along nice throughout all that.