Hi, I’m trying to restore a backup from B2, but I’m getting errors. Specifically, I ran restic restore latest --target [directory] --verbose and I am getting a few lines of this at the beginning:
Load(<data/82de07070c>, 17340938, 0) returned error, retrying after 10.992452277s: ReadFull: unexpected EOF
followed by (many more—over a thousand) lines of:
ignoring error for [file]: StreamPack: ReadFull: unexpected EOF
and a few more lines of
ignoring error for [file]: UtimesNano: no such file or directory
before
Fatal: There were 1383 errors
The restore seems to have worked fine other than the files mentioned in the errors, which seem to be missing entirely. But I’m not sure how to test if the other files are ok.
Note: this may not be a restic issue, but since I’m not sure, I figured it makes sense to seek advice The reason I think it might not be restic’s fault is because my disk seemed to be failing to begin with during my backup. The filesystem (NTFS) seemed to be corrupted, at least. (Scuffed… I know.) So it’s entirely possible the files weren’t uploaded properly to begin with. But I don’t recall seeing any errors during backup.
I still have all the files around, so no data loss, but it would be a hassle to restore them all. I’d prefer to somehow repair the backup if possible, if that’s where the problem is. More importantly though, I want to understand what the errors mean and what it might mean for the files that did restore.
restic version says: restic 0.15.2 compiled with go1.20.3 on linux/amd64. I’m on Ubuntu Server 22.04.2. restic check finds no errors.
That looks like one of the pack files failed to download. Please run restic check to let restic verify whether the repository is intact (it will have to download all data that is usually stored in the cache used by restic).
I think that is a result of the previous failures which lead to missing files in the restored data.
Then it looks like the repository is still intact and that the problem was only temporary. You could try to restore some of the failed files again just to be sure.
I ended up copying the missing files via mounting the repository, but many do seem to be corrupted. Could a broken filesystem or failing drive when I made the original backup cause this? Should I trust the files that did restore? (I don’t know too much about the topic, as you can probably tell, but I’m trying to learn!) Is this an issue with restic or a pebkac error? Let me know if there’s any more info I could give that might help diagnose the situation.
Thank you for the help, at any rate, and to the contributors to the project!
That is the copy operation for the files completed successfully, without restic printing lots of error messages?
If the filesystem returns garbage during a restic backup operation, then the backup will contain that exact garbage. There’s nothing that can be done about that particular case. If only the restic repository was damaged, then it should be possible to salvage some data. If the latter case occurs, then restic will complain loudly.
The StreamPack: ReadFull: unexpected EOF rather looks like a temporary issue, that should resolve itself by restoring the affected files a second time.
Sorry for resurrecting this topic (is “necroing” even considered bad etiquette anymore?). I have been using restic for a while and learned quite a bit about it, so I’d like to check my run my understanding past you guys, just to be sure:
Regardless of how broken or corrupted the original filesystem was, that wouldn’t cause any issues with the restic repo itself. The repo will of course contain garbage if the filesystem returned garbage during backup, but that would not raise any errors during any restic operations. So the errors in my original post cannot be caused by that.
Additionally, restic stores all data with checksums. This guarantees that any data that makes its way into the repo will either be restored exactly as uploaded, or an error will be thrown. There is no way for a restore to give you different data from what you backed up. (Barring any exotic and vanishingly unlikely circumstances, such as an attacker on B2’s side abusing a potential hash collision, etc.)
My repo seems fine. I’ve run restic check --read-data a couple times with no errors, and just to be sure, diff-ed test restores against the original data with no differences. (Though, as I said above, this should not be necessary.) Since as you say:
and since the data is there and I could copy it from a mounted snapshot with no issues, then I suppose that perhaps B2 was simply partially down when I ran the restore, or had some issue that prevented me from downloading a few pack files.
Is my understanding right?
Thanks again for the help, and for creating this great tool of course!
The reason I think it might not be restic’s fault is because my disk seemed to be failing to begin with during my backup. The filesystem (NTFS) seemed to be corrupted, at least. (Scuffed… I know.) So it’s entirely possible the files weren’t uploaded properly to begin with. But I don’t recall seeing any errors during backup.
Let me know if there’s any more info I could give that might help diagnose the situation.