"Out of content" errors

Hmm just ran prune on my B2 repository, and am getting “out of content” errors? How should I proceed? Restic check --read-data then find whatever is missing using find --tree, etc and forget those snapshots?

Restic check found nothing. Another restic prune netted this:

finding data that is still in use for 187 snapshots
[2:52] 100.00% 187 / 187 snapshots
searching used packs…
collecting packs for deletion and repacking
[0:59] 100.00% 257424 / 257424 packs processed

to repack: 0 blobs / 0 B
this removes 0 blobs / 0 B
to delete: 0 blobs / 0 B
total prune: 0 blobs / 0 B
remaining: 3457531 blobs / 1.180 TiB
unused size after prune: 37.407 GiB (3.10% of remaining size)

deleting unreferenced packs
Remove(<data/981caf084d>) returned error, retrying after 552.330144ms: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 1.080381816s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 1.31013006s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 1.582392691s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 2.340488664s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 4.506218855s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 3.221479586s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 5.608623477s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 7.649837917s: Delete: 416: out of content
Remove(<data/981caf084d>) returned error, retrying after 15.394871241s: Delete: 416: out of content
unable to remove <data/981caf084d> from the repository
[0:45] 100.00% 1 / 1 files deleted
done

I had a recent local sync that was only a few snapshots behind. I ran prune, and didn’t have the same error, so I’m just going to rclone the working local repo to b2, rather than use --read-data and pay for the egress. :man_shrugging:t2:

Curious what would cause this error, and what would be the fix had I not had a local copy?

I have no cue what causes the error. But it seems to be either an issue of B2 or of your connection… Can you check the B2 docu under which conditions they return a 416 error?

As prune in your case only wanted to remove one unreferenced pack file, you could easily have done this directly on your B2 repo. (And maybe you would have stumbled over the same error allowing you to find the cause)
prune --dry-run -v -v should have printed out exactly what prune intended to do.

1 Like

It looks like the error is:

416
range_not_satisfiable
The Range header in the request is outside the size of the file.

Ok, this is strange. I didn’t know that restic (or the B2 backend implementation, accordingly) even wants to send the Range header for a Delete request.

If you haven’t yet repaired this, it would be interesting if you encounter the same behavior with a rclone:b2 backend as with a direct b2 backend…

Unfortunately I’ve already “repaired” it by rcloning a working local copy, just a few snapshots earlier. Looks like rclone was able to delete the file outside of restic, at least? If it happens again, I’ll try to dig into it deeper in case it’s a bug.

Screen Shot 2021-04-30 at 10.24.38 PM

Also am I the only one getting a 500 error every time I post here? lol

Yes, could be a bug in the b2 backend within restic…

I’m also getting these annoying 500 errors in the forum

1 Like

Sorry for the post errors, it looked like an upgrade of Discourse was not applied cleanly. It should be fixed by now. Please let me know if you still receive these errors. Have a nice weekend!

3 Likes

The files with the 416 errors were probably empty. The B2 library used in restic currently tries to access the first byte of a file to determine the file version id which is then used to delete a file afterwards.

There is a commit in the upstream version of the B2 library which would solve that problem [b2] use HEAD instead of Range 0-1 to stat files · kurin/blazer@3ad36f1 · GitHub .

So probably an empty file got uploaded to B2.

1 Like