Restic prune with Backblaze returns errors (b2.b2errdeleted)

We are using restic to backup a large amount of files (105TB and 2.9M files) to Backblaze.
When running prune, I get large amount of errors (see below). I am also curious why restic is trying to download files during delete even if I specified --max-unused=unlimited

This looks similar but was apparently resolved "Data blobs seem to be missing" error during prune on Backblaze · Issue #3266 · restic/restic · GitHub

restic was run like this:
restic -r b2:my-bucket prune --max-unused=unlimited

restic version
restic 0.13.1 compiled with go1.18 on linux/amd64

some sample output from the prune:

Remove(<data/7267cecbf1>) returned error, retrying after 485.170407ms: Delete: b2_delete_file_version: 400: File not present: data/72/7267cecbf1f9df720cf182a77f8ce9bcf4eab312dc859d53f6d5864ec6a72214 4_z1fee766953e756af7acf0d15_f117e9a35e3be47b2_d20220422_m181239_c001_v0001165_t0040_u01650651159530
b2_download_file_by_name: 404: : b2.b2errdeleted...
Remove(<data/f4da7f8f9a>) returned error, retrying after 625.193692ms: Delete: b2_delete_file_version: 400: File not present: data/f4/f4da7f8f9af85bfad06fe5a60fc3b3c637c55addbba7fe81dce3317d6aa78975 4_z1fee766953e756af7acf0d15_f1045f6585701de4e_d20220422_m151759_c001_v0001165_t0031_u01650640679854
b2_download_file_by_name: 404: : b2.b2errdeleted...
Remove(<data/033423e537>) returned error, retrying after 680.391926ms: Delete: b2_delete_file_version: 400: File not present: data/03/033423e53779220bff4bdfa3c3666c5788175f152a806ce7b7bc6ef17df36ae8 4_z1fee766953e756af7acf0d15_f11202c37bc097d7f_d20220125_m034029_c001_v0001165_t0002
b2_download_file_by_name: 404: : b2.b2errdeleted...
Remove(<data/dfb4935bc3>) returned error, retrying after 520.819355ms: Delete: b2_delete_file_version: 400: File not present: data/df/dfb4935bc31bdc9b45d2bc13b99f5c2b99a9cd18868dff8efe811f7a696e3bec 4_z1fee766953e756af7acf0d15_f1045f65857010f4f_d20220422_m140001_c001_v0001165_t0057_u01650636001684
b2_download_file_by_name: 404: : b2.b2errdeleted...
Remove(<data/77da6d16a3>) returned error, retrying after 723.864006ms: Delete: b2_delete_file_version: 400: File not present: data/77/77da6d16a310482cfc341448402e70f590db13585207495d52b83d95cec35deb 4_z1fee766953e756af7acf0d15_f1045f658570390c8_d20220422_m174634_c001_v0001165_t0020_u01650649594149
b2_download_file_by_name: 404: : b2.b2errdeleted...
Remove(<data/bc8961d135>) returned error, retrying after 548.638399ms: Delete: b2_delete_file_version: 400: File not present: data/bc/bc8961d135ab15287a4d1ec9098cd77f8022f968cc6d558b4812f96b4ff20614 4_z1fee766953e756af7acf0d15_f11202c37bc09c85a_d20220125_m041351_c001_v0001165_t0021
Remove(<data/60e1b8e337>) returned error, retrying after 286.646999ms: Delete: b2_delete_file_version: 400: File not present: data/60/60e1b8e337d582b1873ea35ae59cb163fda916e7680ff54855759b42d1031207 4_z1fee766953e756af7acf0d15_f1045f65857001818_d20220422_m122331_c001_v0001165_t0011_u01650630211624
b2_download_file_by_name: 404: : b2.b2errdeleted...
b2_download_file_by_name: 404: : b2.b2errdeleted...
Remove(<data/b84f85c81d>) returned error, retrying after 593.198002ms: Delete: b2_delete_file_version: 400: File not present: data/b8/b84f85c81db65e5078c722f1db0819b9e0df30f2f171904f2aa61e0f7145f6ad 4_z1fee766953e756af7acf0d15_f117e9a35e3be1e45_d20220422_m175851_c001_v0001165_t0017_u01650650331377
b2_download_file_by_name: 404: : b2.b2errdeleted...
[13:30:35] 66.65% 581087 / 871884 files deleted...

(note prune running over 13h).

Not sure about the error message, but I avoid repacking (and downloading unnecessary data, which makes sense for my use case) by using restic prune —max-repack-size 0.

I thought --max-unused=unlimited would do the same.

In fact --max-repack-size 0 and --max-unused=unlimited is not exactly the same. The first would also not repack tree packs while the second always repacks partly used tree packs.

However, I strongly recommend to repack tree packs:

  • They are anyway needed to be readable for many commands and are usually cached. So repacking those is not too much effort
  • If you are using a cache, repacking partly used tree packs also reduces the size needed for the cache

In fact the option --max-repack-size IMO is only interesting if you want to bound repacking to a certain amount (usually > 0), e.g. because your available space is bounded or you want prune to finish in a more predictable time.

Moreover, in many settings, the default (which is unbounded --max-repack-size and --max-unused=5%) already results in very few repacks - except trees. But if accessing your storage is expensive compared to just holding some data, it is a good idea to increase --max-unused (and, of course, to use a cache).

1 Like

Can someone add this information to the docs near the description of these two options? I think it would be very valuable.

@vsuontam The b2_download_file_by_name lines are `b2_download_file_by_name: 404` on Backblaze B2 repository init · Issue #3750 · restic/restic · GitHub which is just a left-over debug message in the library we use to access B2, but nothing to worry about.

The Delete: b2_delete_file_version: 400: File not present warnings are likely the result of B2 internally retrying the delete request without noticing that it already succeeded. But that’s also not a real problem.

Thanks @MichaelEischer for the reassuring note. Just FYI the prune on my massive repo took 23h to complete, nothing too severe here IMO. 20h for deletions, and the rest for loading the index, etc. The process was using about 24GBs of memory.

[20:03:03] 100.00%  871884 / 871884 files deleted...
done

real    1390m32.132s
user    432m37.544s
sys     16m45.680s

(in before running the prune, I deleted about 6 snapshots, and kept 4 before prune). Restic reported that about 0.11% “unused space” was used IIRC.

Now running “restic check” for the first time. I will also time that operation and can report the findings/time if it is of interest.