Rebuild-index and prune fail when trying to delete an index that doesn't exist

Restic version: 0.12.0 compiled with go1.15.8 on linux/amd64
Backblaze B2 backend.

Trying to either rebuild the index or prune causes the following error:
Remove(<index/00db1aa600>) returned error, retrying after 511.910153ms: Delete: b2_download_file_by_name: 404: file_state_none

It appears that the index file doesn’t exist on the remote server, I believe because the previous full run hit my bandwidth cap and left things in a foul state.

Any ideas how to solve this? I’ve found similar threads for lock files, but nothing about indexes.

Thanks.

Full output:

# restic rebuild-index --verbose=3
repository 2857dca1 opened successfully, password is correct
loading indexes...
getting pack files to read...
rebuilding index
[1:10] 100.00%  392699 / 392699 packs processed
deleting obsolete index files
removed <index/321bbb3ba6> deleted
removed <index/4917056f3c> deleted
removed <index/04a072e4f1> deleted
removed <index/903bff40d7> deleted
removed <index/9fb1aee14b> deleted
removed <index/7e465e4604>s deleted
removed <index/793c8a3d6f>s deleted
removed <index/360bd8e862>s deleted
removed <index/c4313dc5e7>
removed <index/c614d289ec>s deleted
removed <index/d48379b8fd>es deleted
removed <index/d1fdfee6f6>es deleted
removed <index/0d367b807c>es deleted
removed <index/9bab2a6fd0>es deleted
removed <index/95acd967d5>es deleted
removed <index/b77f25387c>es deleted
removed <index/1759dfb953>
removed <index/508fe18d75>es deleted
removed <index/3f63e0be5e>es deleted
removed <index/81e47476b5>es deleted
removed <index/43db60eba3>es deleted
removed <index/5051d84772>es deleted
removed <index/9136af247a>es deleted
removed <index/236ad6f921>es deleted
removed <index/6e9043d948>es deleted
removed <index/7852cbc518>es deleted
removed <index/3fa897f586>es deleted
removed <index/b5072dd409>es deleted
removed <index/9adb489233>es deleted
removed <index/7fd83199ed>es deleted
removed <index/a622327dff>es deleted
removed <index/79cdb81f70>es deleted
removed <index/44e004cfc0>es deleted
removed <index/532a25fa10>es deleted
removed <index/914a85cd44>es deleted
removed <index/c3cb1235bd>es deleted
removed <index/03ad65b4aa>
removed <index/ba3ce99132>es deleted
removed <index/92f43cca6f>es deleted
removed <index/2447490100>es deleted
removed <index/3f8aae3dc8>es deleted
removed <index/8687e9f3d1>es deleted
removed <index/1cd16f52b4>es deleted
removed <index/bc04ab2473>es deleted
removed <index/2f79cc3b70>es deleted
removed <index/6be8492f5d>es deleted
removed <index/79a3fbb8d8>es deleted
Remove(<index/00db1aa600>) returned error, retrying after 511.910153ms: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 396.227312ms: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 740.619312ms: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 1.868490179s: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 3.734205348s: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 2.200112976s: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 6.235277092s: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 6.847383216s: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 13.380282645s: Delete: b2_download_file_by_name: 404: file_state_none
Remove(<index/00db1aa600>) returned error, retrying after 19.80958851s: Delete: b2_download_file_by_name: 404: file_state_none
unable to remove <index/00db1aa600> from the repository
[1:07] 97.92%  47 / 48 files deleted
b2_download_file_by_name: 404: file_state_none
Delete
github.com/restic/restic/internal/backend/b2.(*b2Backend).Remove
	/restic/internal/backend/b2/b2.go:261
github.com/restic/restic/internal/backend.(*RetryBackend).Remove.func1
	/restic/internal/backend/backend_retry.go:108
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
	/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.0.2/retry.go:52
github.com/cenkalti/backoff/v4.RetryNotify
	/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.0.2/retry.go:31
github.com/restic/restic/internal/backend.(*RetryBackend).retry
	/restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Remove
	/restic/internal/backend/backend_retry.go:107
github.com/restic/restic/internal/cache.(*Backend).Remove
	/restic/internal/cache/backend.go:38
main.deleteFiles.func2
	/restic/cmd/restic/delete.go:42
golang.org/x/sync/errgroup.(*Group).Go.func1
	/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1374

@brettp Thanks for your report!

Can you assure that the file index/00db1aa600 really does not exist? (e.g. by using rclone or the web gui of B2?

If it does exist, I would assume that there in fact is a problem removing a that file. Try to remove it by yourself or check if there is something odd with it.

If the file really does not exist when you start rebuild-index or prune there shouldn’t be an attempt to remove it. AFAIR both commands first do a List() on the index files and only attempt to remove files that were listed.

1 Like

@alexweiss - Thanks for the quick response.

I confirmed that the file exists through both the web UI and the b2 CLI, but I hadn’t tried to download it. When I did, B2 did in fact throw a b2_api_error: not_found error.

Removing the file manually seems to have done the trick, and restic is working as expected.

Thanks again for the help!

Might want to report that to B2 so that they can fix your broken storage :confused: