Should rebuild-index check if rest-server is in append-only mode?

Features and Ideas
1

Nitpicking, found this today when rebuilding an index to optimize it.

When using Restic with rest-server in --append-only mode, it’ll spend some time trying to figure out how to optimize the index then it’ll write a bunch of errors and end with a backtrace, something like:

Remove(<index/3276140d7a>) returned error, retrying after 298.484759ms: blob not removed, server response: 403 Forbidden (403)
Remove(<index/7af0c8fb38>) returned error, retrying after 400.45593ms: blob not removed, server response: 403 Forbidden (403)
Remove(<index/bf7024ea4f>) returned error, retrying after 507.606314ms: blob not removed, server response: 403 Forbidden (403)
...
...
[0:48] 0.00%  0 / 384 files deleted
blob not removed, server response: 403 Forbidden (403)
github.com/restic/restic/internal/backend/rest.(*Backend).Remove
        /restic/internal/backend/rest/rest.go:333
github.com/restic/restic/internal/backend.(*RetryBackend).Remove.func1
        /restic/internal/backend/backend_retry.go:108
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
        /home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.0.2/retry.go:52
github.com/cenkalti/backoff/v4.RetryNotify
        /home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.0.2/retry.go:31
github.com/restic/restic/internal/backend.(*RetryBackend).retry
        /restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Remove
        /restic/internal/backend/backend_retry.go:107
github.com/restic/restic/internal/cache.(*Backend).Remove
        /restic/internal/cache/backend.go:38
main.deleteFiles.func2
        /restic/cmd/restic/delete.go:42
golang.org/x/sync/errgroup.(*Group).Go.func1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1374

This can take quite a while (specially with restic < 0.12) for large repos, and presumably out of restic’s control when not using rest-server, so should restic check if it’s using rest-server in append-only mode and fail fast with a friendlier error message when trying to rebuild the index? Maybe adding an HTTP endpoint to rest-server that restic can check to figure out if it’s in append-only mode without having to try a delete?

Congrats on Restic 0.12, brilliant release. Thank you.

1 Like
2

Presumably this will affect other restic commands that delete data from the repo. I never use those though, so not really sure.

3

Yes, this affects forget, prune and rebuild-index. forget is not that critical, as it “only” tries to delete unneeded snapshot files, but prune and rebuild-index` first write new files and at the last step try to remove the then-obsolete files.

This proceeding however is done for a very good reason: If one of the commands is cancelled for some reason, it should still leave the repo in a sane state. That wouldn’t be the case if first files were removed and then the new files were written.

The problem is - as you correctly pointed out - that restic is not able to detect that deletion is permitted until it attempts to really perform it. With rest-server, the API could be extended, but then this should be also added to other backends which can also implement an append-only repo (e.g. S3). So this would require a major rework in the backend interface.

I agree however that restic shouldn’t retry if a 403 - Forbidden is returned by the backend…

4

Makes sense. On a second thought, it’d be weird for Restic to behave differently here depending on the backend used.

Thanks for the clarity @alexweiss.