We’ve run into this before and could not find a solution last time. Now it is happening again. The device with the restic repo fills up, and then we’re screwed. We can’t even remove data, because commands like forget or prune just abort, complaining “no space left on device.”
Well thanks, Captain Obvious.
The last time this happened, we ended up just deleting the whole repo and starting from scratch. Not a great solution. Is there another way?
Yes, there is. We’ve recently merged a pull request which adds this. Use the latest version from the master branch (beta binaries here) and follow the instructions in the docs.
Please report back! 
Thanks! I downloaded the new version and read the documentation, but it’s not clear what it wants for SOME-ID. Hopefully we’re not talking about a snapshot ID here? Because I wouldn’t know what snapshot ID to put in there, as there are 900 of them.
I just tested it. It seems you have to pass “some ID”, like 1234, for example. Since this is the wrong ID to authorize this potentially unsafe option, the command will fail, and the correct ID is shown to you.
Edit: …and it seems to be the repository ID.
Thanks much, that appears to have worked. I passed ‘blah’ as an ID, and it said, “Fatal: must pass id '753ab400ecc2304e32a8ff6a29bbcf9d787b5f4560ebfc2bc3ef3bb447b1e370”
To the documentation maintainers, I would recommend the following edit. Change…
To prevent accidental usages of the --unsafe-recover-no-free-space option it is necessary to first run prune --unsafe-recover-no-free-space SOME-ID and then replace SOME-ID with the requested ID.
…to…
To prevent accidental usages of the --unsafe-recover-no-free-space option it is necessary to first run prune --unsafe-recover-no-free-space ANY_RANDOM_STRING. Restic will respond with the appropriate ID. Then re-run the command, replacing ANY_RANDOM_STRING with the ID that restic provides.
Can you open a PR on github to make the documentation change? Or should we just copy&paste it?
IMHO it’s worth reconsidering whether this slightly unusual mechanism is actually improving anything. I think it’s safe to assume that a user who went through looking up and passing that verbose and lengthy parameter --unsafe-recover-no-free-space is aware of what they’re about to do. And even if they’re not – c/p some random string because the docs say so won’t make much difference. Just my $0.02.
I agree. Normally the same thing is accomplished with a simple --force flag or something similar.
Whichever you prefer.
What about offering restic prune scratch space on some other file system? You might even define a tmpfs if enough RAM is available.
Yes, it might be slow - but are there risks of losing data?
restic prune could use that as a last resort if the repo filesystem is full, so maybe the other filesystem is only needed for some initial stage?
@forbin A PR would be the easiest way for us.
On Linux you can use overlayfs to provide the additional scratch space. Deleting previously used files is the last step of prune. Until then it’s easy to revert by just deleting all files added by prune.
Now I’m getting errors when trying to follow the docs.
[root@store51a zpool0]# restic -r restic_repo prune --max-repack-size 0 --unsafe-recover-no-free-space b23ace48f051c4d0733b8c6ca4c2e4ab612daea080dcf5e3670aab0607543d36
repository b23ace48 opened (repo version 1) successfully, password is correct
open restic_repo/locks/2ced49991ba50f409e280a27563c5bd6806ed4655825cf3856906ed3946c38ac-tmp-2511701826: no space left on device
github.com/restic/restic/internal/backend/local.(*Local).Save
/home/beta/restic.git/internal/backend/local/local.go:148
github.com/restic/restic/internal/limiter.rateLimitedBackend.Save
/home/beta/restic.git/internal/limiter/limiter_backend.go:30
github.com/restic/restic/internal/backend.(*RetryBackend).Save.func1
/home/beta/restic.git/internal/backend/backend_retry.go:66
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
/home/beta/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.2/retry.go:55
github.com/cenkalti/backoff/v4.RetryNotify
/home/beta/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.2/retry.go:34
github.com/restic/restic/internal/backend.(*RetryBackend).retry
/home/beta/restic.git/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Save
/home/beta/restic.git/internal/backend/backend_retry.go:60
github.com/restic/restic/internal/cache.(*Backend).Save
/home/beta/restic.git/internal/cache/backend.go:59
github.com/restic/restic/internal/repository.(*Repository).SaveUnpacked
/home/beta/restic.git/internal/repository/repository.go:509
github.com/restic/restic/internal/repository.(*Repository).SaveJSONUnpacked
/home/beta/restic.git/internal/repository/repository.go:448
github.com/restic/restic/internal/restic.(*Lock).createLock
/home/beta/restic.git/internal/restic/lock.go:163
github.com/restic/restic/internal/restic.newLock
/home/beta/restic.git/internal/restic/lock.go:107
github.com/restic/restic/internal/restic.NewExclusiveLock
/home/beta/restic.git/internal/restic/lock.go:75
main.lockRepository
/home/beta/restic.git/cmd/restic/lock.go:42
main.lockRepoExclusive
/home/beta/restic.git/cmd/restic/lock.go:27
main.runPrune
/home/beta/restic.git/cmd/restic/cmd_prune.go:155
main.glob..func19
/home/beta/restic.git/cmd/restic/cmd_prune.go:36
github.com/spf13/cobra.(*Command).execute
/home/beta/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
/home/beta/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
/home/beta/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
main.main
/home/beta/restic.git/cmd/restic/main.go:98
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571
unable to create lock in backend
That looks like there is no free space available at all. Is there a way for you to free at least 8kb of space? Otherwise you’ll have to comment out restic/cmd_prune.go at 1dd4b9b60e6d25fa3cf7e2a05d347d15eb58b30b · restic/restic · GitHub
I was able to free some space and then the command worked.