Out of space and even prune --unsafe-recover-no-free-space fails

Getting Help
1

Output of restic version

restic 0.18.1 compiled with go1.25.1 on darwin/arm64

What backend/service did you use to store the repository?

USB drive

Problem description / Steps to reproduce

  1. I have a 5TB hard drive that I use only for restic backups
  2. I currently have 768Ki free on the drive
  3. I tried forget followed by prune --max-repack-size 0 and restic crashed
  4. I tried prune --unsafe-recover-no-free-spaceand restic crashed
  5. Something must have happened after running those commands as I’m now getting “invalid data returned” on a lot of commands

For #3 and #4 I’d paste the output but it’s now too far back in my terminal history

Commands I tried

The following commands:

restic -r ./ prune
restic -r ./ prune --max-repack-size 0
restic -r ./ prune --unsafe-recover-no-free-space {very_long_id}

are all returning:

enter password for repository:
repository b84f41ca opened (version 2, compression level auto)
loading indexes...
[0:01] 68.53%  577 / 842 index files loaded
LoadRaw(<index/6a6533c7ed>): invalid data returned

Check:

restic -r ./ check
[...]
pack 12a633c30fdb630405d6ae5587ee0d4eaa462e6a63e5c050d0381c8f861c20ce contained in several indexes: {a30e4b48 a9e0756d}
pack d4a45d703c33a0284d459bd3a0d226fccc68ee05b24237c65cfdb6975a4cce73 contained in several indexes: {5b7f04a6 d40d1362}
Duplicate packs are non-critical, you can run `restic repair index' to correct this.
error: error loading index 44750a6e894d29fa9f019a75e5712747b2d9aebbfaa3c360640c8aad27549124: LoadRaw(<index/44750a6e89>): invalid data returned
error: error loading index 6a6533c7ed4d17045f2540090060cc6f129b8ba5e14ddc35f15ab8f388f36a86: LoadRaw(<index/6a6533c7ed>): invalid data returned

The repository index is damaged and must be repaired. You must run `restic repair index' to correct this.

Fatal: repository contains errors

Repair index:

restic -r ./ repair index
[...]
reindexing pack file feba3361087da446fddd65479b6f140daa3f7ab1b73e7fd4021786f11be83751 with unexpected size 14155776 instead of 18690442
reindexing pack file fe804fc67244f0134f8fd1c5712c5d2e7776526b30701df1ea37b7d80f0340a5 with unexpected size 0 instead of 16970825
reading pack files
[0:14] 100.00%  796 / 796 packs
Save(<index/20cd2f7a00>) failed: write /Volumes/Backup/Restic/index/20cd2f7a0022e3ebfd76cdac178dc5fa85cb2bcccc4ed6a5045b8bb6f83b34ff-tmp-85884661: no space left on device
write /Volumes/Backup/Restic/index/20cd2f7a0022e3ebfd76cdac178dc5fa85cb2bcccc4ed6a5045b8bb6f83b34ff-tmp-85884661: no space left on device
github.com/restic/restic/internal/backend/local.(*Local).Save
	github.com/restic/restic/internal/backend/local/local.go:162
github.com/restic/restic/internal/backend/limiter.rateLimitedBackend.Save
	github.com/restic/restic/internal/backend/limiter/limiter_backend.go:45
github.com/restic/restic/internal/backend/sema.(*connectionLimitedBackend).Save
	github.com/restic/restic/internal/backend/sema/backend.go:73
github.com/restic/restic/internal/backend/logger.(*Backend).Save
	github.com/restic/restic/internal/backend/logger/log.go:31
github.com/restic/restic/internal/backend/retry.(*Backend).Save.func1
	github.com/restic/restic/internal/backend/retry/backend_retry.go:172
github.com/restic/restic/internal/backend/retry.(*Backend).retry.func1
	github.com/restic/restic/internal/backend/retry/backend_retry.go:137
github.com/restic/restic/internal/backend/retry.retryNotifyErrorWithSuccess.func1
	github.com/restic/restic/internal/backend/retry/backend_retry.go:53
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1
	github.com/cenkalti/backoff/v4@v4.3.0/retry.go:18
github.com/cenkalti/backoff/v4.doRetryNotify[...]
	github.com/cenkalti/backoff/v4@v4.3.0/retry.go:88
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
	github.com/cenkalti/backoff/v4@v4.3.0/retry.go:61
github.com/cenkalti/backoff/v4.RetryNotify
	github.com/cenkalti/backoff/v4@v4.3.0/retry.go:49
github.com/restic/restic/internal/backend/retry.retryNotifyErrorWithSuccess
	github.com/restic/restic/internal/backend/retry/backend_retry.go:62
github.com/restic/restic/internal/backend/retry.(*Backend).retry
	github.com/restic/restic/internal/backend/retry/backend_retry.go:135
github.com/restic/restic/internal/backend/retry.(*Backend).Save
	github.com/restic/restic/internal/backend/retry/backend_retry.go:166
github.com/restic/restic/internal/backend/cache.(*Backend).Save
	github.com/restic/restic/internal/backend/cache/backend.go:75
github.com/restic/restic/internal/repository.(*Repository).saveUnpacked
	github.com/restic/restic/internal/repository/repository.go:496
github.com/restic/restic/internal/repository.(*internalRepository).SaveUnpacked
	github.com/restic/restic/internal/repository/repository.go:465
github.com/restic/restic/internal/repository/index.(*Index).SaveIndex
	github.com/restic/restic/internal/repository/index/index.go:374
github.com/restic/restic/internal/repository/index.(*MasterIndex).saveIndex
	github.com/restic/restic/internal/repository/index/master_index.go:586
github.com/restic/restic/internal/repository/index.(*MasterIndex).Flush
	github.com/restic/restic/internal/repository/index/master_index.go:599
github.com/restic/restic/internal/repository.(*Repository).Flush
	github.com/restic/restic/internal/repository/repository.go:547
github.com/restic/restic/internal/repository.RepairIndex
	github.com/restic/restic/internal/repository/repair_index.go:108
main.runRebuildIndex
	github.com/restic/restic/cmd/restic/cmd_repair_index.go:84
main.newRepairIndexCommand.func1
	github.com/restic/restic/cmd/restic/cmd_repair_index.go:35
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.10.1/command.go:1015
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.10.1/command.go:1148
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.10.1/command.go:1071
github.com/spf13/cobra.(*Command).ExecuteContext
	github.com/spf13/cobra@v1.10.1/command.go:1064
main.main
	github.com/restic/restic/cmd/restic/main.go:172
runtime.main
	runtime/proc.go:285
runtime.goexit
	runtime/asm_arm64.s:1268

Repair snapshots:

restic -r ./ repair snapshots
enter password for repository:
repository b84f41ca opened (version 2, compression level auto)
[0:01] 69.71%  587 / 842 index files loaded
LoadRaw(<index/6a6533c7ed>): invalid data returned

Any thoughts on how to get out of this situation?

2

This is what I would do:

If you do not have larger disk to copy everything to, use one more external disk or even system disk directory “add” extra space to your repo drive. Use mergerfs or rclone union remote to achieve it. Most likely it is far better and safer option than trying to run “random” restic commands which can damage your repo beyond repair.

This will give you space to breathe and opportunity to get out of the hole you put yourself in.

1 Like
3

@evess I agree that whatever you do next, you must have free space available for that operation to succeed. I’m curious, can you tell us how large your repository’s index/ directory is currently?

4

@kapitainsky thank you! do I understand correctly — I could use mergerfs to expand the size of the repo drive. at that point, would you recommend against running forget/prune and then un-merging (to go back to the original repo size)?

5

Thank you for your reply. index/ is 478MB.

6

Correct. Mergerfs will allow you to add another directory on another storage device as additional storage. Then you run prune on the mergerfs mountpoint. When done, there should be a mergerfs command to remove the extra directory you created before, and it will push all changes back into the external directory structure.

If by chance your external drive uses btrfs, you could instead btrfs dev add a new device to that filesystem temporarily, which will accomplish much the same thing as mergerfs would.

2 Likes
7

Best is if you can add more space as per previous suggestions. Worst case you can probably fix the index other ways. But for now let’s continue trying to do it in a way that solves the initial problem – there not being enough space – so that you can then carry on with normal procedures.

8

@cdhowie I’m on a mac and the drive uses APFS

thank you all, will try this week and report back.

1 Like
9

I am more familiar with rclone but mergerfs approach is conceptually the same.

In a nutshell (so probably you will need to read a bit of docs for details:)):

Create rclone union remote with full disk marked as nc (meaning do not create any new files here). Rclone config file needs a section like this:

[union_disk]
type = union
upstreams = /path/to/full_disk:nc /path/to/some_free_space

Thx to restic supporting rclone directly you can use union_disk rclone remote as your restic repo destination (avoiding another set of issues with mount - can be pita on macOS):

$ restic -r rclone:union_disk: forget... 

$ restic -r rclone:union_disk: prune

Make sure that /path/to/some_free_space has enough of free space. Use dry-run with prune to estimate what is needed and/or use --max-repack-size flag to play within what you have.

Copy all data from /path/to/some_free_space to /path/to/full_disk - merge both.

Done.

Give me a shout if you need any help.

3 Likes