Upfront: I made this post to highlight that the error reported does not easily lead me to what is bad. It seems to use shorthand for blobs and packs, the output should be improved so it is clear what is what and how to figure out what is bad. The program is great and better usability will help less experienced people get some great backups.
This is a record of recovering from a nonce is invalid error on blobs combined with the difficulty determining what the error meant. I hope it helps the great devs see how someone who’s not that well versed with Restic worked through what it was saying; I am sure it can be more clear for someone in the future!
The crux of the issue is that Fatal: decrypting blob <data/ad9f3cf7> from ddb5cca9 failed: nonce is invalid did not match other examples I found that seemed to just name the whole blob. These seem to name shorthand parts of the blob and pack ID, and it’s not clear that “from” means “from pack ID ddb5cca9”, nor how to find the full pack ID name to work with.
Restic version:
restic 0.18.1 compiled with go1.25.1 on linux/amd64
Command that errored:
restic -r /mnt/remote/repo --cache-dir /mnt/restic/cache forget --prune --max-unused 5G --verbose --tag systemd.timer --group-by "paths,tags" -o local.connections=10 -o rclone.connections=10 --cleanup-cache --password-file /opt/restic/pass.pass --limit-download 4000 --limit-upload 1000 --pack-size 128 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2
Error (backend is local, remote repo mounted locally via Samba share):
Nov 29 07:05:48 Nork bash[2832634]: deleting unreferenced packs
Nov 29 07:05:49 Nork bash[2832634]: [0:00] 100.00% 5 / 5 files deleted
Nov 29 07:05:49 Nork bash[2832634]: repacking packs
Nov 29 23:49:01 Nork bash[2832634]: [16:43:11] 63.46% 990 / 1560 packs repacked
Nov 29 23:49:02 Nork bash[2832634]: Fatal: decrypting blob <data/ad9f3cf7> from ddb5cca9 failed: nonce is invalid
Fix:
restic -r /mnt/remote/repo --cache-dir /mnt/restic/cache repair packs ddb5cca9d42926050459c89a1db69e0961261608da9d01ad046549df89c674fc --verbose -o local.connections=10 -o rclone.connections=10 --cleanup-cache --password-file /opt/restic/pass.pass --limit-download 4000 --limit-upload 1000 --pack-size 128
It failed on many blobs in the pack and then deleted the pack.
[0:04] 100.00% 433 / 433 index files loaded
saving backup copies of pack files to current folder
salvaging intact data from specified pack files
failed to load blob 9c308c025b12d675b43d5d3f1f9288d10501bdd1bdbdd15cf0a7138f4ad46c3d: decrypting blob <data/9c308c02> from ddb5cca9 failed: ciphertext verification failed
failed to load blob ad9f3cf75d4da0cf1cc092f9952448eb18f38adb12044eb189e00d2f4de39c7d: decrypting blob <data/ad9f3cf7> from ddb5cca9 failed: nonce is invalid
failed to load blob f44b16dca49d87e4cd5fe455b279a87badedb2ce9c9d9764f2e407ce69de7423: decrypting blob <data/f44b16dc> from ddb5cca9 failed: nonce is invalid
...
<cut>
...
failed to load blob ef51560cae5a0751e935070c86f760ba6672c6db2df7a88f9c6f93f19a27de16: decrypting blob <data/ef51560c> from ddb5cca9 failed: nonce is invalid
failed to load blob 259a134ef3973467186061e64b97fe6f4ecb95cb3be455a584997e427f8b1c3e: decrypting blob <data/259a134e> from ddb5cca9 failed: nonce is invalid
[5:55] 100.00% 1 / 1 pack files
rebuilding index
[0:13] 100.00% 434 / 434 indexes processed
[2:21] 100.00% 379 / 379 old indexes deleted
removing salvaged pack files
[0:00] 100.00% 1 / 1 files deleted
I then did the snapshot repair as recommended after that command
restic -r /mnt/remote/repo --cache-dir /mnt/restic/cache repair snapshots --forget --verbose -o local.connections=10 -o rclone.connections=10 --cleanup-cache --password-file /opt/restic/pass.pass --limit-download 4000 --limit-upload 1000 --pack-size 128
The two files that were bad are:
/mnt/time4timapple/MacBook.sparsebundle/bands/a9e
/mnt/time4timapple/MacBook.sparsebundle/bands/aaa
Which are bits of the mac program Time Machine’s sparse bundle (which it uses to hold its backups in the most fragile of states) for an old macbook.
It deleted them from all of the snapshots they were involved in.
I guess I succeeded?
Commentary:
From searching I was not able to find what data/ad9f3cf7 and ddb5cca9 meant, any previous mentions of nonce issues just list the blob in full.
I was able to determine that theddb5cca9 was the pack ID by going into the restic repo raw in the data folder and going into the dd folder and then finding the pack ID that began with ddb5cca9 . That was the naughty pack, but it was not clear to me at that time that theddb5cca9 was the beginning of the pack ID. No permutations of find that I tried worked; I didn’t try too hard, just slapped an * at the end of data/ad9f3cf7 and ddb5cca9and didn’t try to divine more from the docs on matching rules or anything. find just reported it couldn’t find them. Also find --blob didn’t work on the whole blob string ddb5cca9d42926050459c89a1db69e0961261608da9d01ad046549df89c674fc; likely because that’s actually the pack ID. Probably should have used --pack, but didn’t know till I just tried the long sequence as a pack ID in the repair packs command that it was a pack ID
In hindsight, I should have done post-repair pack is to do backup before the repair snapshot. Per the recovery docs Troubleshooting — restic 0.18.1-dev documentation that may have prevented the two files from being removed from the snapshots if they could be re-uploaded.
Lastly, I am not sure how this issue occurred. The destination is on a BTRFS mirror that does scrubbing, and no scrub errors were there. Between me and the destination are many hurdles (wireguard VPN, Samba mount, occasional power losses on both sides, etc.). I guess one of them overcame Restic’s defenses and managed to get a bad write down.
And finally, I lack the bandwidth to do a full check. It runs a check on 2 GB every day, but that is it. And because of that, I shall not be resting easily. Oh well!
even with the small spot check I can manage