Failed: nonce is invalid

1 
restic -r repo prune
enter password for repository:
repository 10274fb1 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 999 snapshots
[0:00] 0.00%  0 / 999 snapshots
decrypting blob 399b78aae65000bcdfb0d25ab8aad978e6ce9b04c2db347a4692219a7e2e3932 failed: nonce is invalid
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
        /restic/internal/repository/repository.go:204
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        /restic/internal/repository/repository.go:728
github.com/restic/restic/internal/restic.loadTreeWorker
        /restic/internal/restic/tree_stream.go:37
github.com/restic/restic/internal/restic.StreamTrees.func1
        /restic/internal/restic/tree_stream.go:164
golang.org/x/sync/errgroup.(*Group).Go.func1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371

What does it mean and what should I do?

2

What does restic check --read-data report? Any idea why the repository got damaged? Did anything unexpected happen? Do you run prune or check regularly?

Which restic version are you using?

3

I am using 0.12.1, now I updated to the latest, but of course nothing has changed

The repository is about 1TB, on HDD, so I can’t just do a restic check --read-data, that would take a long time and slow down the backup server. Also, if I understand correctly, this won’t help me, it will just point out the error at most. Or am I wrong and this will help us somehow?

I’m using rest-server, maybe some upload failed, that’s a guess. I do prune regularly, check --read-data rarely.

The main question is, what do I do now, how do I make prune? Or do I have to delete the whole repo?

5 
restic -r repo/ check --read-data
using temporary cache in /tmp/restic-check-cache-2711023182
enter password for repository:
repository 10274fb1 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-2711023182
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree 3eb176cf:9 snapshots...
  decrypting blob 3eb176cf5ecec20ebce3fa33d59e5988fb5ad1298161ddf617fac7c2500aee9c failed: nonce is invalid
[0:48] 100.00%  1009 / 1009 snapshots...
read all data
Pack ID does not match, want ba2bdce3, got bfc205e4
Pack ID does not match, want b1819685, got 475527c2
Pack ID does not match, want b4abc274, got a7eea265
Pack ID does not match, want 46c76693, got 560804d0
Pack ID does not match, want f0298183, got eee07425
Pack ID does not match, want aa687bcb, got 6e1c7e1b
Pack ID does not match, want 597ec488, got e8cda98f
Pack ID does not match, want bce342fd, got 0ab5a483
Pack ID does not match, want 10798a7a, got bd2d4173
Pack ID does not match, want e57fe1a2, got 42c5d854
Pack ID does not match, want 26516bc2, got b92a38c4
Pack ID does not match, want f297e279, got 4bb64d05
Pack ID does not match, want c897b4d9, got 722540d2
Pack ID does not match, want 6696c58a, got 8e6f2b9a
Pack ID does not match, want 30fa7362, got 8dfa5a5b
Pack ID does not match, want b21c9180, got 63bce1e2
Pack ID does not match, want cfca61b0, got 430864a5
Pack ID does not match, want 6b3ba370, got c8d155b2
[15:01:17] 100.00%  195139 / 195139 packs...
Fatal: repository contains errors

What do I do now?

6

Which rest-server version are you using? Make sure to use the latest version as it fixes a corner case that can cause data loss in some cases (your current problem seems to be unrelated tough) and it verifies uploaded files which might have prevented the Pack ID does not match problem (depending on where it originates from).

An interrupted upload alone will not cause rest-server to leave behind damaged files. For that rest-server either has to crash or the whole system has to fail or have some HDD problems.

Did you notice any errors reported in prior restic runs to the prune failure or did something unexpected happen? Is this the first failing prune run? Do you have logs of the backup and prune run before the failing one? Did restic report any errors there?

To repair a repository we first need to know what is damaged. The main problem here are likely the Pack ID does not match errors and not the nonce is invalid error, the latter is likely just a result of the former errors.

Judging from the error messages, the damaged pack files are not truncated but still in some way damaged. The nonce-error suggests that some parts of these files were zeroed out.

I’d first like to try to find out how the damage occurred as it otherwise can show up again.

Let’s start by first looking up the pack file in which the damaged tree blob is stored:

Please look up the packfile containing the tree blob using restic find --show-pack-id --tree 3eb176cf5ecec20ebce3fa33d59e5988fb5ad1298161ddf617fac7c2500aee9c.

For the pack id mismatch errors, restic detected that e.g. for ba2bdce3 the sha256 of the file at data/ba/ba2bdce3[...] did not match its filename. Please use shasum -a256 data/ba/ba2bdce3[...] for the reported pack id mismatches to check that the filename and the file hash indeed do not match. It would also be very interesting to learn the creation and modification time of these files.

Please run memtest at the backup client and the rest-server to verify that there are no memory problems on the host, which could also cause the pack id mismatches. Then also check the health of the HDD used to store the repository, e.g. defect sectors could also cause the reported errors.

To repair the repository you’ll have to follow the steps at Recover from broken pack file · Issue #828 · restic/restic · GitHub . As there are quite a few damaged pack files, it’s probably a good idea to follow the steps from Route 1, as otherwise quite a few snapshots might end up missing data, which in turn will require them to be deleted or continues to prevents prune from working.

2 Likes