I’d want to make sure the data wasn’t my plaintext list of missile launch codes, but I will most likely be fine sharing both with you or other developers so long as they aren’t posted publicly.
Confirmed OK.
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat blob 73b786ec7e6c1ea7d0d1ae078cccec07248349b45d000f11def2f5f9babee72b | sha256sum
enter password for repository:
73b786ec7e6c1ea7d0d1ae078cccec07248349b45d000f11def2f5f9babee72b -
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat blob 090726a0a3fdec0aad46a727875b7ff3598f6a4dd74c4410b21085d02cf9378f | sha256sum
enter password for repository:
090726a0a3fdec0aad46a727875b7ff3598f6a4dd74c4410b21085d02cf9378f -
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat blob 2b9f1aeceea423735f1d2a31153888c65ea619d27cf8b57478c56038cbbafa8a | sha256sum
enter password for repository:
2b9f1aeceea423735f1d2a31153888c65ea619d27cf8b57478c56038cbbafa8a -
Also confirmed OK:
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat pack fe2e225c16ff29e0445b2953e08e6c3c65daacda27d89016469f9af5d25ae236 | sha256sum
enter password for repository:
fe2e225c16ff29e0445b2953e08e6c3c65daacda27d89016469f9af5d25ae236 -
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat pack 6e320160a550fe12222aaf2fecefa0e039181faeecac4000698340f4a6a062fc | sha256sum
enter password for repository:
6e320160a550fe12222aaf2fecefa0e039181faeecac4000698340f4a6a062fc -
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat pack 7ff4089107bf42674cc417221cf051b6c9ab72acf02a73eb34dd16468056f036 | sha256sum
enter password for repository:
7ff4089107bf42674cc417221cf051b6c9ab72acf02a73eb34dd16468056f036 -
$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name cat pack 51aa2bf43d75554d7f9bcfe8a19510d5da3b8c5be249410f076d76775b130e75 | sha256sum
enter password for repository:
51aa2bf43d75554d7f9bcfe8a19510d5da3b8c5be249410f076d76775b130e75 -
So, more verification that the data at rest is good and that the issue is transient.
I have considered if this could have anything to do with Intel security issues and the associated microcode updates. It’s an older CPU (Xeon E3-1220 v1). spectre-meltdown-checker reports it’s running the latest microcode, released in 2019, and that that microcode is not -known- to have stability issues.
For hardware stress testing, I ran linpack concurrently with mprime for 3 hours before switching to the below, no issues.
I set up 3 partial copies of the repository locally, did an add key, and ran 3 instances of the following fish shell loop with different directories and different pack IDs:
while test $status -eq 0
env RESTIC_PASSWORD=not-my-password DEBUG_LOG=$PWD/restic-debug.log ~/restic-debug-1999/restic -r $PWD --no-cache debug examine --extract-pack fe2e225c16ff29e0445b2953e08e6c3c65daacda27d89016469f9af5d25ae236
end
Looking through the code in the 1999 branch, it doesn’t look like debug examine will exit with nonzero status in the case of my issue. But all the .bin files written out have the correct- prefix, and grep -i 'error' restic-local-repo*/restic-debug.log | grep -v 'repository/key.go:153' yields nothing. The exclusion there is because the primary key wasn’t changed and it throws an error when it attempts it.
3 instances of that loop (again, different local repos, different pack IDs) running for ~18 hours has failed to reproduce the issue. Not yet ready to say that this path won’t produce the issue, but getting there. I note that it spends most of its time examining the indexes. Will deleting snapshots reduce the number of indexes?
I’m using --no-cache in the loop to avoid any potential issues with the multiple processes running with a shared cache. I don’t see that affecting this issue, but it’s conceivable. Are we sure a disk issue couldn’t be at fault here?
Thanks! I have started this running.
I’d like to offline the machine to run memtest86 for 24 hours but that is difficult. I am however running 
Not what I was expecting! We’ll track it down though, whether software or hardware.
