Possible Backup Bug (Part 2) & Bit Rot Detection

Getting Help
1

Crap. It happened again. This is a continuation of this thread, however with a slightly new issue I thought warranted a new thread (plus the old one was pretty sprawling over time).

I thought it was the SAT SMART Driver I was using on my home and work machines, but now the only thing in common is that I’m using CalDigit docks (TS4 at home, TS3 at work). I’ll test that with several back-to-back full reads with the drive plugged directly into my MacBook and report back on the original thread.

THIS thread, however, is specifically about Restic’s “bit rot detection” (diff showing “M?” if there’s a modified file with the same timestamp in two different snapshots).

So let’s set the stage. I’m running Restic v0.18.0. MacOS Ventura plugged into a CalDigit TS4 at home, and MacOS Sequoia plugged into a CalDigit TS3 at work. All my filenames have CRC32 hashes in them, as an extra protection against bit rot. Both CRC32 hashes fail in snapshot da4846fa. Both da4846fa and 79545193 were ran with --force to ensure re-reading all data.

Here’s my snapshots:

ID        Time                 Host   Tags     Size
--------------------------------------------------------
6f2d6aab  2025-03-15 11:18:20  Media           1.144 TiB
3102cfca  2025-04-01 22:59:24  Media           1.191 TiB
da4846fa  2025-05-14 15:32:59  Media  corrupt  1.208 TiB
79545193  2025-05-14 16:49:56  Media           1.208 TiB
--------------------------------------------------------

There was another snapshots before da4846fa, with DIFFERENT corrupt files, but I removed that one. So yes, corrupt files two times in a row - until I plugged the drive directly into my MacBook.

Now here’s the purpose of my thread. When I do a restic diff 3102cfca da4846fa to compare the 2nd and 3rd snapshots I get:

M    /Volumes/Media/Photos & Videos/2011/2011-12-24 (Christmas Eve @ Becky's)/MVI_0487 [2149E9E5].MOV
M    /Volumes/Media/Photos & Videos/2022/2022-01-19/2022-01-19 20-31-28 (2) [8E1B79A8].mov

Note the lack of “M?”. I assumed corruption anyway because (a) I didn’t modify those files intentionally and (b) the CRC32 hash in the filename doesn’t match in snapshot da4846fa.

Here’s restic diff da4846fa 79545193, the 3rd and 4th snapshots:

M?   /Volumes/Media/Photos & Videos/2011/2011-12-24 (Christmas Eve @ Becky's)/MVI_0487 [2149E9E5].MOV
M?   /Volumes/Media/Photos & Videos/2022/2022-01-19/2022-01-19 20-31-28 (2) [8E1B79A8].mov

That has the M? this time.

What about restic diff 3102cfca 79545193 - comparing the 2nd and 4th snapshots?

That shows no difference between the two files. :exploding_head:

Here’s the date information for snapshots 2-4:

akrabu@akrabu-macbook-pro ~ % restic ls -l 3102cfca:"/Volumes/Media/Photos & Videos/2011/2011-12-24 (Christmas Eve @ Becky's)" | rg 2149E9E5 
-rwxrwxrwx   501    20 1304792252 2011-12-24 11:38:10 /MVI_0487 [2149E9E5].MOV
akrabu@akrabu-macbook-pro ~ % restic ls -l da4846fa:"/Volumes/Media/Photos & Videos/2011/2011-12-24 (Christmas Eve @ Becky's)" | rg 2149E9E5
-rwxrwxrwx   501    20 1304792252 2011-12-24 11:38:10 /MVI_0487 [2149E9E5].MOV
akrabu@akrabu-macbook-pro ~ % restic ls -l 79545193:"/Volumes/Media/Photos & Videos/2011/2011-12-24 (Christmas Eve @ Becky's)" | rg 2149E9E5
-rwxrwxrwx   501    20 1304792252 2011-12-24 11:38:10 /MVI_0487 [2149E9E5].MOV

So my question is… why does comparing 3102cfca (2nd) with da4846fa (3rd) show “M” and da4846fa (3rd) with 79545193 (4th) show “M?” and comparing 3102cfca (2nd) with 79545193 (4th) show NO modifications?

Was there perhaps a sub-second timestamp change? But if that was the case, (a) how can I check to make sure, and (b) why wouldn’t that change show on the diff between the 2nd and 4th snapshots?

Ps. What am I doing wrong with the code blocks to get random burnt orange / brown lines, and grey italic lines? lol

2

I can’t answer your question but the colors in your code blocks is because those are interpreted as markdown (class="hljs language-markdown") and the dashes and the lines above are highlighted as section headings (class="hljs-section"). And the gray lines are highlighted as “meta” in a language-apache.

You can add a language for code blocks and plain (or I guess anything unknown) prevents highlighting:

```plain
some text
---------
more text
```
2 Likes
3

Ha, thanks, that’s good to know!

Hmm looks like I’ve stumped everyone with the “diff” weirdness haha

I will say, in regards to my original issue - with my CalDigit dock out of the picture, I have had no further random corruption. Going to buy an Anker dock instead after I’ve done this for awhile without issues. So there’s that at least.

4

I have to say that this is a part of personal computing that I have never really worried about too much. But ever since using restic, I see cracks (albeit small ones) in the shiny polish. Thanks for sharing the case and infos!

5

What does restic cat tree 3102cfca:"/Volumes/Media/Photos & Videos/2011/2011-12-24 (Christmas Eve @ Becky's)" and so on for the other snapshots show? (Only the JSON object for MVI_0487 [2149E9E5].MOV is relevant. Or rather, only the diff between them).

6

I though about this a bit more and I think I found the reason: by default diff only compares file content but not metadata changes. The diff between snapshot 2 and 3 says that both the file metadata and content has changed. Between snapshot 3 and 4 the file metadata did not change, but the file content did. Based on the diff between snapshot 2 and 4, we know that the file content remained the same. Based on the steps above the file metadata must still be different. If we look at the options of the diff command, then there’s a tiny option called --metadata which also enables diffing metadata but which is disabled by default. That is, the expected metadata change won’t be shown.

tl;dr: restic diff --metadata 3102cfca 79545193

2 Likes