`restic restore` speeds asymptotically dropping towards zero!

[ifedorenko]

@durval

But it seems beyond doubt that the blobs these files belong to are corrupted. And that would be on all snapshots since the first

This is expected. The first snapshot stored corrupted blobs (regardless of how they got corrupted) together with size, timestamp and other file metadata. All subsequent snapshots reference the same corrupted blobs for as long as the file size and timestamp remain the same.

@durval

What would be the best way to ‘repair’ them in the repo, as fast as possible, and losing as little as possible of those snapshots

All data in the repository is immutable and covered by sha256. You can’t change existing snapshots or anything else stored in the repository. But if you touch the corrupted files and run restic backup, that should create new snapshot with just the touched files and hopefully correct data.

[durval]

@ifedorenko

This is expected. The first snapshot stored corrupted blobs (regardless of how they got corrupted) together with size, timestamp and other file metadata. All subsequent snapshots reference the same corrupted blobs for as long as the file size and timestamp remain the same

This is exactly my thinking, thanks for confirming it. Seems I’m finally starting to wrap my head around restic’s workings

@ifedorenko

All data in the repository is immutable and covered by sha256. You can’t change existing snapshots or anything else stored in the repository. But if you touch the corrupted files and run restic backup , that should create new snapshot with just the touched files and hopefully correct data.

And then restic forgetting all the previous snapshots, to make sure the corrupted files can’t be restored by mistake, correct?

[durval]

@durval

Can you restore the same two files with restic 0.9.5 and maybe with 0.8.x (which was the last version before restore rework was merged, iirc). If both 0.9.5 and 0.8.x both produce the same corrupted files, I’d say something happened during backup.

Right away, captain!

So I did it with standard restic_0.9.5, with the exact same results: the two files came with the exact same corruption as the restic_out-of-order-restore-no-progress restic.

I tried with 0.8.3 (the last release in the 0.8.x line) and got an error:

  Fatal: parsing repository location failed: invalid backend
If the repo is in a local directory, you need to add a `local:` prefix

I think this means 0.8.x doesn’t support the rclone backend, correct?

Cheers,
– Durval.

[durval]

@durval

Anyway, let’s be paranoid: the next time I can schedule a shutdown on that server, I will open it up and visually check whether it really has ECC RAM.

Just confirmed it, and without opening the machine: installed dmidecode as indicated here, and ran it:

# dmidecode 
    Handle 0x0008, DMI type 17, 34 bytes
    Memory Device
        Array Handle: 0x0005
        Error Information Handle: No Error
        Total Width: 72 bits
        [...]
        Part Number: 0x4D33393342324737305148302D434D412020
   [the same repeats for the other 3 DIMMs]  

The “72 bits” should be enough to confirm it’s indeed ECC memory, but nonetheless I googled for 0x4D33393342324737305148302D434D412020 and found this:

Memory Module: DIMM1, 16 GB, DDR3 ECC, 1866 MHz, 0x80CE, 0x4D33393342324737305148302D434D412020

So I think it’s settled: this machine indeed has all-ECC RAM.

Cheers,
– Durval.

[durval]

@ifedorenko

Unfortunately, when I verified the restored files (using shasum) I detected that two of them had their contents corrupted

Can you try restoring the corrupted files again, both using restic 0.9.5 and the out-of-order branch? If corruption is reproducible, can you try restore with --verify parameter, which tells restic to read the files after restore and verify checksum? Thank you in advance.

OK, I just found and fixed the issue. I’m reporting this in a new thread, both to avoid this one going more OOT than it already has, and for the problem and solution to stand more prominently, please follow it there: [PSA] ATTENTION anyone not using a recent master (ie, > 0.9.5) version: restic will NOT detect changed files if mtime is reset by the application (eg: MS Excel and .xls/.xlsx files on macOS)

Cheers,
– Durval.