Repository corruption after interrupted SFTP backup and non-deterministic repository checks?

Getting Help
1

I have just been starting with Restic and I already have issues. My goal has been to backup ca. 5 TB from my personal machine to a Hetzner Storage Box. I tried this via SFTP with all the options for better resilience. Nevertheless, even having the best ISP that I can get at my place, I usually have one or two short network outages per day. This is often not noticeable with day-to-day usage, but it did often break the SFTP connection, which would have needed to run over 9 days for the initial backup (less on resumed backups).

I didn’t think this would be a problem, because Restic is supposed to be able to resume backups, with most of the already transferred data not needing to be transferred again.

Nevertheless, before the first snapshot was finished, I wanted to confirm that the repository is fine. So I decided to run restic check --read-data. Since this command needs to run without interruptions, and a few TB of data had already been transferred, I used an Rclone-over-WebDAV backend instead of SFTP. Surprisingly, it reported a damaged repository (without any details). Since I had forgotten to save its output to include in this question, I ran it again. Even more surprisingly, now it reported no errors at all.

I had read the docs, but I was still puzzled, so I tried to find answers using ChatGPT. But I suspect there were many hallucinations.

  • It told me that while Restic itself could handle interrupted backups, this would require certain file system semantics which are not guaranteed by SFTP - in particular, atomic writes. Thus, if I want to make sure that backup interruptions will not corrupt a repository, I would need to switch to S3-compatible storage.
  • It also was not consistent about using an Rclone back end or not. On one hand, it suggested it, because it would make the backup more resilient, but on the other hand, it warned that this could also mask errors that would lead to corruption.
  • It also was not consistent about whether WebDAV has an edge over SFTP or not.

I have many questions about this - the most important ones being:

  • Is it true that interrupted backups are expected to sometimes corrupt the repository, unless the backup is a local filesystem or an S3-compatible back end?

  • For unstable connections, is Rclone (over SFTP or WebDAV) recommended or discouraged?

  • Why would restic check --read-data be non-deterministic?

Windows 11
restic 0.18.1 compiled with go1.25.1 on windows/amd64

2

Let me try to reply to you with my somewhat limited knowledge of restic internals.

Yes, you are right. It will indeed resume backups, and you can both cancel or get interruptions in your backup process – restic will either retry if still running the backup command, or off where it needs to. Note that when there’s no snapshot created yet, restic will need to re-scan the files to know what to back up, but it will not transmit already stored data blobs.

I don’t recall ever seeing restic state that the repository is damaged without saying in what way it is damaged.

I wouldn’t say that. It is known that if the underlying storage and/or transport that is responsible for giving restic the green light after e.g. a write hasn’t actually completed that write, then there’s not much restic can do about it. Regardless, restic takes a lot of precaution and applies a lot of measures to avoid corruption, including measures directly related to corner-cases with certain backends.

So while it isn’t expected that you run into problems, it is of course possible the more stuff you have going on between your restic client and the actual repository storage. Certain backends are more prone to problems than others, e.g. the B2 backend is not something we recommend, instead it is better to use the S3 backend with B2’s S3 compatible API.

When there are problems, there can be quite a bunch of different causes. It’s for example been seen that some storage tells restic that a file has been written, but upon reading that file it doesn’t yet exist. I.e. a race condition. Other bugs or problems on the storage or even network or API ends of things have been seen as well.

I can’t really say, but I personally wouldn’t think so. Maybe someone else can correct me if I’m wrong, but I don’t see what adding another layer to the mix for a backend such as SFTP, which restic supports natively, would be good for unless there’s some specific feature you need in it. For WebDAV, probably a good idea as I don’t think restic supports WebDAV natively.

I would say that restic check itself is deterministic. If the supporting/underlying hardware, software, networking and such does what it should do and hence returns the same data every time restic asks for it, you’d get the same deterministic results. At least that’s what I’d expect.

If you don’t, then there’s a problem somewhere, be it hardware, software, networking issues or race conditions.

We’d really need to see the output of any unsuccessful commands (and the commands themselves) to be much wiser. Again, I’m quite surprised if restic told you something is corrupt without giving you any information about what is corrupt. To me that sounds like you are wrapping restic or something, and missing out on some output. I just haven’t seen that lack of output before.

Is there any reason you didn’t just run the check over the SFTP backend just like you did with the backup run? Maybe I’m missing something, but I’d use the same backend for both backing up and checking.

Also, note the --read-data-subset option to the check command. It allows you to read only a part of the repository in a controlled way.

Generally speaking, without knowing more and considering you say that you did get a successful check (which I’d trust – I don’t see restic telling you that the repository is fine if it isn’t), I’d simply go ahead and continue the backup runs over and over until you got your data set backed up. Then I’d run a check using the same backend.

Finally, ignore whatever ChatGPT is telling you, for now :slight_smile:

3

SFTP with only restic to Hetzner works perfectly fine here in two cases I’m the admin at. One thing that has come up in this forum a few times is that the hardware of the machine being backed up has problems (example) and it only comes up because of restic’s checksums. It happened to me personally once too and it’s scary because we don’t usually notice on consumer hardware and file systems.

Personal note: I find working with large data sets in the cloud always somewhat challenging. If you can, make a local backup and then periodically rsync that to the cloud (3-2-1 backup). Then regularily check the cloud backup with --read-data-subset as @rawtaz recommended. In my experience it’s much faster to do so from a Hetzner VM but my DSL lines are usually extremely slow.

Oh yes and forget chatty… he’s full of b/s.

4

Sorry it took me so long to get back here. Thank you for your replies!

As I understand it, adding RClone as an intermediate layer between Restic and SFTP (or WebDAV) allows running a command that would otherwise fail because of an interrupted SFTP connection. Instead of a single interrupted connection that leads to a failed command, Restic would see a stable “virtual” connection that uses multiple consecutive interrupted SFTP connections with waiting periods in between. I.e., Rclone would used here for its retry features.

In case of a long-running initial backup, rescanning already backed-up files would then not be necessary again and again, as opposed to when running the backup multiple times until it succeeds.

Yes. Over SFTP, it would probably never complete given the already large amount of data in my repository and my network connection quality.

I ran that before check --read-data, but no issues were discovered.

Did it happen with a Hetzner Storage Box? The docs say that they use a RAID with redundancy and that “checksums for the individual data blocks are used to detect and correct bit errors”. Thus, bit rot should be very unlikely. And yes, on consumer hardware without a file system that is specifically designed to prevent bit rot (e.g. ZFS), this can happen indeed.

It doesn’t fit my current setup. I want to frequently backup my laptop no matter whether I am at home or not. I do make less frequent local backups, but the main backup is supposed to be online only.

But anyway: Would that not just move any issues related to consistently uploading large amounts of data from Restic to rysnc? I guess if a backend or network protocol can fail Restic, it could also fail rsync.

You mean running restic check --read-data-subset?

5

Okay, yes then I guess there’s little choice.

Anything with --read-data, yes.

Hetzner has enterprise hardware, yes. But do you? Take a few minutes and scan the old threads on the forum and you will find a few occasions where data was already corrupt when it arrived in the restic repo.