True, but even that should not be relevant. I am under the assumption that Restic ensures that a backup has fully completed until it marks that as a successful snapshot. In database lingo that’s an atomic transaction. In other words, if I interrupt a backup mid way through, the previous backups on the medium should be unaffected, and the half-completed backup should be discarded.
If there is disk corruption at hardware level that’s another issue, but then even journaling may not help.
But I’m happy to be proven wrong, ideally by means of an example.
Interesting. Indeed, I don’t need that scanning. May try that when I give it another shot. For now I’m cautious, as Restic acts as some kind of stress test, and no stress test before has actually managed to crash my machine.
I could be barking up the wrong tree here, but I wonder if this is an error about your internal M.2 drive, rather than the external USB one?
My reasoning for thinking this is I would not expect a drive connected over USB to show up as an M.2 drive in the BIOS error message, and I also wouldn’t expect a USB drive dying to cause the laptop to crash so completely.
A quick internet search for “2100: Detection error on Storage Device (M.2)” did turn up a bunch of people having problems with their internal M.2 drives on various lenovo systems with this exact message. I couldn’t see anyone having this error with an externally attached storage device, but it might be I just didn’t look for long enough.
The above is all just a theory, but it could be worth giving your internal M2 a once over (e.g. check the smart stats, maybe run a smart self-test) just in case that is the problem, rather than the external drive.
Oh as an aside, if you shared some numbers about how much data you were trying to back up, and how long it took (you mentioned many hours longer than macrium, but we don’t know how long your macrium backups take ), one of the forum regulars might be able to provide some guidance on whether the performance you were seeing was normal or not.
What internal SSD is that anyway? Brand, model, etc.
Fact is that I and a huge number of the restic community is backing up their systems having internal SSDs or similar storage devices, without any issues. Even if restic is so fast that it stresses your hardware, the hardware the rest of us has does not find this to be problematic. So, the conclusion is that your hardware is most likely not up to snuff
It could also be that it is not a bad model per se, but that it is partially defective in some way, and this starts to show when it is stressed a bit. This happens every now and then with restic, people discovering that their hardware has issues, where they just didn’t notice it earlier. But you already know that since you mentioned it yourself earlier
The answer is actually more complex. The backup command by default reads up to two files in parallel (--read-concurrency). After chunking, the file parts are deduplicated, compressed and encrypted. That step already defaults to using all CPU cores in parallel. And finally, completed pack files are written with a default parallelism of 2 for the local backend. To actually achieve a speed up you’ll likely have to increase both read-concurrency and the backend concurrency.
This relies on the guarantees offered by the underlying filesystem. exFAT does not support journaling and thus is vulnerable to data corruption if the system crashes at the wrong point in time. Restic relies on the filesystem to guarantee reliable file writes and atomic renames, but exFAT can’t offer that guarantee.
Plugging those numbers into a calculator, I get approximate transfer speeds of ~120MB/s for the Macrium backup, and ~40MB/s for the restic backup.
From your description, neither of those should be close to saturating the connection to the remote storage. It could be CPU performance is the limiting factor here; I’m not sure what Macrium does, but restic is going to have to read, chunk, and upload, every file in the dataset the first time it performs a backup, which does take a non-trivial amount of time.
To provide some numbers for comparison, I backed up some data from an internal NVMe SSD, to a new restic repository on an internal SATA SSD, and these were the results:
processed 44152 files, 6.943 GiB in 0:30
Which works out to ~230MB/s.
To give an illustration of why the first backup performance isn’t actually that critical though, here’s a subsequent backup of the same dataset:
processed 44152 files, 6.943 GiB in 0:01
Obviously that’s a small dataset, with no changes to files between the backups, but you get the idea.
Looking at your screenshot above, I think you have real-time protection enabled and you are not able to disable it due to organisational settings. I add an exception for the restic binary so that files it accesses are not scanned, which can significantly speed things up.
I suggest you do a good scan of both of your disks to check if any of them have errors.
What makes you think so? In the screenshot, it says that real-time protection is off. In fact, I disabled it long ago, right after installing Win 11 Pro. To do so permanently was non-trivial. There was an article that I followed: How to Disable Windows Defender in Windows 10/11
But just in case, I added an exclusion. See the other screenshot from above:
Defender shouldn’t be the issue.
What likely happens is that the system overheats, and then the internal SSD shuts down. Note that I needed a cold reboot to get the SSD up and running again.
Quite possible! That machine is old, a 2018 ThinkPad X1 Tablet 3rd Gen with an Intel i7 and 16GiB of RAM (20KJ-001JMH). I also use Restic on a server, which is considerably more powerful. There, Restic is really fast. But that’s a Linux system. Anyhow, thanks for doing the math!
I think I misread the screenshot, it looks like defender is on. Do you have anything else running that’s similar?
A computer that overheats and shuts down probably needs maintenance. Is it dusty inside? Does the CPU thermal paste need to be reapplied? Does it simply have inadequate cooling? m.2 SSDs for example overheat very easily without a heatsink and usually a fan near it, but older 2.5" SSDs usually have enough surface area they’re ok.
No. I make sure that there is no real-time virus check running, exactly as it slows down the system, not just Restic. I prefer to check individual files manually, if needed.
I don’t recall the SSD ever overheating and cutting out before. Of course, it should not happen, even if Restic stresses the system a lot. This SSD does not have a heatsink, just some metallic tape on top of it.
I don’t recall the SSD ever overheating and cutting out before. Of course, it should not happen, even if Restic stresses the system a lot. This SSD does not have a heatsink, just some metallic tape on top of it.
There are a number of reports online about your drive having overheating issues when used without a heatsink, eg
Obviously an initial Restic backup is doing a sustained read for a long time, so this will expose any overheating issues in a way that regular use may not; that could be what you’re seeing here.
Thanks, I wasn’t aware of that. Fortunately, Macrium Reflect did it’s job doing a full-disk backup with miminum resource usage and fairly quickkly, allowing me to work comfortably at the same time. For various reasons, see above, I would like to eventually substitute it with Restic.
I had an m.2 SSD completely fail due to heat. It was in an enclosure with a Raspberry Pi, it had no heatsink or cooling. It lasted about 6 months. I had to replace the enclosure with one that has more cooling and added a heatsink, since then the new m.2 has been fine.
I think some temperature monitoring may provide more information to help diagnose the problem. Restic may just be stressing the hardware more than it’s used to.
), one of the forum regulars might be able to provide some guidance on whether the performance you were seeing was normal or not.