Backing up restic backups

Getting Help
1

Hi,

I am running linux (Kubuntu 19.10)
I just started using restic as a better backup solution than what I have mostly done (by copying entire home directory manually to external hdd.)

Using restic will ensure that all data, at all times is backed up. I have about 180,000 files (707GB) of data in my home directory which when backed up with restic is 137,000 files (684GB).

I have never backed up that many fails at one copy operating before, i’ve only done it over a course of a week or so by manually copying certain folders under my home directory until i got everything over.

Now I always try to make backups of backups at certain times to ensure if one backup drive malfunctions, I got another. This seems based on its copy speed like it would be copying much slower than when restic originally performed backup.

When I did a backup with restic command was: ./restic -r path/to/repo --verbose 2 backup ~/
After backup was done (and ran restic check), i plugged in another external hdd to perform the copy of this newly made repo to it. I did that with rsync using command: rsync --delete --delete-after -a --info=progress2 path/to/source path/to/destination (I know the --delete flags wont matter during it’s first copy, but will be used for future operations).

The speeds are anywhere from 25-40mb/s when it first started. After about 20 minutes or so, it went down to average of 15mb/s and hardly goes up to 16mb/s so it stays around this average speed unless restarted. Now I understand that copy 100,000 files of smaller sizes can tend to take longer than a few big sized files but I dont understand how this rsync operating will take longer than the original backup when the original backup performs more operatings all together such as encryption & compression.

In case it matters, all my external hard drives are fully encrypted with veracrypt. My internal disk encrypted with luks.

It doesn’t seem like backup a restic repo of 137,000 files should take longer than backing up 180,000 files not part of restic repo.

I did boot into a live linux disc and tried the copy using rsync their to see if anything on my computer may be conflicting with it but it was about same speeds. I tried different usb ports, not using a hub. Even tried different hard drives, and cables so it doesn’t seem to be a hardware related problems. The drives themselves are fairly new. I am using one thats usb3.1 & another thats usb 3.0 both WD brands.

2

You mentioned both drives are external. If both drives are plugged into the same USB controller, this could make the controller the bottleneck. Note that only USB 3.x in SuperSpeed mode is full-duplex. This means that all devices, ports, and USB controllers involved must support USB 3.x SuperSpeed to get full-duplex throughput; otherwise, half-duplex is used, causing contention between simultaneous reads and writes.

15MB/s = 120Mbit/s. If both are on the same USB controller, this is 240Mbit/s of throughput. USB 2 can sustain 480Mbit/s in perfect (theoretical) conditions. 240 being exactly half of 480 is suspicious to me. It’s entirely possible your USB controller only supports 240Mbit/s total throughput, which would perfectly explain the transfer being capped at 15MB/s.

While restic was running the initial backup, unless you were backing up other external drives as well, restic could use the full available throughput of the USB controller (based on these numbers, I would speculate it would max out at 30MB/s).

To rule this out, you could attach both drives to different computers and try the copy over a 1Gbps-capable wired network. If your throughput magically doubles, we can reasonably conclude that the USB controller is the bottleneck. (Note that some devices have their onboard Ethernet controller actually attached internally using USB. This is the reason why the 1Gbps NIC on the Raspberry Pi cannot even achieve 400Mbps. If you perform this test, make sure the NICs are attached to the PCI bus, otherwise the drive and NIC could wind up splitting available USB bandwidth.)

1 Like
3

@dhammel Why are you trying to do it this way instead of just making two separate backups? If your main backup becomes corrupted, your second one will be too. Better to make two separate backups instead.

4

The main reason is because I want the backups residing my external drives to be exactly the same. Creating new restic backups will cause things like the encryption key to be different which means all the data, will be different (In terms of its encrypted form). If the main backup became corrupted, it will only corrupt the 2nd backup if it was copied over, it wont magically become corrupted. I run constant test on my main backup to ensure it is not corrupted plus I backup in other ways like clonezilla (to get a actual clone of my hard drive). I also want to be sure that the data is the same in the backups. Due to the time it takes to backup all my data, I dont have the time to manually perform a backup with restic on each backup drive right in a row, and in the course of times it takes to do this certain data, mostly in the hidden directories such as browser data or program settings (and even elsewhere if i’ve added/changed data) will change which causes some of that data to be different on one drive than on the other. A lot of this data I manage myself, and that would end up confusing me (not know right off which drive may contain said data).

5

I don’t have an availible computer to test this on. However, I managed to get it at a reasonable speed by using a new drive I got thats equipped with USB Type C 3.1, I have one usb type c port on my computer with 1 usb 3.0 and 2 2.0, so by using one backup drive plugged into type c and the other in usb 3.0, the speeds are average 70mb/s to 80mb/s. If I backup an individule file, such as one of my linux containers thats 128GB, it has an average speed of 120mb/s. The speed is quite higher for an individual file, but i did expect that.