Hi there, I am testing restic for backing up a file of ~4G to Backblaze.
I was expecting a bit more speed from restic I have to say and I wonder I am doing anything wrong on my side.
These are the logs/details:
scan finished in 1.917s: 2 files, 4.379 GiB
modified /var/lib/dir/file.enc.md5, saved in 0.001s (44 B added)
modified /var/lib/dir/file.enc, saved in 841.057s (21.533 MiB added)
modified /var/lib/dir/, saved in 841.073s (0 B added, 194.604 KiB metadata)
modified /var/dir/, saved in 841.073s (0 B added, 369 B metadata)
modified /var/, saved in 841.074s (0 B added, 367 B metadata)
Files: 0 new, 2 changed, 0 unmodified
Dirs: 0 new, 3 changed, 0 unmodified
Data Blobs: 12 new
Tree Blobs: 4 new
Added to the repo: 21.725 MiB
I am not sure what the bottleneck is but I can definitely try things out in order to provide more info.
In any case, I got up and running very quickly so that’s a big plus!
Depends on the connection. I wouldn’t want to have that slow of a speed either. But I doubt the problem is caused by restic to be honest.
Which restic version are you using? I you try it with another backend, e.g. S3 or the REST backend, you might see different numbers. But it’s too early to say.
Please check which version of restic you are using. If you are not on version 0.11.0, please grab that version from https://github.com/restic/restic/releases/tag/v0.11.0 and try your tests again. The issue you link to is really old.
Can you possibly try using another backend? If you have a server somewhere you can just put up the REST server or even Minio for an S3 compatible backend.
Could it be that the 841s are the time that restic needs to
read the 4GB of the file
chunks this into blobs and hash each blob
for each blob: check if already existing and encrypt and save it if not
As only 22MB have been uploaded, lots of blobs were obviously identical, maybe big parts of the files being zeroes?
Now uploading to Backblaze is only done in 3. and if this would take 841s, there is a big problem…
Point 2. shouldn’t take much time as this is done in memory and recent CPUs are fast enough.
Actually, I would guess that 1. might take quite some time depending on the block device you are using… 4GB in 841s would be around 5 MB/s. What kind of device are you using? Could that be the bottleneck?
No slow connection. It is an internal device not a USB device.
Setting up a server/S3 requires some work. I will try my best but it seems the problem identified above is quite likely to be the one.
A quick thing I am going to try is to use the same setup for a “normal” dir, this time containing only relatively small files. This will help understand if the problem is in the chunking.
This is from my morning backup to wasabi. It is similar in upload size, but the total time was in seconds. I use version 0.10.0 and the rclone back end.
I don’t think the time is a restic issue.
-------------------------------
Starting backup of wasabi Wed Dec 2 02:00:01 AM AST 2020
-------------------------------
Files: 6 new, 635 changed, 100571 unmodified
Dirs: 0 new, 153 changed, 7440 unmodified
Added to the repo: 22.922 MiB
processed 101212 files, 360.937 GiB in 0:19
snapshot 2b166b2a saved
-------------------------------
Starting check of wasabi Wed Dec 2 02:00:22 AM AST 2020
-------------------------------
If it is helpful, I can offer some performance data from my setup, which includes local, AWS, and B2 components.
In this data set, the following operations were performed:
restic backup to a local drive (I use 3 different repos, one for each major filesystem)
restic --forget --prune a few snapshots in each of those 3 repos
aws s3 sync those repositories to AWS
restic backup of the same filesystems directly to a separate repository in Backblaze.
(I use the local+sync process for the main backup, because I like to have a local copy for fast restores and prunes, and I put it on AWS for the very inexpensive GDA storage. I do the Backblaze step because I am paranoid that my local restic repositories may get corrupted somehow, and that I will sync that corruption to the cloud, so I like to have a second repository)
Here are some performance results from Dec 05:
Repository 1: 17GiB total size, 25MiB added to repo on this run
Local backup: 2 minutes, 46 seconds
Sync to Amazon: 10 minutes, 41 seconds (includes many deletions)
B2 backup: 4 minutes, 2 seconds
Repository 2: 546GiB total size, 51KiB added to repo on this run
Local backup: 1 minutes, 11 seconds
Sync to Amazon: 4 minutes, 48 seconds (includes many deletions)
B2 backup: 3 minutes, 10 seconds
Repository 3: 51GiB total size, 97MiB added to repo on this run
Local backup: 54 seconds
Sync to Amazon: 7 minutes, 18 seconds (includes many deletions)
B2 backup: 5 minutes, 4 seconds
I don’t think these comparison are actually apples to apples. I suspect the most accurate diagnosis of the problem was what @alexweiss mentioned.
There might be some pathological case with my block file device that restic cannot diff/handle efficiently.
I since moved to another way to backup that particular data format, as I was evaluating restic only…
Reproducing should be easy, just create a file filled with zeros and try to upload it.
