Restic upload speed to Backblaze is not great

Hi there, I am testing restic for backing up a file of ~4G to Backblaze.

I was expecting a bit more speed from restic I have to say and I wonder I am doing anything wrong on my side.

These are the logs/details:

scan finished in 1.917s: 2 files, 4.379 GiB
modified  /var/lib/dir/file.enc.md5, saved in 0.001s (44 B added)
modified  /var/lib/dir/file.enc, saved in 841.057s (21.533 MiB added)
modified  /var/lib/dir/, saved in 841.073s (0 B added, 194.604 KiB metadata)
modified  /var/dir/, saved in 841.073s (0 B added, 369 B metadata)
modified  /var/, saved in 841.074s (0 B added, 367 B metadata)

Files:           0 new,     2 changed,     0 unmodified
Dirs:            0 new,     3 changed,     0 unmodified
Data Blobs:     12 new
Tree Blobs:      4 new
Added to the repo: 21.725 MiB

I am not sure what the bottleneck is but I can definitely try things out in order to provide more info.

In any case, I got up and running very quickly so that’s a big plus!

Thanks for this project!

1 Like

Welcome!

Can you explain in a bit more detail what you think is unexpectedly slow in what you’re seeing?

The scan of 4 GB took less than two seconds, seems fine to me.

Then it uploaded 22 MB in how long?

It of course depends on your connection speed, connection latency and the performance of Backblaze.

Please be more specific!

Hi @rawtaz thanks for your answer.

I think I am (of course) happy with the 2s parsing of the file :smiley:

My connection is able to hold Zoom meetings so I am not sure the problem is there.

Backblaze also works like a charm with the current backup tool I am using while evaluating restic.

I will definitely try again but I would be very surprised if the problem is the speed 22MB/840s ~ 26 KB/s

Depends on the connection. I wouldn’t want to have that slow of a speed either. But I doubt the problem is caused by restic to be honest.

Which restic version are you using? I you try it with another backend, e.g. S3 or the REST backend, you might see different numbers. But it’s too early to say.

I found out that there definitely seem to be some problem and it was already reported here:

I guess I should have checked the issues before opening this issue.

In any case thank you again for the prompt replies!

Please check which version of restic you are using. If you are not on version 0.11.0, please grab that version from https://github.com/restic/restic/releases/tag/v0.11.0 and try your tests again. The issue you link to is really old.

Yeah I have got the latest and the issue is still open it seems.

$ restic version
restic 0.11.0 compiled with go1.15.3 on linux/amd64

Darn, sorry to hear that :wink:

Can you possibly try using another backend? If you have a server somewhere you can just put up the REST server or even Minio for an S3 compatible backend.

1 Like

I can but my main backup target is Backblaze :man_shrugging:

It’s for isolation purposes to see if you have the same type of symptoms with other backends or if it’s a matter with B2.

Could it be that the 841s are the time that restic needs to

  1. read the 4GB of the file
  2. chunks this into blobs and hash each blob
  3. for each blob: check if already existing and encrypt and save it if not

As only 22MB have been uploaded, lots of blobs were obviously identical, maybe big parts of the files being zeroes?

Now uploading to Backblaze is only done in 3. and if this would take 841s, there is a big problem…

Point 2. shouldn’t take much time as this is done in memory and recent CPUs are fast enough.

Actually, I would guess that 1. might take quite some time depending on the block device you are using… 4GB in 841s would be around 5 MB/s. What kind of device are you using? Could that be the bottleneck?

@alexweiss thank you for the break down.

My files are coming from an encrypted ext4 partition on a SanDisk X400 SSD.

Connected how?

Regardless, are you going to follow up on our suggestions to isolate this problem?

This might be true, cause the actual data is ~ 2.5G. The rest is “free” space.

No slow connection. It is an internal device not a USB device.

Setting up a server/S3 requires some work. I will try my best but it seems the problem identified above is quite likely to be the one.

A quick thing I am going to try is to use the same setup for a “normal” dir, this time containing only relatively small files. This will help understand if the problem is in the chunking.

1 Like

This is from my morning backup to wasabi. It is similar in upload size, but the total time was in seconds. I use version 0.10.0 and the rclone back end.

I don’t think the time is a restic issue.

-------------------------------
Starting backup of wasabi Wed Dec  2 02:00:01 AM AST 2020
-------------------------------

Files:           6 new,   635 changed, 100571 unmodified
Dirs:            0 new,   153 changed,  7440 unmodified
Added to the repo: 22.922 MiB

processed 101212 files, 360.937 GiB in 0:19
snapshot 2b166b2a saved
-------------------------------
Starting check of wasabi Wed Dec  2 02:00:22 AM AST 2020
-------------------------------

Well, not really the same cause I am not processing many files but one giant binary blob.

If it is helpful, I can offer some performance data from my setup, which includes local, AWS, and B2 components.

In this data set, the following operations were performed:

  1. restic backup to a local drive (I use 3 different repos, one for each major filesystem)
  2. restic --forget --prune a few snapshots in each of those 3 repos
  3. aws s3 sync those repositories to AWS
  4. restic backup of the same filesystems directly to a separate repository in Backblaze.

(I use the local+sync process for the main backup, because I like to have a local copy for fast restores and prunes, and I put it on AWS for the very inexpensive GDA storage. I do the Backblaze step because I am paranoid that my local restic repositories may get corrupted somehow, and that I will sync that corruption to the cloud, so I like to have a second repository)

Here are some performance results from Dec 05:

Repository 1:  17GiB total size, 25MiB added to repo on this run
  Local backup:  2 minutes, 46 seconds
  Sync to Amazon: 10 minutes, 41 seconds  (includes many deletions)
  B2 backup: 4 minutes, 2 seconds

Repository 2:  546GiB total size, 51KiB added to repo on this run
  Local backup:  1 minutes, 11 seconds
  Sync to Amazon: 4 minutes, 48 seconds  (includes many deletions)
  B2 backup: 3 minutes, 10 seconds

Repository 3:  51GiB total size, 97MiB added to repo on this run
  Local backup: 54 seconds
  Sync to Amazon: 7 minutes, 18 seconds (includes many deletions)
  B2 backup: 5 minutes, 4 seconds

I don’t think these comparison are actually apples to apples. I suspect the most accurate diagnosis of the problem was what @alexweiss mentioned.

There might be some pathological case with my block file device that restic cannot diff/handle efficiently.

I since moved to another way to backup that particular data format, as I was evaluating restic only…
Reproducing should be easy, just create a file filled with zeros and try to upload it.

This is only a wild guess of course :smiley:

Thank you for the time spent helping though!

What should be fairly easy to test is a local restic repository to rule out any bottlenecks on the file reading/handling side of things.