Hi there, I am testing restic for backing up a file of ~4G to Backblaze.
I was expecting a bit more speed from restic I have to say and I wonder I am doing anything wrong on my side.
These are the logs/details:
scan finished in 1.917s: 2 files, 4.379 GiB
modified /var/lib/dir/file.enc.md5, saved in 0.001s (44 B added)
modified /var/lib/dir/file.enc, saved in 841.057s (21.533 MiB added)
modified /var/lib/dir/, saved in 841.073s (0 B added, 194.604 KiB metadata)
modified /var/dir/, saved in 841.073s (0 B added, 369 B metadata)
modified /var/, saved in 841.074s (0 B added, 367 B metadata)
Files: 0 new, 2 changed, 0 unmodified
Dirs: 0 new, 3 changed, 0 unmodified
Data Blobs: 12 new
Tree Blobs: 4 new
Added to the repo: 21.725 MiB
I am not sure what the bottleneck is but I can definitely try things out in order to provide more info.
In any case, I got up and running very quickly so that’s a big plus!
Could it be that the 841s are the time that restic needs to
read the 4GB of the file
chunks this into blobs and hash each blob
for each blob: check if already existing and encrypt and save it if not
As only 22MB have been uploaded, lots of blobs were obviously identical, maybe big parts of the files being zeroes?
Now uploading to Backblaze is only done in 3. and if this would take 841s, there is a big problem…
Point 2. shouldn’t take much time as this is done in memory and recent CPUs are fast enough.
Actually, I would guess that 1. might take quite some time depending on the block device you are using… 4GB in 841s would be around 5 MB/s. What kind of device are you using? Could that be the bottleneck?
If it is helpful, I can offer some performance data from my setup, which includes local, AWS, and B2 components.
In this data set, the following operations were performed:
restic backup to a local drive (I use 3 different repos, one for each major filesystem)
restic --forget --prune a few snapshots in each of those 3 repos
aws s3 sync those repositories to AWS
restic backup of the same filesystems directly to a separate repository in Backblaze.
(I use the local+sync process for the main backup, because I like to have a local copy for fast restores and prunes, and I put it on AWS for the very inexpensive GDA storage. I do the Backblaze step because I am paranoid that my local restic repositories may get corrupted somehow, and that I will sync that corruption to the cloud, so I like to have a second repository)
Here are some performance results from Dec 05:
Repository 1: 17GiB total size, 25MiB added to repo on this run
Local backup: 2 minutes, 46 seconds
Sync to Amazon: 10 minutes, 41 seconds (includes many deletions)
B2 backup: 4 minutes, 2 seconds
Repository 2: 546GiB total size, 51KiB added to repo on this run
Local backup: 1 minutes, 11 seconds
Sync to Amazon: 4 minutes, 48 seconds (includes many deletions)
B2 backup: 3 minutes, 10 seconds
Repository 3: 51GiB total size, 97MiB added to repo on this run
Local backup: 54 seconds
Sync to Amazon: 7 minutes, 18 seconds (includes many deletions)
B2 backup: 5 minutes, 4 seconds