Hi, I’m going to use restic to perform block device backup.
I run the following command to perform the backup:
cat /dev/drbd/by-res/backup-one-vm-8947-disk-2/0 | \
restic backup --tag=one-vm-8947-disk-2 --stdin --stdin-filename /dev/drbd/by-res/one-vm-8947-disk-2/0
Right now I have already 37TB used of 1299 snapshots, located on software zfs draid of 45x6TB disks
backup speed is pretty fine and fully utilizes 10gb/s line:
https://asciinema.org/a/gg8zH0WtQwcKyR13bR3gj7jqf
However the recovery from the backup is quite slow even if running locally:
# restic -r /data/backup --verbose dump 46a169a5 /dev/drbd/by-res/one-vm-7994-disk-2/0 | dd of=/dev/null status=progress
2960962048 bytes (3.0 GB) copied, 222.868152 s, 13.3 MB/s
and ends with:
41943719+13209 records in
41950384+0 records out
21478596608 bytes (21 GB) copied, 833.371 s, 25.8 MB/s
Thus the speed of recovery usually it is not going over 26MB/s
CPU usage isn’t so high:
# top
top - 17:03:48 up 1 day, 21:21, 6 users, load average: 4.29, 3.51, 3.82
Tasks: 530 total, 1 running, 380 sleeping, 0 stopped, 0 zombie
%Cpu(s): 16.5 us, 0.5 sy, 0.0 ni, 75.1 id, 7.9 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 74190032 total, 23056312 free, 47609356 used, 3524368 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 25678432 avail Mem
# top | grep restic
14709 root 20 0 12.6g 12.3g 13320 S 77.8 17.3 11:51.01 restic
14709 root 20 0 12.6g 12.3g 13320 S 75.8 17.3 11:53.30 restic
14709 root 20 0 12.6g 12.3g 13320 S 71.0 17.3 11:55.45 restic
14709 root 20 0 12.6g 12.3g 13320 S 71.2 17.3 11:57.60 restic
14709 root 20 0 12.6g 12.3g 13320 S 63.9 17.3 11:59.53 restic
14709 root 20 0 12.6g 12.3g 13320 S 58.9 17.3 12:01.31 restic
# htop
1 [||| 16.7%] 5 [ 0.0%] 9 [||||||47.5%] 13 [|| 17.9%]
2 [|| 4.0%] 6 [||| 14.4%] 10 [|||| 33.5%] 14 [| 7.0%]
3 [| 0.5%] 7 [||||||51.0%] 11 [|||| 25.4%] 15 [ 0.0%]
4 [ 0.0%] 8 [|| 12.9%] 12 [||||||45.0%] 16 [|| 2.0%]
Mem[|||||||||||||||||||49.0G/70.8G] Tasks: 195, 864 thr; 5 running
Swp[ 0K/0K] Load average: 4.30 3.60 3.84
Uptime: 1 day, 21:22:17
Memory is enough:
# free -h
total used free shared buff/cache available
Mem: 70G 49G 19G 244M 2.0G 20G
Swap: 0B 0B 0B
Utilization of the disks fluctuates between 0 and 17%
# iostat -x 2
avg-cpu: %user %nice %system %iowait %steal %idle
6.54 0.00 2.01 0.91 0.00 90.53
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 9.00 0.50 170.00 8.00 37.47 0.06 5.74 6.06 0.00 5.79 5.50
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdas 0.00 0.00 5.50 0.00 484.00 0.00 176.00 0.06 10.45 10.45 0.00 7.55 4.15
sdt 0.00 0.00 6.00 0.00 504.00 0.00 168.00 0.04 7.25 7.25 0.00 5.42 3.25
sdaj 0.00 0.00 4.50 0.00 342.00 0.00 152.00 0.04 8.44 8.44 0.00 7.67 3.45
sdaa 0.00 0.00 5.00 0.00 404.00 0.00 161.60 0.04 7.30 7.30 0.00 6.50 3.25
sdar 0.00 0.00 0.50 0.00 42.00 0.00 168.00 0.00 2.00 2.00 0.00 2.00 0.10
sdac 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.03 5.18 5.18 0.00 4.00 2.20
sdp 0.00 0.00 0.50 0.00 44.00 0.00 176.00 0.00 0.00 0.00 0.00 0.00 0.00
sdy 0.00 0.00 4.00 0.00 336.00 0.00 168.00 0.02 4.75 4.75 0.00 4.62 1.85
sdat 0.00 0.00 4.00 0.00 352.00 0.00 176.00 0.02 4.25 4.25 0.00 4.38 1.75
sdw 0.00 0.00 4.50 0.00 360.00 0.00 160.00 0.05 10.22 10.22 0.00 9.33 4.20
sdi 0.00 0.00 0.50 0.00 44.00 0.00 176.00 0.00 7.00 7.00 0.00 7.00 0.35
sdc 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.05 9.36 9.36 0.00 5.73 3.15
sdn 0.00 0.00 4.00 0.00 352.00 0.00 176.00 0.02 5.12 5.12 0.00 5.00 2.00
sdal 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.04 6.45 6.45 0.00 5.45 3.00
sdx 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdah 0.00 0.00 4.50 0.00 378.00 0.00 168.00 0.02 3.89 3.89 0.00 3.78 1.70
sdag 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 0.00 4.00 0.00 336.00 0.00 168.00 0.01 3.38 3.38 0.00 3.25 1.30
sdu 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 6.00 0.00 528.00 0.00 176.00 0.06 9.58 9.58 0.00 6.08 3.65
sdad 0.00 0.00 6.50 0.00 402.00 0.00 123.69 0.03 4.69 4.69 0.00 4.62 3.00
sdq 0.00 0.00 4.50 0.00 354.00 0.00 157.33 0.03 6.00 6.00 0.00 6.00 2.70
sdam 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.09 15.27 15.27 0.00 9.45 5.20
sdr 0.00 0.00 4.50 0.00 360.00 0.00 160.00 0.03 6.78 6.78 0.00 6.11 2.75
sdap 0.00 0.00 4.50 0.00 378.00 0.00 168.00 0.02 4.33 4.33 0.00 4.00 1.80
sdz 0.00 0.00 5.00 0.00 402.00 0.00 160.80 0.03 5.40 5.40 0.00 4.80 2.40
sdk 0.00 0.00 5.50 0.00 484.00 0.00 176.00 0.06 11.27 11.27 0.00 7.55 4.15
sdaq 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.04 7.55 7.55 0.00 5.64 3.10
sdau 0.00 0.00 5.00 0.00 402.00 0.00 160.80 0.04 7.70 7.70 0.00 7.00 3.50
sdh 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.07 12.18 12.18 0.00 8.36 4.60
sdae 0.00 0.00 8.00 0.00 374.00 0.00 93.50 0.08 10.62 10.62 0.00 10.31 8.25
sdo 0.00 0.00 4.00 0.00 336.00 0.00 168.00 0.02 4.25 4.25 0.00 4.12 1.65
sdg 0.00 0.00 4.50 0.00 342.00 0.00 152.00 0.04 7.89 7.89 0.00 7.56 3.40
sdak 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdao 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdai 0.00 0.00 4.50 0.00 360.00 0.00 160.00 0.02 3.78 3.78 0.00 3.00 1.35
sds 0.00 0.00 5.50 0.00 484.00 0.00 176.00 0.04 6.82 6.82 0.00 5.18 2.85
sdv 0.00 0.00 4.50 0.00 360.00 0.00 160.00 0.04 8.44 8.44 0.00 7.56 3.40
sdm 0.00 0.00 5.50 0.00 462.00 0.00 168.00 0.07 12.45 12.45 0.00 8.64 4.75
sdd 0.00 0.00 4.50 0.00 342.00 0.00 152.00 0.04 8.78 8.78 0.00 7.89 3.55
sdj 0.00 0.00 4.00 0.00 336.00 0.00 168.00 0.02 5.25 5.25 0.00 5.00 2.00
sdf 0.00 0.00 4.50 0.00 342.00 0.00 152.00 0.05 10.44 10.44 0.00 9.44 4.25
sdaf 0.00 0.00 4.50 0.00 360.00 0.00 160.00 0.04 8.78 8.78 0.00 8.22 3.70
sdan 0.00 0.00 4.00 0.00 336.00 0.00 168.00 0.02 5.75 5.75 0.00 5.88 2.35
sdab 0.00 0.00 5.50 0.00 382.00 0.00 138.91 0.02 3.73 3.73 0.00 3.45 1.90
Any ideas?