Trying to understand how hard links are handled by restic

I decided to run a test on my computer to better understand restic’s behavior. TL;DR: it works as advertised, minus the confusing total estimate which does not take hardlinks into account.

I created a bunch of hard links on my computer:

➜  Desktop mkdir hardlinks
➜  Desktop cd hardlinks 
➜  hardlinks dd if=/dev/urandom of=file1 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.12821 s, 49.3 MB/s
➜  hardlinks for i in {2..10} ; do ln file1 file1_hardlink_$i ; done
➜  hardlinks ll
total 1001M
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_10
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_2
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_3
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_4
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_5
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_6
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_7
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_8
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_9

I’ll now try to upload them to s3, and observe the behavior. If all goes as intended the final s3 bucket size should ~ 100MB. Let’s see.

➜  hardlinks restic -r s3:s3.amazonaws.com/mybucket/umberto/hardlinks init                               
created restic repository d9083000f7 at s3:s3.amazonaws.com/mybucket/umberto/hardlinks
                                                                                                                      
Please note that knowledge of your password is required to access                                                     
the repository. Losing your password means that your data is                                                          
irrecoverably lost.                

Here is the backup output:

➜  hardlinks restic -r s3:s3.amazonaws.com/mybucket/umberto/hardlinks --verbose --verbose backup $PWD 2>&1 | tee ../restic_backup.log
open repository
created new cache in /home/umberto/.cache/restic
lock repository
load index files
no parent snapshot found, will read all files
start scan on [/home/umberto/Desktop/hardlinks]
start backup on [/home/umberto/Desktop/hardlinks]
scan finished in 0.550s: 10 files, 1000.000 MiB
cd ..
new       /home/umberto/Desktop/hardlinks/file1, saved in 680.670s (10.205 MiB added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_2, saved in 681.028s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_3, saved in 0.738s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_4, saved in 0.725s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_5, saved in 0.689s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_6, saved in 0.694s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_7, saved in 0.689s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_8, saved in 0.738s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_9, saved in 0.752s (0 B added)
new       /home/umberto/Desktop/hardlinks/file1_hardlink_10, saved in 843.410s (89.795 MiB added)
new       /home/umberto/Desktop/hardlinks/, saved in 843.412s (0 B added, 45.666 KiB metadata)
new       /home/umberto/Desktop/, saved in 843.412s (0 B added, 384 B metadata)
new       /home/umberto/, saved in 843.412s (0 B added, 381 B metadata)
new       /home/, saved in 843.412s (0 B added, 380 B metadata)

Files:          10 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Data Blobs:     65 new
Tree Blobs:      5 new
Added to the repo: 100.046 MiB

processed 10 files, 1000.000 MiB in 14:19
snapshot cd863199 saved

so looks like the backup worked as expected. S3 console confirms the same size :+1:

While I was there I ran a restore as well:

➜  Desktop restic -r s3:s3.amazonaws.com/mybucket/umberto/hardlinks --verbose --verbose restore latest --target hardlinks_restore
repository d9083000 opened successfully, password is correct
restoring <Snapshot cd863199 of [/home/umberto/Desktop/hardlinks] at 2021-04-08 08:10:19.442100909 +0200 CEST by umberto@ultra> to hardlinks_restore
➜  Desktop ll hardlinks_restore/home/umberto/Desktop/hardlinks      
total 1001M
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_10
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_2
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_3
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_4
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_5
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_6
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_7
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_8
-rw-rw-r-- 10 umberto umberto 100M Apr  8 08:02 file1_hardlink_9
➜  Desktop du -hs hardlinks_restore/home/umberto/Desktop/hardlinks
101M    hardlinks_restore/home/umberto/Desktop/hardlinks

all look good! I’m going to retry and be more patient next time :crossed_fingers:

2 Likes