I’ve been running a rather large backups with restic (105TB). However, restic never finds a parent snapshot and always reads entire repo, which makes it a very slow process.
Afaik paths and hostname was used for selecting parent snapshot, but I am not totally sure about it (& couldn’t find it on docs in a quick search).
If I understand correctly, hostname changes on every run since you’re inside a container. Could you try enforcing a stable hostname with --hostname flag while giving the backup command?
When I started the container for the third time, I read the beginning of the operation carefully, and restic did find the parent snapshot. I was mistaken because the second backup snapshot seemed to be as slow as the first one.
The operation still takes about 12-13 days (even if no new files added). Sometimes files that are already uploaded seem to just take minutes, even though I would imagine they would be cached.
I run restic like this:
restic --repo=b2: backup --exclude=/data/builds /data
Anything I could do to speed backup up? How could I make sure that the parent snapshot and caching works as intended?
How much data does restic add to the repository for later backup runs? Is the repository cache shared across container runs? Or does restic have to download it each time? How large does the cache folder become?
When restic detects the proper parent snapshot, then it will be able to skip files that have not changed as long as the file metadata has not changed. Which filesystem is used for /data? 12-13 days is approximately what I get when dividing 100TB/100MB/s . So it looks like restic has to read everything again? How much CPU does restic use?
The restic doesn’t add much data between the runs. It does look like it is reading all files again (hence my initial suspicion of not finding the parent snapshot).
Restic practically doesn’t use any CPU (5% or so on the new run).
The /data is mapped into the container from outside of the container. The filesystem of /data inside and outside of the container is cifs.
Cache folder for some reason is very small.
This is from inside the container now:.
bash-5.1# ls -la
total 4
drwx------ 1 root root 152 Mar 14 08:52 .
drwx------ 1 root root 12 Mar 14 08:52 ..
drwx------ 1 root root 50 May 20 11:34 4ea1c84219c682f4ceda40c0cf41a24becc73548331cd3cea2341d44aa6908b1
-rw-r--r-- 1 root root 44 Mar 14 08:52 CACHEDIR.TAG
bash-5.1# pwd
/root/.cache/restic
And this is what mount shows for the /data inside the container: \134\13410.1.10.8\134videostore on /data type cifs (ro,relatime,vers=3.02(syno),blocksend,sec=ntlmssp,cache=strict,username=XXXXXXX,domain=WORKGROUP,uid=1026,forceuid,gid=100,forcegid,addr=10.1.10.8,file_mode=0777,dir_mode=0777,iocharset=utf8,nocase,nounix,mapposix,rsize=1048576,wsize=1048576,actimeo=1)
I am on restic version restic 0.12.1 compiled with go1.16.6 on linux/amd64