I just initialized a brand new, empty repository and performed a backup to it from a remote server over nfs. The source folder is 1.1T in size, but the repo is 3.8T after just one snapshot! What the heck is restic doing? Is that normal?
Please paste here the complete commands you used to do this, and all of their output. Please also include the commands you use to verify that the size of the repository is 300% larger than the source data.
The source server is STORE50B. The source folder is /zpool0/db_rsyncs. It contains a number of subfolders, where each subfolder contains a separate MySQL database for a different customer. The size of the parent folder is 1TB…
[root@store50b zpool0]# du -hs /zpool0/db_rsyncs
1011G /zpool0/db_rsyncs
As you can see, the parent folder has 94 subfolders, one for each customer…
[root@store50b ~]# ls -1 /zpool0/db_rsyncs|wc -l
94
The backup is done by a script, where restic is called separately against each subfolder. This is done to keep the backup and restore process for each customer separate and smple. Here is a log showing the backups and the commands that were used.
[root@store50b db_backup_stage2]# cat db_backup_stage2-2022_0417.log
12:01:00: — site000 —
12:01:00: exec: restic --tag site000 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site000
12:01:04: open repository
12:01:04: lock repository
12:01:04: load index files
12:01:04: no parent snapshot found, will read all files
12:01:04: start scan on [/zpool0/db_rsyncs/site000]
12:01:04: start backup on [/zpool0/db_rsyncs/site000]
12:01:04: scan finished in 0.239s: 410 files, 2.464 GiB
12:01:04: Files: 410 new, 0 changed, 0 unmodified
12:01:04: Dirs: 11 new, 0 changed, 0 unmodified
12:01:04: Data Blobs: 757 new
12:01:04: Tree Blobs: 12 new
12:01:04: Added to the repo: 394.952 MiB
12:01:04: processed 410 files, 2.464 GiB in 0:03
12:01:04: snapshot 51f5a3c7 saved
12:01:04: — site018 —
12:01:04: exec: restic --tag site018 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site018
12:02:00: open repository
12:02:00: lock repository
12:02:00: load index files
12:02:00: no parent snapshot found, will read all files
12:02:00: start scan on [/zpool0/db_rsyncs/site018]
12:02:00: start backup on [/zpool0/db_rsyncs/site018]
12:02:00: scan finished in 1.606s: 12481 files, 26.737 GiB
12:02:00: Files: 12481 new, 0 changed, 0 unmodified
12:02:00: Dirs: 11 new, 0 changed, 0 unmodified
12:02:00: Data Blobs: 53753 new
12:02:00: Tree Blobs: 12 new
12:02:00: Added to the repo: 23.510 GiB
12:02:00: processed 12481 files, 26.737 GiB in 0:55
12:02:00: snapshot 4aa6bec1 saved
12:02:00: — site031 —
12:02:00: exec: restic --tag site031 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site031
12:03:05: open repository
12:03:05: lock repository
12:03:05: load index files
12:03:05: no parent snapshot found, will read all files
12:03:05: start scan on [/zpool0/db_rsyncs/site031]
12:03:05: start backup on [/zpool0/db_rsyncs/site031]
12:03:05: scan finished in 1.775s: 12395 files, 31.645 GiB
12:03:05: Files: 12395 new, 0 changed, 0 unmodified
12:03:05: Dirs: 11 new, 0 changed, 0 unmodified
12:03:05: Data Blobs: 57898 new
12:03:05: Tree Blobs: 12 new
12:03:05: Added to the repo: 28.237 GiB
12:03:05: processed 12395 files, 31.645 GiB in 1:03
12:03:05: snapshot 6fa8798c saved
12:03:05: — site032 —
12:03:05: exec: restic --tag site032 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site032
12:04:35: open repository
12:04:35: lock repository
12:04:35: load index files
12:04:35: no parent snapshot found, will read all files
12:04:35: start scan on [/zpool0/db_rsyncs/site032]
12:04:35: start backup on [/zpool0/db_rsyncs/site032]
12:04:35: scan finished in 1.660s: 12653 files, 46.326 GiB
12:04:35: Files: 12653 new, 0 changed, 0 unmodified
12:04:35: Dirs: 12 new, 0 changed, 0 unmodified
12:04:35: Data Blobs: 84673 new
12:04:35: Tree Blobs: 13 new
12:04:35: Added to the repo: 42.019 GiB
12:04:35: processed 12653 files, 46.326 GiB in 1:30
12:04:35: snapshot f325d3cd saved
12:04:35: — site040 —
12:04:35: exec: restic --tag site040 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site040
12:05:51: open repository
12:05:51: lock repository
12:05:51: load index files
12:05:51: no parent snapshot found, will read all files
12:05:51: start scan on [/zpool0/db_rsyncs/site040]
12:05:51: start backup on [/zpool0/db_rsyncs/site040]
12:05:51: scan finished in 1.876s: 12656 files, 38.436 GiB
12:05:51: Files: 12656 new, 0 changed, 0 unmodified
12:05:51: Dirs: 11 new, 0 changed, 0 unmodified
12:05:51: Data Blobs: 69873 new
12:05:51: Tree Blobs: 12 new
12:05:51: Added to the repo: 34.215 GiB
12:05:51: processed 12656 files, 38.436 GiB in 1:15
12:05:51: snapshot 323b177b saved
12:05:51: — site044 —
12:05:51: exec: restic --tag site044 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site044
12:07:35: open repository
12:07:35: lock repository
12:07:35: load index files
12:07:35: no parent snapshot found, will read all files
12:07:35: start scan on [/zpool0/db_rsyncs/site044]
12:07:35: start backup on [/zpool0/db_rsyncs/site044]
12:07:35: scan finished in 1.939s: 12498 files, 54.215 GiB
12:07:35: Files: 12498 new, 0 changed, 0 unmodified
12:07:35: Dirs: 11 new, 0 changed, 0 unmodified
12:07:35: Data Blobs: 99267 new
12:07:35: Tree Blobs: 12 new
12:07:35: Added to the repo: 49.672 GiB
12:07:35: processed 12498 files, 54.215 GiB in 1:43
12:07:35: snapshot 3902f98c saved
12:07:35: — site050 —
12:07:35: exec: restic --tag site050 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site050
12:08:37: open repository
12:08:37: lock repository
12:08:37: load index files
12:08:37: no parent snapshot found, will read all files
12:08:37: start scan on [/zpool0/db_rsyncs/site050]
12:08:37: start backup on [/zpool0/db_rsyncs/site050]
12:08:37: scan finished in 2.581s: 18139 files, 21.816 GiB
12:08:37: Files: 18139 new, 0 changed, 0 unmodified
12:08:37: Dirs: 12 new, 0 changed, 0 unmodified
12:08:37: Data Blobs: 25438 new
12:08:37: Tree Blobs: 12 new
12:08:37: Added to the repo: 21.289 GiB
12:08:37: processed 18139 files, 21.816 GiB in 1:02
12:08:37: snapshot d5808b17 saved
12:08:38: — site055 —
12:08:38: exec: restic --tag site055 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site055
12:09:10: open repository
12:09:10: lock repository
12:09:10: load index files
12:09:10: no parent snapshot found, will read all files
12:09:10: start scan on [/zpool0/db_rsyncs/site055]
12:09:10: start backup on [/zpool0/db_rsyncs/site055]
12:09:10: scan finished in 2.216s: 11967 files, 12.578 GiB
12:09:10: Files: 11967 new, 0 changed, 0 unmodified
12:09:10: Dirs: 12 new, 0 changed, 0 unmodified
12:09:10: Data Blobs: 22382 new
12:09:10: Tree Blobs: 13 new
12:09:10: Added to the repo: 10.309 GiB
12:09:10: processed 11967 files, 12.578 GiB in 0:31
12:09:10: snapshot c9438cba saved
12:09:10: — site057 —
12:09:10: exec: restic --tag site057 -r /restic_remote_repo.nfs --verbose backup /zpool0/db_rsyncs/site057
12:15:18: open repository
12:15:18: lock repository
12:15:18: load index files
12:15:18: no parent snapshot found, will read all files
12:15:18: start scan on [/zpool0/db_rsyncs/site057]
12:15:18: start backup on [/zpool0/db_rsyncs/site057]
12:15:18: scan finished in 2.215s: 12325 files, 200.551 GiB
12:15:18: Files: 12325 new, 0 changed, 0 unmodified
12:15:18: Dirs: 11 new, 0 changed, 0 unmodified
12:15:18: Data Blobs: 372802 new
12:15:18: Tree Blobs: 12 new
12:15:18: Added to the repo: 190.257 GiB
12:15:18: processed 12325 files, 200.551 GiB in 6:07
12:15:18: snapshot b07b5eae saved
The log continues for a total of 94 invocations of restic, one for each subfolder.
The target server is STORE50A.
On that server, the restic_repo folder, which was empty before the backup, is now 3.8 TB in size.
[root@store50a ~]# du -hs /zpool0/restic_repo
3.8T /zpool0/restic_repo
The repo contains 1 snapshot for each source dabatabase subfolder.
[root@store50a ~]# restic -r /zpool0/restic_repo snapshots|grep site|wc -l
94
Hi @forbin
Welcome 
I see the added size lower than the processed size which looks normal
Can you please share the output of
restic stats --mode raw-data
restic stats --mode restore-size
du -hs /zpool0/restic_repo/*
Thank you
Sd/
root@store50a zpool0]# restic stats -r restic_repo --mode raw-data
repository cc56b432 opened successfully, password is correct
scanning…
Stats in raw-data mode:
Snapshots processed: 94
Total Blob Count: 7716191
Total Size: 3.737 TiB
[root@store50a zpool0]# restic -r restic_repo stats --mode restore-size
repository cc56b432 opened successfully, password is correct
scanning…
Stats in restore-size mode:
Snapshots processed: 94
Total File Count: 1149713
Total Size: 4.079 TiB
[root@store50a zpool0]# du -hs restic_repo
3.8T restic_repo
@forbin, can you verify if deduplication and/or compression is enabled on the zfs pool on store50b ?
zpool list
zfs get all zpool0 | grep compress
Both zfs features would cause du to show you incorrect source data sizes.
Compression is enabled on both the source server and the destination server.
Source:
[root@store50a ~]# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zpool0 17.5T 5.50T 12.0T - - 18% 31% 1.00x ONLINE -
[root@store50a ~]# zfs get all|grep compress
zpool0 compressratio 1.42x -
zpool0 compression lz4 local
zpool0 refcompressratio 1.42x -
Destination:
[root@store50b ~]# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zpool0 17.5T 12.8T 4.63T - - 14% 73% 1.00x ONLINE -
[root@store50b ~]# zfs get all zpool0 | grep compress
zpool0 compressratio 1.90x -
zpool0 compression lz4 local
zpool0 refcompressratio 1.90x -
[root@store50b ~]#
The compression ratio is a bit lower on the source than on the destination.
I guess the main reason for the size difference is that restic doesn’t compress it’s repositories yet (see Implement compression support by MichaelEischer · Pull Request #3666 · restic/restic · GitHub ). What size does du -hs --apparent-size print?
Sorry, got caught up yesterday and was away today. I really can’t add anything of value that others haven’t already said. I think that if restic saves X amount of data, it’s because it actually was served/read that amount of data from the filesystem. If the filesystem reports that there’s less data than restic read, there has to be something going on with your ZFS, and compression sounds like a good idea to look at. Are you also deduplicating in your ZFS? Maybe you could post the complete log file for the backup run, just so people have 100% of that information available.
@MichaelEischer Here’s what it says…
[root@store50a zpool0]# du -hs --apparent-size restic_repo
3.9T restic_repo
I don’t see how the lack of repo compression could be a factor, though. Even without compression, the repo size should not exceed the size of the source data (maybe a little allowing for some metadata, but not by 300%).
Blockquote If the filesystem reports that there’s less data than restic read, there has to be something going on with your ZFS
The problem is that restic seems to be writing three times more data than it is reading.
forbin,
i would say you have not yet proven that restic is writing 3x the amount of data that it reads.
Bear in mind that tools like du get easily fooled by advanced filesystems like zfs.
I am no expert in zfs but e.g. with compression du on zfs shows the size after compression, not the orignal size… Also setting different blocksize for zfs could cause du to report wrong sizes.
If restic is reading and then encrypts the source data, I can imagine that at your destination server the zfs compression of the restic repo files is not effective anymore.
Are you worried abour the (reported) space of source vs destination, or the integrity of your backup?
Not if the root cause is that the data is of the size restic stores, but on your ZFS for various reasons it’s using less disk space. Presumably ZFS reports the actually used disk space, so if you store a file that’s 3 GB and it only takes up 1 GB on disk, you have this type of situation.
The problem is that there is 1 TB if source data, and after a few consecutive days of backups (maybe 3 or 4 days) it completely fill the 17 TB of storage on the destination server.
The issue I’m struggling with is there is 1 TB if source data, and after a few consecutive days of backups (maybe 3 or 4 days) it completely fill the 17 TB of storage on the destination server.
Yeah, totally with you that it’s a problem 
We should try to figure it out.
Can you please post the output of zfs list -tall | grep zpool0/db_rsyncs (or similar so that we can see all the ZFS filesystems that are related to the db_syncs directory)?
Restic will only grow with new or changed data so the amount that will be added depends on how static / dynamic the source data is.
You now anticipate on ~100% new or changed data every day, that seems a bit wild?
Restic has backuped 4.079 TiB of data and saved 3.737 TiB in the repository.
The problem is not that restic saves more in the repo than it backups, but the mismatch between the 1.1TiB you are reporting and the 4.079 TiB which restic obviously did read. This mismatch, however, does not depend on restic, but on zfs and how you use it to get this information.