After a short venture to Borg I’ve come back to restic to backup a lot of files (also depulicated ones from several places on different HDDs and Machines) combined with a Minio Server via S3-API.
I do have around 221.335 files in S3 now at 1TB - which makes for a theoretical packsize of less then 6MB if I’m right?
I was under the impression that the default pack size was 16MB?
Restic init was simply done via restic init (so no flags, just defaults).
Since I’m backing this up from spinning rust to spinning rust (HDDs) this is a little painfull (nothing that can’t be fixed be re-doing it with a bigger pack-size).
OT/Idea:
Maybe a restic init agent (if done without flags) that asks for the approximate conditions would be really cool that chooses a sane value for the conditions from experience …
…say how large you expect the repository to grow at, with how many files (if known) … so that the pack-size and other parameters like compression paramters - that would really help.
Number of files in your minio storage ≠ number of files in your restic repo
Use minio specific tools like mc or MinIO Console to gather your S3 storage statistics. Underlying number of files can be very different depending on your configuration (e.g. versioning or redundancy).
I am not aware of restic functionality to provide you with packs’ stats. Myself I use rustic for this:
rustic -r repo repoinfo
will give you stats like:
| File type |
Count |
Total Size |
| Key |
1 |
466 B |
| Snapshot |
4 |
1.2 kiB |
| Index |
2 |
923.3 kiB |
| Pack |
51 |
3.0 GiB |
| Total |
58 |
3.0 GiB |
Well, obviously this is specifically a single repo for that case that was previously empty.
Number of files in total was gathered from Minio Console.
Isn’t the Pack Size refering to the packed objects = files inside the /data folder?
If I go in there all files are <6 mb like I expected to see.
Is there a mechanism that decides the initial pack size implemented in the last version or a I misunderstanding how Restics* Packs work? Just did a quick read trough the docs and didn’t see anything in that regard.
I’m using restic 0.16.0
https://restic.readthedocs.io/en/stable/047_tuning_backup_parameters.html#pack-size
Edit: Thanks for the recommendation in regards to rustic
Default pack size is nowadays 16MB. It would be very strange if average for new repo is 6MB.
What rustic repoinfo says?
BTW - I do not recommend to use rustic over restic. Neither vice versa.
But in this case rustic can provide stats restic can not. This is how I use both nowadays - sometimes they complement each other very well:)
quick-check: rustic doesn’t have native s3 support yet so no chance of using it for that purpose right now …
you could use rclone to get stats only
Well, I’ll never know because everything was so pitty slow that I just deleted everything and got rid of Minio and switched to the rest-server with append-only instead
KISS - keep it simple and stupid
This time I set the pack size to 64M from the start manually via ENV.
That dual core i686 atom with 1GB ram really struggled despite not doing a lot. In the end minio would even crash after some time. rest-server looks much better from the start and is blazing fast compared to that nightmare of an slow experience.
1 Like
Haha! Great … I actually had an old restic version on the path - dang, that one didnt know pack-size and most likely defaulted to 6M! My fault … a restic version from the start would’ve helped.
Hah! Glad you found the real cause – I bet it would’ve otherwise been something nagging you for the next couple of weeks 
Going back to minio now? As the latest releases of restic have plenty of performance enhancements regarding S3 and overall.
1 Like
Nah I think the Atom is really streching it there … sticking to the rest-server now for that machine.
This is basically the backup of all the backups …
I do have Minio deployed on the Backup Master for the other machines tought …
Restic 0.16.0 also includes some new repository statistics: restic stats --mode debug
1 Like
Thx.
This is very useful stats command but seems to be well hidden. E.g. when I run:
$ restic stats --help
it only lists following modes:
* restore-size: (default)
* files-by-contents
* raw-data
* blobs-per-file