I’m using Wasabi S3 as our primary backup repository for multiple sites and servers and a staggered ‘rclone sync’ every other week to pull copies of that repo to different local destinations.
The repo consists of ~3.5 TB of data and contains over 2 million files, which causes a normal rclone sync wasabi-s3:restic/ /mnt/local/restic/ to take several hours just checking for changes.
Using the --size-only flag reduces the time to sync dramatically (just a couple of seconds on an up to date repo-copy), but I was wondering if comparing by size is actually enough to keep consistent copies of our repository.
As far as I understand restic’s design, this should result in proper clones, as no existing file should ever change its content and a prune would re-pack changed data into new files, while deleting the old ones.
Is my assumption correct?
Thanks a lot in advance and of course for restic itself! It’s been a pleasure working with it
have a consistent source and all existing files in the target with equal name as in the source should have the identical content to the source
select the right files to remove or copy - here a selection on the file name is actually enough for almost all files in the repo (the config file is the exception, but this usually doesn’t change)
make sure that the copying process and the removal process really works as intended.
If you assume that your source repository and your target repository are in an already-consistent state and that copying does not introduce any inconsistency, you should be fine.
Sorry, if I was being unclear by asking about consistent copies. Obviously I don’t care that much about any metadata as long as all the repository files themself and their contents are in sync.
So apart from the config file there aren’t any other files inside the repo that could get missed by a “dumb” sync operation (by name or size)?
Recent restic versions atomically create files in locally stored repositories. This works by first creating a temporary file and renaming it once it has been written completely. Once a file was written completely, restic will never modify it (except for the config file, but that also only happens when migrating to a newer repository format version). Thus, once a file has its final filename, its content won’t change.
