We today happily announce the release of restic 0.14.0! \o/
This is a companion discussion topic for the original entry at https://restic.net/blog/2022-08-25/restic-0.14.0-released/
This is a companion discussion topic for the original entry at https://restic.net/blog/2022-08-25/restic-0.14.0-released/
15 Likes
Not sure if I got it right, but after the repository has been migrated, every subsequent backups will use compression, whether or not the flag --compression is added (unless its value is off obviously), right?
I’ve wondered the same 
For a repository using repo format version 2, the default is --compression auto. So, yes compression will just work after migrating a repository.
1 Like
I’m guessing this is a question that is going to be an FAQ. I’ll write up a section and submit a PR. I’ll address “Restic 0.14 supports compression. How is this handled for new and existing repositories?” and “How do I use compression?”, any suggestions for other likely new FAQs?
One thing that isn’t clear from the documentation is what decides the type of compression used? Both init and backup accept the --compression flag. My tests suggest that the --compression flag to init doesn’t seem to do anything; one might expect that a repo’s init --compression flag would decide the backup --compression auto behaviour.
1 Like
I’d suggest waiting a bit so that we can determine what actually becomes frequently asked questions and take it from there, instead of trying to predict things.
It’s stated in the documentation, in the Compression section:
For a repository using at least repository format version 2, you can configure how data is compressed with the option
--compression. It can be set toauto(the default, which will compress very fast) …
FWIW, we managed to get a bug into the documentation when releasing 0.14.0 
- it states in the repository version table that the default version is v1. This is incorrect - since restic 0.14.0 the default repository version is v2, and this means that compression is automatically enabled when backing up. The documentation error will be fixed in 0.14.1.
1 Like
So after migration, I have v2 and already 400+G of data.
Now a couple of new chunks come in that are already there (uncompressed). How will this be handled?
- Reuse of existing chunks (still uncompressed)
- Replace existing chunks (now compressed)
- Keep both, in worst case exploding the repo size
The compression level is always only applied for that run of restic. The default is auto (hardcoded into restic), but it can be changed via an option or by setting the environment variable $RESTIC COMPRESSION. The idea is that you can decide to use max compression if you’re behind a low-bandwidth uplink but don’t mind the extra time (e.g. during vacation) whereas at the end of a workday with a high-bandwidth connection you’d like to get home a soon as possible and disable compression altogether for that run.
The repository version 2 can have blocks of files stored either in compressed or uncompressed form, the compression level during backup and prune decides what happens. The metadata (info about filenames, directories, the index etc) is always compressed as it is stored as JSON documents.
Does that help? You can recycle parts of my text for the FAQ if you like
2 Likes
That depends on the operation:
- for a run of
restic backupit’ll just reuse the existing (uncompressed) data, but new data is stored in compressed form (depends on what you pass to restic via--compressionor the environment variable$RESTIC_COMPRESSION). - for a run of
restic pruneit depends on whether or not the chunk is to be repacked (reuploaded): if it is, the chunk is compressed (depending on the selected compression level), otherwise is is left uncompressed in the repo
The general idea is that existing data in the repo is still valid after upgrading it to repo version 2. We think that most people just want to start using compression from now on but are not keep to reupload all their data. And if you’d like to do that there’s an option to the prune command you can use to compress and reupload all data.
Does that help?
3 Likes
Thank you, yes it does help; I hadn’t thought about prune runs. As @rawtaz suggested I’ll keep an eye out for other potential questions and send over a PR.
2 Likes
Congratulations! Great news!
If I understood correctly, to migrate a repository, I should issue:
restic -r repo migrate upgrade_repo_v2
restic -r repo prune
The prune is without any retention policy. Is that correct?
Does this compress only new data or compress old data as well?
From docs, do I compress old data if I run
restic -r repo prune --repack-uncompressed
1 Like
Yes, as restic forget deals with retention policies, whereas restic prune does repository cleanup and garbage collection tasks. You may have mixed up the two because there’s restic forget [...] --prune, which will run restic prune afterwards but only if snapshots have been removed (so it’s likely that data will be removed from the repo). In this regard, restic forget is deciding which branches of a plant should be trimmed or removed, and restic prune is actually getting out the trimmer and start cutting.
Only data that restic needs to touch is compressed, that applies to both restic backup and restic prune, although you can call restic prune --repack-uncompressed to make it compress all uncompressed data.
2 Likes
Thank you for the hard work. I’ve updated a few repos (including a 7 TB one) and everything so far is great.
1 Like
Very cool @bbrendon! Did you prune --repack-uncompressed? If yes, can you share some stats on how big it was before and after that process? Of course, if you hadn’t pruned it without the compression support enabled, some of the reduction would be due to the pruning itself rather than the compression.
1 Like
No, I did not run repack. I wasn’t that ambitious.
Mostly because the server /tmp is only 7 GB and it does a lot of stuff so I didn’t want an extended load spike slowing other things down.
1 Like
It’s time to participate in a podcast again, and reveal the next step for restic!
3 Likes
Now with compression support i use it to backup my Minecraft Server. Some stats for people who are curious. Minecraft 1.19.2 World with around 1.3 TB of Data, 6328507 files in total. Compressed with max settings and stored on Minio S3 Server. Compressed size is around 530 GB.
Thank you devs for this awesome release.
1 Like
One thing to note is, if you tend to use --compression max and then run a prune job without specifying the compression level either as an environment variable or as a command switch, then anything repacked will be compressed using auto, instead of max. Also currently there’s no way to recompress data backed up with auto compression with max compression, besides copying to another repo, then copying back. Correct?
The error message when a v2 repository is mistakenly upgraded can be made more helpful. Right now the error message is
Migration upgrade cannot be applied: check failed
If you want to apply this migration anyways re-run with option – – force
Instead, the message could be:
Error: repository is already in v2 format.
2 Likes
I was first thinking that there seems to be the caveat that these mixed compression levels won’t deduplicate against each other, as they’re not equal. But then I figured: probably the filename is the sha2 hash of the decompressed version, so deduplication works perfectly between compression levels.
“edit” I was going to post this as a question but actually this is indeed the answer as mentioned by fd0 in a hidden comment in the PR