Restic 0.14.0 Released

Thank you, yes it does help; I hadn’t thought about prune runs. As @rawtaz suggested I’ll keep an eye out for other potential questions and send over a PR.

2 Likes

Congratulations! Great news!

If I understood correctly, to migrate a repository, I should issue:

restic -r repo migrate upgrade_repo_v2

restic -r repo prune

The prune is without any retention policy. Is that correct?

Does this compress only new data or compress old data as well?

From docs, do I compress old data if I run

restic -r repo prune --repack-uncompressed

1 Like

Yes, as restic forget deals with retention policies, whereas restic prune does repository cleanup and garbage collection tasks. You may have mixed up the two because there’s restic forget [...] --prune, which will run restic prune afterwards but only if snapshots have been removed (so it’s likely that data will be removed from the repo). In this regard, restic forget is deciding which branches of a plant should be trimmed or removed, and restic prune is actually getting out the trimmer and start cutting.

Only data that restic needs to touch is compressed, that applies to both restic backup and restic prune, although you can call restic prune --repack-uncompressed to make it compress all uncompressed data.

2 Likes

Thank you for the hard work. I’ve updated a few repos (including a 7 TB one) and everything so far is great.

1 Like

Very cool @bbrendon! Did you prune --repack-uncompressed? If yes, can you share some stats on how big it was before and after that process? Of course, if you hadn’t pruned it without the compression support enabled, some of the reduction would be due to the pruning itself rather than the compression.

1 Like

No, I did not run repack. I wasn’t that ambitious. :slight_smile:

Mostly because the server /tmp is only 7 GB and it does a lot of stuff so I didn’t want an extended load spike slowing other things down.

1 Like

It’s time to participate in a podcast again, and reveal the next step for restic!

3 Likes

Now with compression support i use it to backup my Minecraft Server. Some stats for people who are curious. Minecraft 1.19.2 World with around 1.3 TB of Data, 6328507 files in total. Compressed with max settings and stored on Minio S3 Server. Compressed size is around 530 GB.

Thank you devs for this awesome release.

1 Like

One thing to note is, if you tend to use --compression max and then run a prune job without specifying the compression level either as an environment variable or as a command switch, then anything repacked will be compressed using auto, instead of max. Also currently there’s no way to recompress data backed up with auto compression with max compression, besides copying to another repo, then copying back. Correct?

The error message when a v2 repository is mistakenly upgraded can be made more helpful. Right now the error message is

Migration upgrade cannot be applied: check failed

If you want to apply this migration anyways re-run with option – – force

Instead, the message could be:

Error: repository is already in v2 format.

2 Likes

I was first thinking that there seems to be the caveat that these mixed compression levels won’t deduplicate against each other, as they’re not equal. But then I figured: probably the filename is the sha2 hash of the decompressed version, so deduplication works perfectly between compression levels.

“edit” I was going to post this as a question but actually this is indeed the answer as mentioned by fd0 in a hidden comment in the PR :slight_smile:

Will Restic codebase itself be pruned from repository v1 code?

That would be good. People could install 0.13 by a flag using a different non-maintained code.

Otherwise, there will be a lot of dead code and bloated flags and vulnerability sources.

Thank you for this update!!!, @all devs :slight_smile:
Compression was really the missing link, and now we have it!

availability of compression sure is great news. thanks to the devs!

simple question: what would be the considered “most reasonable” (if not best) practice for migrating and further use of a jpeg-only repo (yes: photos :))?

it would seem that even at the chunk level jpeg data are not really compressible further significantly I guess?

so: should I just migrate and enforce compression off afterwards for all interactions with that repo? or stay with the default compression auto nonetheless? how much computational overhead (%) does auto compression cause, roughly?

I use restic to back up a variety of different systems. One of them is a nextcloud instance which mostly hosts (jpeg) images. Here are the numbers:

254G without compression
238G with compression

So, not a lot.

For my home directory, it looks like this:

382G without compression
292G with compression

A bit better.
Finally, a whole system backup (several raspi installations with a variety of data):

40G without compression
13G with compression

That’s quite significant and also expected, as a typical installation contains quite a bit of easily compressible data.

I didn’t notice a significant/measurable slow down during backup.

2 Likes

There are no plans at the moment to remove version 1 compatibility. After all it’s just a few hundred lines of code.

1 Like

Awesome release and thank you for a great project!

Here are my three machines running restic before and after compression was enabled and run for all old snapshots. Really good results :slight_smile:

4 Likes

I just converted my home setup and one customer over to repo v2 and compressed the existing repos. My main repo at home has to stay as it is because I don’t have enough free space to recompress. So far everything went perfectly smooth without a single problem or error. Many thanks to the awesome dev team!

In case someone is interested, here are some repo sizes relative to their uncompressed original:

  • Some family computers: 89%, 94%, 94% (lots of JPGs)
  • Moodle training platform: 71%
  • Web servers: 53%, 78%
  • Linux software development workstation: 64%
2 Likes

Then recompress iteratively. For my TiB remote I do it in chunks of 10GiB per day. Will take ages, though probably save me some money as it is not doubling the space used temporarily.

$  restic -r rest:https://… prune --repack-uncompressed --compression max --max-repack-size 10G
5 Likes

Oh nice, thank you! I somehow missed that option.

2 Likes