Upcoming restic 0.17.0 release

Hi,

the development cycle for restic 0.17.0 is finally nearing its end :partying_face: after taking way longer than expected. I hope we’ll be able to release the version in one or two weeks. The upcoming version includes a vast number of new features and internal improvements. I’ve picked just a few highlights in the following table, see the draft changelog for a complete list.

  • restore: Overwrite, delete and dry-run support (docs)
  • Support printing snapshot size in snapshots command (docs)
  • Windows: Backup and restore file attribute, xattrs and SecurityDescriptor (changelog 1, changelog 2, changelog 3)
  • macOS: support for fuse-t
  • Redesign backend error handling to improve reliability (changelog)
  • Deprecate legacy index format and s3legacy layout (changelog)
  • Support repositories with empty password (docs)
  • backup can omit snapshot creation if there was no change (docs)
  • Improve dump performance for large files
  • prune: Much lower memory usage and can be resumed now (changelog 1, changelog 2)

Please give those new features a try before we release them soon. To do so, either build restic from master (or fetch a pre-built binary from the beta builder). Please let us know how the new features work for you and report all issues you find!

22 Likes

The empty password repo and omitting snapshot for no changes. Super excited for those. Cannot wait for this update. Thank you for all the hard work. :pray:t4:

3 Likes

The empty password repo

What is the purpose or usefulness of making backups that are not password protected? I ask because I don’t know.

not everyone remembers passwords, maybe there is someone who wants to make a backup without a password.

1 Like

There has been endless discussions about this in the GitHub issues in the restic repository :slight_smile:

I think it would derail if we started discussing it in this thread, so I suggest that anyone who is interested to know more about this go there and just search for “empty password”. Let’s keep the discussion about it out of this thread, thanks!

8 Likes

OK, no problem, thanks

Thank you for your hard work on this! My one comment thus far is that when running a beta version, restic version doesn’t really give you much to go on if I log a fault? It just says

restic 0.16.5-dev (compiled manually) compiled with go1.22.0 on linux/amd64

Shouldn’t it have a git commit ID or a date or something in case you needed to narrow down the build I was using?

But thank you - I am testing now on a bunch of repos that I won’t get too crabby if I break them :slight_smile:

2 Likes

I guess I broke that a while ago: Fix VERSION file and dev version in binaries by MichaelEischer · Pull Request #4909 · restic/restic · GitHub . The git tag will be included when built using make restic or go run build.go. Thanks for noticing!

4 Likes

Hmm… I love the new snapshot size feature. I wonder if it would make sense to include it on the “compact” list? It’s often the paths that take up most of the screen real estate. Or perhaps something more configurable like snapshots --no-path / --no-size?

2 Likes

et voilà: snapshots: also show snapshot size in compact view by MichaelEischer · Pull Request #4913 · restic/restic · GitHub

3 Likes

The new option --dry-run is very interesting. The main use I will give to this option is to previously know the size of what I am restoring. I do not want to fill up my disk with a restore that does not fit in the free space I have. So the idea is to first make the restore with --dry-run option and if the size is less than free disk space, then launch it.

The first results I obtained seems not very promising in terms of speed, until I realize a little detail I would like to share. First, I will show the results:

  • I downloaded the latest linux kernel sources and did a snapshot on a local repository:
./restic_v0.16.5-811-g8e27a934d_linux_amd64 backup -r repo kernel_source/
enter password for repository:
repository 63b5531a opened (version 2, compression level auto)
created new cache in /root/.cache/restic
no parent snapshot found, will read all files
[0:00]          0 index files loaded

Files:       84283 new,     0 changed,     0 unmodified
Dirs:         5491 new,     0 changed,     0 unmodified
Added to the repository: 1.350 GiB (276.936 MiB stored)

processed 84283 files, 1.324 GiB in 0:07
snapshot 3b8dc776 saved
  • Restore:
# time ./restic_v0.16.5-811-g8e27a934d_linux_amd64 -r repo/ restore latest --target restored_files/
enter password for repository:
repository 63b5531a opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
restoring snapshot 3b8dc776 of [/root/kernel_source] at 2024-07-09 07:35:38.59864869 +0200 CEST by root@host to /root/restored_files/
Summary: Restored 89834 files/dirs (1.324 GiB) in 0:26

real	0m30.569s
user	0m38.612s
sys	0m6.988s
  • Then I did the same restore using --dry-run:
# time ./restic_v0.16.5-811-g8e27a934d_linux_amd64 -r repo/ restore latest --target restored_files/ --dry-run
enter password for repository:
repository 63b5531a opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
restoring snapshot 3b8dc776 of [/root/kernel_source] at 2024-07-09 07:35:38.59864869 +0200 CEST by root@host to /root/restored_files/
Summary: Restored 5551 files/dirs (0 B) in 0:24, skipped 84283 files/dirs 1.324 GiB

real	0m29.349s
user	0m25.988s
sys	0m1.524s

I thought that the --dry-run option will be faster than the actual restore, and at this point I felt a little sad about the results. The trick is to restore to a empty dir, the restore process is a lot faster than before. I guess that restic does not check all the local files on the restored_files/ directory. I understand that this is the desired behaviour.

# time ./restic_v0.16.5-811-g8e27a934d_linux_amd64 -r repo/ restore latest --target empty_dir/ --dry-run
enter password for repository:
repository 63b5531a opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
restoring snapshot 3b8dc776 of [/root/kernel_source] at 2024-07-09 07:35:38.59864869 +0200 CEST by root@host to /root/empty_dir/
Summary: Restored 89834 files/dirs (1.324 GiB) in 0:06

real	0m11.320s
user	0m8.144s
sys	0m0.372s

Another option is to restore pointing to /dev/null:

# time ./restic_v0.16.5-811-g8e27a934d_linux_amd64 -r repo/ restore latest --target /dev/null --dry-run
enter password for repository:
repository 63b5531a opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
restoring snapshot 3b8dc776 of [/root/kernel_source] at 2024-07-09 07:35:38.59864869 +0200 CEST by root@host to /dev/null
Summary: Restored 89834 files/dirs (1.324 GiB) in 0:05

real	0m10.837s
user	0m7.908s
sys	0m0.452s

Maybe an addition to the docs explaining this behaviour would be helpful for other people that are on the same boat than me. The --dry-run option works very well and the results are fast, but if you only want to know the size of the restored content, it will be faster to output the restore to /dev/null, in that case restic will not check local files on the restore destination, I guess it would consume less resources as well because it does not use local IO to check files stat.

3 Likes

3 cheers for restore overwrite and delete!!! looooong overdue…can finally do a true rollback restore…

4 Likes

Amazing work! restic backup --skip-if-unchanged and the snapshot size on restic snapshots --compact will make two of my scripts obsolete, which is great! :tada:

8 Likes

I’ve opened a PR to add an extra paragraph to the documentation: restore: document how to speed up dry-run by MichaelEischer · Pull Request #4917 · restic/restic · GitHub . The alternative to specifying a non-existing directory is to set --overwrite never. That might be a bit easier to use than having to find a non-existing target directory.

2 Likes

Finally got around to testing the restore --overwrite feature. Unfortunately it didn’t go well.

I picked an ~900GB snapshot. I started a restore… then did a SIGQUIT. I then started another restore, and used --overwrite if-changed and --verify - I assumed anything with the wrong file size would be replaced. Well, a ton of files had “invalid file size” on the verification round.

So then I did --overwrite always --delete --verify and ran it on the same path. Unfortunately that also failed, and said it skipped files for some reason - I’d assume nothing would be skipped, since it would either be overwritten or deleted, right?

restic restore 78f0f48b --delete --overwrite always --verify -t /Volumes/8TB_Backup/pics/01-B4_IMAGEOPTIM && notify test restore done || alert test restore failed
repository 24344e18 opened (version 2, compression level max)
[0:00] 100.00%  69 / 69 index files loaded
restoring snapshot 78f0f48b of [/Volumes/Media] at 2022-09-19 11:21:27.181769 -0700 PDT by smitmark@Media to /Volumes/8TB_Backup/pics/01-B4_IMAGEOPTIM
Summary: Restored 5170 / 5315 files/dirs (284.901 GiB / 312.089 GiB) in 1:10:02, skipped 119867 files/dirs 533.726 GiB
verifying files in /Volumes/8TB_Backup/pics/01-B4_IMAGEOPTIM
ignoring error for /Volumes/8TB_Backup/pics/01-B4_IMAGEOPTIM/Volumes/Media/Pictures/!Automatic Upload/iPhone 13 Mini/2022-02-18 19-28-15.mov: Invalid file size for /Volumes/8TB_Backup/pics/01-B4_IMAGEOPTIM/Volumes/Media/Pictures/!Automatic Upload/iPhone 13 Mini/2022-02-18 19-28-15.mov: expected 1323355135, got 1294087012
(there were hundreds more, but it was all the same error)

In theory, --overwrite always --delete should have overwritten everything and not left anything that shouldn’t have been in there, right? It was the same snapshot, not a different one. Anyway, let me know if I should open a GitHub issue for this. Thanks!

Ps. Also, SIGINIT isn’t stopping this build of Restic, hence the SIGQUIT. I’m using the latest beta build from yesterday around 2pm PST.

3 Likes

I don’t see any size info if i use --compact. Just curious what you meant?
I too LOVE --skip-if-unchanged - it’s so perfect for a number of my use cases

It’s in the latest master! Or you can grab a beta build.

1 Like

So it is, my beta wasn’t up to date enough. Thanks!

1 Like

Thanks for the bug report :slight_smile: .

Verify errors are always a bug independent of the --overwrite variant. In this case, there was a bug that caused the restorer to restore blobs in the wrong place for larger files (roughly >25MB) if those were already partially restored before.

That should be fixed by restore: fix cancelation and partial updates of large files by MichaelEischer · Pull Request #4921 · restic/restic · GitHub along with the hang after SIGINT and the broken progress bar.

4 Likes

I’m about to perform a file share migration between hosts on Windows. I’ve done this a few times in my career, all long ago, and I believe the tools used in the past (for better or worse) were robocopy, RichCopy, SyncToy, etc.

Given that restic now supports backing up file attributes and SecurityDescriptors, I am wondering if anyone has any comments on whether it is a suitable tool for conducting the migration (backup from source host to restic repo, restore to new host). I recognize that using this method is somewhat less efficient because it involves having a third place for the data (the restic repo) but my thought was that it might allow me to conduct the migration in a few steps (allow people to continue to use the file share on the source while most of the data is migrated, i.e. backed up and restored, then backup the changes, then restore what are essentially deltas).

Does anybody have any thoughts on whether restic is suitable for this purpose? I’m sorta thinking out loud here and haven’t decided how I’ll do the migration yet.

Thanks!