Hi, I recently found that the download folder slipt into some of my backups, and I wanted to rewrite them to not have it anymore, though the restic rewrite fails with the message in the subject.
The ID printed is not reproducible across tries and changes:
$ RESTIC_COMPRESSION=max restic rewrite --exclude='Downloads/*'
repository 46211214 opened (version 2, compression level max)
snapshot 0a877585 of [/home/nmelzer] at 2023-05-20 01:00:13.880221453 +0200 CEST)
could not load snapshots: context canceled
Fatal: unable to rewrite snapshot ID "0a877585": cannot encode tree at "/" without loosing information
Tried that, and it seems as if that doesn’t work either:
$ RESTIC_COMPRESSION=max restic rewrite --exclude='/home/nmelzer/Downloads/*'
repository 46211214 opened (version 2, compression level max)
snapshot 08a5e216 of [/home/demo] at 2022-01-25 22:15:54.804224483 +0100 CET)
could not load snapshots: context canceled
Fatal: unable to rewrite snapshot ID "08a5e216": cannot encode tree at "/home/demo" without loosing information
$ RESTIC_COMPRESSION=max restic rewrite --exclude='/home/nmelzer/Downloads/*' --host mimas
repository 46211214 opened (version 2, compression level max)
snapshot 0a877585 of [/home/nmelzer] at 2023-05-20 01:00:13.880221453 +0200 CEST)
could not load snapshots: context canceled
Fatal: unable to rewrite snapshot ID "0a877585": cannot encode tree at "/" without loosing information
$ RESTIC_COMPRESSION=max restic rewrite --exclude='/home/nmelzer/Downloads' --host mimas
repository 46211214 opened (version 2, compression level max)
snapshot 0a877585 of [/home/nmelzer] at 2023-05-20 01:00:13.880221453 +0200 CEST)
could not load snapshots: context canceled
Fatal: unable to rewrite snapshot ID "0a877585": cannot encode tree at "/" without loosing information
And why isn’t in the first run the snapshot not just skipped, as it clearly does not contain any paths to rewrite…
Also, isn’t “loosing information” kind of what we expect from a rewrite?
Or might this be related to that fact that backups are actually created using rustic rather than restic and there is some metadata that restic can not properly move?
Add some test directory to your repo - then check if you can rewrite it. If no then there is repo problem, if yes then it would point to some “issue” in Downloads folder.
The snapshots on which the rewrite fail for me seems to be a snapshots that was copied from one repository to another.
snapshot a33a1f45 of [/home/user] at 2023-04-25 17:33:42.700107197 +0200 CEST)
could not load snapshots: context canceled
Fatal: unable to rewrite snapshot ID "a33a1f45": cannot encode tree at "/" without loosing information
restic -r sftp:someRemoteRepo snapshots a33a1f45 --json
enter password for repository:
[{"time":"2023-04-25T17:33:42.700107197+02:00","tree":"984af441a9e5de8de4eb27ab9b9a39c3fe38d57c597a9aad42585d4d3f304d18","paths":["/home/user"],"hostname":"<anotherHost>","original":"1c93a52c6a8359c6ccc3b58dd1ee11d60cd1e46f4b1b69ae90f5f9198954eafb","id":"a33a1f4591ea5864398b17fad185cf77d290eca18ef717a1f7ab47e80d8d9f81","short_id":"a33a1f45"}]
edit:
Even running the command restic -r sftp:someRemoteRepo rewrite --dry-run --exclude-file /tmp/exclude a33a1f45 with an empty excludeFile fails
That is likely the cause. rewrite verifies that saving the unmodified tree metadata would exactly recreate the existing metadata. That ensures that there are no unexpected changes to a snapshot during a rewrite. In restic 0.16.0 (or using a beta build) restic repair snapshots will be able to fix the problem.
@wnklmnn What is the output of restic cat blob 984af441a9e5de8de4eb27ab9b9a39c3fe38d57c597a9aad42585d4d3f304d18? The problems with rewriting the snapshot show up while processing that (tree) blob.
I encountered the same problem, these snapshots were backed up by rustic.
After running the command “restic repair snapshots”, it was restored normally in restic, but rustic reported an error:
error: deserializing from bytes of JSON Text failed: Error("missing field total_dirs_processed`", line: 1, column: 923)
This about 1TB repository is now damaged and unusable for rustic.
@MichaelEischer This also ensures that restic rewrite is basically never able to rewrite unmodified trees generated by another program different from restic itself. This is because of:
The specification in restic/doc/design.rst at master · restic/restic · GitHub about how tree jsons are generated is not very precise - some of the unclear points are: How to exactly encode non-unicode filenames? When to use null or omit non-existing fields? What are the exact rules to serialize things like dates? Moreover there is no “full” specification which fields are generally allowed and under which circumstances they are to be filled.
By comparing serialized trees, you also check that the order of elements in the generated json is exactly as restic produces it - this is again not specified but also against the principles of a named serialized format like JSON…
May I suggest that you either change the way how you do this check or add a note somewhere about this rewrite restriction?
The deduplication relies on having a perfectly reproducible serialization of tree blobs. So some defined order is necessary, which for the restic repository format happens to be the order defined by restic.
The check is there to ensure that when using rewrite with an older restic version that is does not silently drop attributes only known by a later restic version.
Yes, you are right about the deduplication. Wouldn’t that make a more precise documentation of the format even more valuable/necessary?
Or did restic switch philosophy from “the repo format specification is leading and restic is just an implementation” to “the implementation defines the specification”?
Yes, I know the intention of that check (and note that it also covers extensions which were added by some other tools writing the restic repository). Just wanted to suggest that the restriction coming with how it is currently implemented should be either overthought or at least documented.
By the way: The treatment of parent trees is actually quite similar. Does restic also perform this kind of check for the backup command (if all tree entries are not changed, check if the serialized tree matches the parent one and if not stop with some “you might be loosing information” error?)
Feel free to help with documenting it. However, at some point the necessary level of detail in the documentation would be so high that it’s much more reasonable to just look at the code. That doesn’t make the existing documentation pointless. In particular, the high level perspective would be very hard to piece together from the code.
I don’t get why that should make sense. All that seems to achieve is causing failed backups.
It is possible to convert snapshots back into the canonical encoding using restic repair snapshots <IDs...>. I guess we can document that specific error somewhere.
There was no tonality at all intended and also no intend to misquote anything. The only thing I was having in mind was that the specification would be the leading document and everything else is just an implementation of it. And that the fact that it could be used or re-implemented is one of the strengths of the format.
Now I have to admit that I can no longer remember how I came to this assumption. And reading your answers I could be just simply wrong. So a “you are wrong here” would be perfectly fine for me, actually.
Yes. And what the current rewrite check seems to achieves is failing rewriting snapshots
I’ll try to explain the analogy in more detail: The problem with rewriting snapshots is not that any information is lost when writing a new snapshot. The problem is that typically users do want to remove old snapshots once they are rewritten. And yes, this could lead to the situation that information in trees of the old snapshot is not reproduces during the rewrite. At least this is how I understood the intention of that check.
Now with backup using parents, it is exactly the same: backup could detect that all entries of a tree are unchanged w.r.t. the parent, so the tree (of the parent snapshot) should be also unchanged. But it could also detect that the tree it is going to generate in fact is not identical to the parent one. If the new snapshot is simply generated from the new tree, users are loosing information once the parent snapshot is removed… Additionally we also do have the deduplication argument when running backup with a parent snapshot.
TL;DR: In this situation (rewrite and backup with parent), there are 3 options how to handle new identical trees: 1) use the existing tree 2) write the new tree 3) quit if existing and new tree are not identical. IMO it is inconsequent to handle rewrite and backup differently. Also, IMO the third option is the worst one.
).
