I have been trying to go through all the documentation (it seems to be scattered and really incomplete), but I cannot seem to find a way to see how much storage my snapshots are taking. I can do this in borg (which I have been testing for a few days), and was hoping to utilize restic instead of a borg+rclone combo. Even the built in help for some of the commands is cryptic and incomprehensible.
It would be great if you could be more specific regarding what is lacking in the documentation, and especially the commands help - what is cryptic and incomprehensible? If we don’t know what you’re reacting on, it’s hard to improve it.
I also want to see snapshot ‘size’ (or something related to size)
But at the same time I can’t define what does ‘size’ means… It will be cool to see followed info somewhere:
- Amount of data that is referenced by snapshot (disk space needed to do ‘restic restore ID’)
- Amount of data added to repository with this snapshot
- Amount of data that will be removed from repository in case of ‘forget & prune’ for this particular snapshot.
#2 probably can be calculated during backup, but it’ll be outdated on first ‘forget+prune’ cycle.
Maybe some sort of ‘restic du’ command is needed. Also #1 can be calculated using ‘restic mount’ and then ‘du’ on fuse mount point.
This is what I was thinking of:
It would be great if you could be more specific regarding what is lacking in the documentation
When I look at the documentation between Borg and Restic, it appears that Restic doesn’t offer a lot of the same functionality as Borg. To me, Restic seems like a good blend of Borg (dedup and archive) and rclone (cloud storage connection)
what is cryptic and incomprehensible?
If I issue
restic --help I get
root@backup1:/ # restic --help restic is a backup program which allows saving multiple revisions of files and directories in an encrypted repository stored on different backends. Usage: restic [command] Available Commands: autocomplete generate shell autocompletion script backup create a new backup of files and/or directories cat print internal objects to stdout check check the repository for errors dump dump data structures find find a file or directory forget forget removes snapshots from the repository help Help about any command init initialize a new repository key manage keys (passwords) list list objects in the repository ls list files in a snapshot migrate apply migrations mount mount the repository prune remove unneeded data from the repository rebuild-index build a new index file restore extract the data from a snapshot snapshots list all snapshots tag modifies tags on snapshots unlock remove locks other processes created version print version information Flags: -h, --help help for restic --json set output mode to JSON for commands that support it --no-lock do not lock the repo, this allows some operations on read-only repos -o, --option key=value set extended option (key=value, can be specified multiple times) -p, --password-file string read the repository password from a file (default: $RESTIC_PASSWORD_FILE) -q, --quiet do not output comprehensive progress report -r, --repo string repository to backup to or restore from (default: $RESTIC_REPOSITORY) Use "restic [command] --help" for more information about a command.
yet, if I say
restic list I get
root@backup1:/ # restic list Fatal: type not specified
I gave it an invalid command, but instead of a --help screen, I see this error message. Part of that is me confusing borg commands and Restic commands, but on most any other program if I give it an invalid command it would list the possible commands and options for me (man page style)
Now if I issue:
restic list --help:
root@backup1:/ # restic list --help The "list" command allows listing objects in the repository based on type. Usage: restic list [blobs|packs|index|snapshots|keys|locks] [flags] Flags: -h, --help help for list Global Flags: --json set output mode to JSON for commands that support it --no-lock do not lock the repo, this allows some operations on read-only repos -o, --option key=value set extended option (key=value, can be specified multiple times) -p, --password-file string read the repository password from a file (default: $RESTIC_PASSWORD_FILE) -q, --quiet do not output comprehensive progress report -r, --repo string repository to backup to or restore from (default: $RESTIC_REPOSITORY)
OK, but if I issue:
restic list snapshots:
root@backup1:/ # restic list snapshots Fatal: Please specify repository location (-r)
This seems to contradict what the documentation says:
restic -r /tmp/backup snapshots
so to me, the help screen and the documentation do not match. I don’t know what commands I am supposed to use (are there more commands not listed in documentation?). I have become quite confused by what commands I am supposed to issue and what the expected output should be. I only found this because I was trying to see if I could get archive/snapshot sizes to display.
Sorry for my ramblings, I was confused last night, and it still doesn’t make a whole lot of sense to me today.
This may be a nice improvement indeed.
But just tested with 2 first basics commands that came in my mind “ls -z” and “cat -z” and this only report “invalid option – ‘z’” …
ok, so here 109.40GB is the total space used on the remote storage ? And the value you were looking for with restic ?
I have not seen that option with restic (but I’m new). This may be a good improvement indeed.
The documentation looks ok to me… but I think you are mixing 2 commands
- restic snapshots (to list snapshots)
- restic list (to list objects)
restic list [blobs|packs|index|snapshots|keys|locks] [flags]
snapshots is one of the objects
Hey, thanks for the feedback.
That’s in my opinion a philosophical issue. I’m a bit annoyed sometimes by error messages that are printed at the top of a very long manpage/options list, so I need to scroll up and find the error. I actually prefer printing the error, and having the users (when they need help) issue
restic cmd --help.
I agree that the documentation and especially the error messages can be improved (a lot). Part of the problem is that you stumbled upon the
list command, which is actually used to list things in the repository on a very low level, it is basically not meant for end-users to use. The
snapshots command is used to display information about snapshots in the repo.
The manual starts by explaining how to display help for commands and how to initialize a repository. Among others, this also explains the environment variable
RESTIC_REPOSITORY, here: https://restic.readthedocs.io/en/latest/manual.html#initialize-a-repository
If you have an idea on how to improve the documentation, please let us know!
Adding more statistics is (among others) tracked in https://github.com/restic/restic/issues/693.
Actually, I want to see the ‘DEDUPLICATED’ size because that is what I will transfer to my B2 bucket
What? It literally says that I can use restic list snapshots right there…as well as blobs, packs, index, keys, and locks.
I am testing restic, borg, rclone, and any other deduplication backup/archive software I can right now. I have been using Crashplan for the last 6+ years and now they are cancelling all their home plans, so I have to find a replacement. I like the idea of something like restic or borg+rclone, with a Backblaze B2 storage, but I need to be cognizant of how much space I am taking up before I transfer it over my horrible internet connection, and I also need to know how much my backups will cost me monthly. I have several test repos that I am playing around with and I do not want to muck with ENV variables until I decide what option I am going to go with.
With Crashplan going away, I think we will see an uptick in software like Restic/etc being used, and I think most people will be interested in knowing how much space their backups are taking.
And you can do that. As I pointed out above,
restic list is a low-level command.
restic list snapshots will print a list of snapshot IDs, nothing more. On the other hand,
restic snapshots prints a nice table with human-readable information. That’s also described in the manual here: https://restic.readthedocs.io/en/stable/manual.html#list-all-snapshots
Right, I have seen the documentation. This is where I am confused. The documentation has commands backwards from how the man page describes the commands.
I guess what I am looking for doesn’t exist in Restic, and perhaps Restic is the wrong tool for me. From the onset, I saw it to be a replacement for borg + rclone combination, but I feel like I cannot get the information I need from it. The documentation page has a list of commands available, but doesn’t go into detail on how to use those commands or what they are for, and the man page has a different command structure than the documentation page. I have had no issues with learning how to use Borg, so I thought I wouldn’t have any issues with Restic, but I am just not understanding the layouts here.
Let me try to explain my frustration a little better.
From there, you can see the usage states:
then there is a list of commands and a list of flags, with no indication of when or where to use those flags or in what order (if it matters) in how those flags can be used.
In the list of commands, many are ‘undocumented’ on the documentation site, and the only way to see what they do is to use the help in the command line, with no indication of examples or structure.
When I read https://borgbackup.readthedocs.io/en/stable/usage.html#general I feel like the documentation is structured in a way to explain nearly EVERYTHING the program does without having to switch back and forth from a documentation page to a terminal.
It’s a layout and understanding issue on my part. I think you guys understand it more because you work with it all the time and know the ins and outs, where idiots like me are fumbling around trying to figure out how to utilize this program best.
It’s been a while since this post. Curious if there is still motivation for a borg-style summary of the original/deduplicated and total/snapshot sizes. Cloud storage costs money, and it sure is nice to see how much space is being taken up without having to call run the aws cli or equivalent after a backup.
Oh, and I’m fine with the docs, but maybe they’ve improved since this thead.
rclone size - it should support any and all restic backends. It doesn’t give you the size of an individual snapshot, but if what you’re worried about is how much your bill is… it should be just what you need.
Oh yes, there is! I’m currently working on the new archiver code (see #1494) which will allow this. I’m thinking of building something similar to what
restic diff prints at the bottom, e.g.:
Files: 27 new, 82 removed, 3506 changed Dirs: 0 new, 0 removed Others: 0 new, 0 removed Data Blobs: 1486 new, 1575 removed Tree Blobs: 5237 new, 5237 removed Added: 395.188 MiB Removed: 393.600 MiB
Cool! Didn’t know about the diff command either
There’s also a PR open to implement size counting into restic, feedback and contributions welcome: https://github.com/restic/restic/pull/1729 (there’s a lot of ways to count sizes!)
Just coming a bit late on this subject.
Wouldn’t it make sense to add a kind of a journal aside of the snapshot such as the detail of the
backup command output and so on?
backupPC logs are very neat with the Xfer logs, error logs, and statistics. On the restic case the xfer log might not be useful since there is already the
ls command but the
stats command is quite slow on big repos.
There is an issue open regarding the slowness of stats.