I have been trying to go through all the documentation (it seems to be scattered and really incomplete), but I cannot seem to find a way to see how much storage my snapshots are taking. I can do this in borg (which I have been testing for a few days), and was hoping to utilize restic instead of a borg+rclone combo. Even the built in help for some of the commands is cryptic and incomprehensible.
Hi,
It would be great if you could be more specific regarding what is lacking in the documentation, and especially the commands help - what is cryptic and incomprehensible? If we donāt know what youāre reacting on, itās hard to improve it.
Thanks!
I also want to see snapshot āsizeā (or something related to size)
But at the same time I canāt define what does āsizeā meansā¦ It will be cool to see followed info somewhere:
- Amount of data that is referenced by snapshot (disk space needed to do ārestic restore IDā)
- Amount of data added to repository with this snapshot
- Amount of data that will be removed from repository in case of āforget & pruneā for this particular snapshot.
#2 probably can be calculated during backup, but itāll be outdated on first āforget+pruneā cycle.
Maybe some sort of ārestic duā command is needed. Also #1 can be calculated using ārestic mountā and then āduā on fuse mount point.
This is what I was thinking of:
It would be great if you could be more specific regarding what is lacking in the documentation
When I look at the documentation between Borg and Restic, it appears that Restic doesnāt offer a lot of the same functionality as Borg. To me, Restic seems like a good blend of Borg (dedup and archive) and rclone (cloud storage connection)
what is cryptic and incomprehensible?
If I issue restic --help I get
root@backup1:/ # restic --help
restic is a backup program which allows saving multiple revisions of files and
directories in an encrypted repository stored on different backends.
Usage:
restic [command]
Available Commands:
autocomplete generate shell autocompletion script
backup create a new backup of files and/or directories
cat print internal objects to stdout
check check the repository for errors
dump dump data structures
find find a file or directory
forget forget removes snapshots from the repository
help Help about any command
init initialize a new repository
key manage keys (passwords)
list list objects in the repository
ls list files in a snapshot
migrate apply migrations
mount mount the repository
prune remove unneeded data from the repository
rebuild-index build a new index file
restore extract the data from a snapshot
snapshots list all snapshots
tag modifies tags on snapshots
unlock remove locks other processes created
version print version information
Flags:
-h, --help help for restic
--json set output mode to JSON for commands that support it
--no-lock do not lock the repo, this allows some operations on read-only repos
-o, --option key=value set extended option (key=value, can be specified multiple times)
-p, --password-file string read the repository password from a file (default: $RESTIC_PASSWORD_FILE)
-q, --quiet do not output comprehensive progress report
-r, --repo string repository to backup to or restore from (default: $RESTIC_REPOSITORY)
Use "restic [command] --help" for more information about a command.
yet, if I say restic list I get
root@backup1:/ # restic list
Fatal: type not specified
I gave it an invalid command, but instead of a --help screen, I see this error message. Part of that is me confusing borg commands and Restic commands, but on most any other program if I give it an invalid command it would list the possible commands and options for me (man page style)
Now if I issue: restic list --help:
root@backup1:/ # restic list --help
The "list" command allows listing objects in the repository based on type.
Usage:
restic list [blobs|packs|index|snapshots|keys|locks] [flags]
Flags:
-h, --help help for list
Global Flags:
--json set output mode to JSON for commands that support it
--no-lock do not lock the repo, this allows some operations on read-only repos
-o, --option key=value set extended option (key=value, can be specified multiple times)
-p, --password-file string read the repository password from a file (default: $RESTIC_PASSWORD_FILE)
-q, --quiet do not output comprehensive progress report
-r, --repo string repository to backup to or restore from (default: $RESTIC_REPOSITORY)
OK, but if I issue: restic list snapshots:
root@backup1:/ # restic list snapshots
Fatal: Please specify repository location (-r)
This seems to contradict what the documentation says:
restic -r /tmp/backup snapshots
so to me, the help screen and the documentation do not match. I donāt know what commands I am supposed to use (are there more commands not listed in documentation?). I have become quite confused by what commands I am supposed to issue and what the expected output should be. I only found this because I was trying to see if I could get archive/snapshot sizes to display.
Sorry for my ramblings, I was confused last night, and it still doesnāt make a whole lot of sense to me today.
This may be a nice improvement indeed.
But just tested with 2 first basics commands that came in my mind āls -zā and ācat -zā and this only report āinvalid option ā āzāā ā¦
ok, so here 109.40GB is the total space used on the remote storage ? And the value you were looking for with restic ?
I have not seen that option with restic (but Iām new). This may be a good improvement indeed.
The documentation looks ok to meā¦ but I think you are mixing 2 commands
- restic snapshots (to list snapshots)
- restic list (to list objects)
restic list [blobs|packs|index|snapshots|keys|locks] [flags]
snapshots is one of the objects
Hey, thanks for the feedback.
Thatās in my opinion a philosophical issue. Iām a bit annoyed sometimes by error messages that are printed at the top of a very long manpage/options list, so I need to scroll up and find the error. I actually prefer printing the error, and having the users (when they need help) issue restic cmd --help.
I agree that the documentation and especially the error messages can be improved (a lot). Part of the problem is that you stumbled upon the list command, which is actually used to list things in the repository on a very low level, it is basically not meant for end-users to use. The snapshots command is used to display information about snapshots in the repo.
The manual starts by explaining how to display help for commands and how to initialize a repository. Among others, this also explains the environment variable RESTIC_REPOSITORY, here: https://restic.readthedocs.io/en/latest/manual.html#initialize-a-repository
If you have an idea on how to improve the documentation, please let us know!
Adding more statistics is (among others) tracked in Print out the backup size when listing snapshots (enhancement) Ā· Issue #693 Ā· restic/restic Ā· GitHub.
2 Likes
Actually, I want to see the āDEDUPLICATEDā size because that is what I will transfer to my B2 bucket
What? It literally says that I can use restic list snapshots right thereā¦as well as blobs, packs, index, keys, and locks.
I am testing restic, borg, rclone, and any other deduplication backup/archive software I can right now. I have been using Crashplan for the last 6+ years and now they are cancelling all their home plans, so I have to find a replacement. I like the idea of something like restic or borg+rclone, with a Backblaze B2 storage, but I need to be cognizant of how much space I am taking up before I transfer it over my horrible internet connection, and I also need to know how much my backups will cost me monthly. I have several test repos that I am playing around with and I do not want to muck with ENV variables until I decide what option I am going to go with.
With Crashplan going away, I think we will see an uptick in software like Restic/etc being used, and I think most people will be interested in knowing how much space their backups are taking.
And you can do that. As I pointed out above, restic list is a low-level command. restic list snapshots will print a list of snapshot IDs, nothing more. On the other hand, restic snapshots prints a nice table with human-readable information. Thatās also described in the manual here: https://restic.readthedocs.io/en/stable/manual.html#list-all-snapshots
Right, I have seen the documentation. This is where I am confused. The documentation has commands backwards from how the man page describes the commands.
I guess what I am looking for doesnāt exist in Restic, and perhaps Restic is the wrong tool for me. From the onset, I saw it to be a replacement for borg + rclone combination, but I feel like I cannot get the information I need from it. The documentation page has a list of commands available, but doesnāt go into detail on how to use those commands or what they are for, and the man page has a different command structure than the documentation page. I have had no issues with learning how to use Borg, so I thought I wouldnāt have any issues with Restic, but I am just not understanding the layouts here.
Let me try to explain my frustration a little better.
https://restic.readthedocs.io/en/stable/manual.html#usage-help
From there, you can see the usage states:
restic [command]
then there is a list of commands and a list of flags, with no indication of when or where to use those flags or in what order (if it matters) in how those flags can be used.
In the list of commands, many are āundocumentedā on the documentation site, and the only way to see what they do is to use the help in the command line, with no indication of examples or structure.
When I read https://borgbackup.readthedocs.io/en/stable/usage.html#general I feel like the documentation is structured in a way to explain nearly EVERYTHING the program does without having to switch back and forth from a documentation page to a terminal.
Itās a layout and understanding issue on my part. I think you guys understand it more because you work with it all the time and know the ins and outs, where idiots like me are fumbling around trying to figure out how to utilize this program best.
Itās been a while since this post. Curious if there is still motivation for a borg-style summary of the original/deduplicated and total/snapshot sizes. Cloud storage costs money, and it sure is nice to see how much space is being taken up without having to call run the aws cli or equivalent after a backup.
Oh, and Iām fine with the docs, but maybe theyāve improved since this thead.
2 Likes
Use rclone size - it should support any and all restic backends. It doesnāt give you the size of an individual snapshot, but if what youāre worried about is how much your bill isā¦ it should be just what you need.
1 Like
Oh yes, there is! Iām currently working on the new archiver code (see #1494) which will allow this. Iām thinking of building something similar to what restic diff prints at the bottom, e.g.:
Files: 27 new, 82 removed, 3506 changed
Dirs: 0 new, 0 removed
Others: 0 new, 0 removed
Data Blobs: 1486 new, 1575 removed
Tree Blobs: 5237 new, 5237 removed
Added: 395.188 MiB
Removed: 393.600 MiB
3 Likes
Cool! Didnāt know about the diff command either
Thereās also a PR open to implement size counting into restic, feedback and contributions welcome: https://github.com/restic/restic/pull/1729 (thereās a lot of ways to count sizes!)
Just coming a bit late on this subject.
Wouldnāt it make sense to add a kind of a journal aside of the snapshot such as the detail of the backup command output and so on?
The backupPC logs are very neat with the Xfer logs, error logs, and statistics. On the restic case the xfer log might not be useful since there is already the ls command but the stats command is quite slow on big repos.
There is an issue open regarding the slowness of stats.
Is there currently any way to get the same kind of details for the list of snapshots as with the new backup output? I tried the new stats command but this only lists the total space used by all snapshots (if I used it correctly).
You can specify a snapshot-ID to show stats for a single snapshot:
bash:~$ restic stats a1234567 repository 98a9a987 opened successfully, password is correct scanning... Stats for a1234567 in restore-size mode: Total File Count: 12345 Total Size: 12.345 GiB
See restic stats --help for more help on this command.
2 Likes