Backend Storage Space Usage

Getting Help
1

First of all, thanks for building restic.

Exponential Increase in Backup Storage
We are using restic with backend AWS S3. I notice that every time backup runs, the storage space increases by current server space, though it is incremental backup.
The first backup stored 30GB. The second the time the command was run, the storage space used is 60GB, even though very few files were added, less then few KBs.

The command I use to schedule backups:-
/usr/bin/ionice -c2 -n0 restic --quiet --host servername backup / --exclude=*.log

I was expecting only increment data to add up to the storage. Am I doing something wrong?

We are currently using restic_0.9.4_linux_amd64 binary.

Backup Speed
Also the backup takes a lot of time. In another case, I saw first backup of 17GB on VPS take 32 hrs.
The storage and server should be like 50-70ms apart. I was running with nice -n19.
I am currently testing without “nice”.

Next, I may try with other options of ionice. Anyone tested with ionice can make any recommendations?

2

Welcome to the forum!

From what you describe it sounds like restic includes its own cache directory in the backup, which nicely explains the exponential space increase you see. It should not do that, but can you please check it?

3

I missed to mention one point in restic command.

/usr/bin/ionice -c2 -n0 restic --quiet --host servername backup / --exclude=*.log --cache-dir /var/cache/.bountysite/.cache

I am using --cache-dir in the options. Is this causing the issue?

4

I have been experimenting with different options. Lost a bit of track.
Sorry!

Here is the current option, which is still causing the issue.

/usr/bin/ionice -c2 -n0 /usr/local/bountysite/bin/restic_0.9.4_linux_amd64 --quiet --host servername backup / --exclude=*.log --exclude=/bin --exclude=/boot --exclude=/dev --exclude=/lib --exclude=/lib64 --exclude=/media --exclude=/proc --exclude=/run --exclude=/srv --exclude=/sys --exclude=/usr --exclude=/var/ --cache-dir=/var/cache/.bountysite/.cache

–cache-dir is set, but the parent folder is excluded.
I checked the cache dir is only 42M.

Wondering if anyone is seeing the same storage consumption?

5

How are you measuring this?

6

I was just going to update with my findings.

I am using restic stats, which was showing this value. The actual space used is less. Sorry for the confusion.

The question now really is “should restic stats show actual disk space usage?”

7

The answer is “read the documentation.” :slight_smile:

$ restic help stats

The "stats" command walks one or all snapshots in a repository and
accumulates statistics about the data stored therein. It reports on
the number of unique files and their sizes, according to one of
the counting modes as given by the --mode flag.

If no snapshot is specified, all snapshots will be considered. Some
modes make more sense over just a single snapshot, while others
are useful across all snapshots, depending on what you are trying
to calculate.

The modes are:

* restore-size: (default) Counts the size of the restored files.
* files-by-contents: Counts total size of files, where a file is
   considered unique if it has unique contents.
* raw-data: Counts the size of blobs in the repository, regardless of
  how many files reference them.
* blobs-per-file: A combination of files-by-contents and raw-data.

Refer to the online manual for more details about each mode.

It sounds like you expected the mode raw-data.

8

Yeah! Thank you. :grimacing:

I did not even check the actual disk usage to confirm, in the first place. I thought I was doing something wrong and went on with that assumption.

9

I did come across these options, but could not make out that option raw-data does the job.