Minimalist output when used from scheduler

Getting Help
1

Hello,

I’m starting restic from a scheduler and so by default it does not output any progress information, as explained in the manual.
I thus added the RESTIC_PROGRESS_FPS environment variable with a value of 0.0016666 to get one progress information sent every ten minutes.
However, those progress detail also include the current list of files that are being processed a the time of output, which I find not desirable.

For example, here is the amount of lines that I get each time progress is output:

[10:00] 6.87% 75293 files 50.084 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:14:23
/files/download/Divers/LibreOffice_7.6.4_Win_x86-64.msi
/files/download/Electronique/MPLABX-v5.25-windows-installer.exe
/files/download/Electronique/com-microchip-mcc-v5.5.0.zip
/files/download/Electronique/kicad-8.0.7-x86_64.exe
/files/download/Electronique/mcc-installer-5.5.0-windows.exe
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/LICENSE.txt
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/esm/bits.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/esm/bufs.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/esm/index.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/esm/leb.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/lib/bits.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/lib/bufs.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/lib/index.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/lib/leb.js
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/leb128/package.json
/files/download/Linux/kibana/pkg/kibana-withnode/usr/share/kibana/node_modules/@webassemblyjs/utf8/LICENSE

I know this is because I’m using --read-concurrency 16 but that value is the one that I find is giving me the best performance for this backup.

What I’d like to have is just the first line, the one that gives global progress information, so that in the end, I only have those kind of lines:

[10:00] 6.87% 75293 files 50.084 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:14:23
[20:00] 13.61% 85461 files 99.177 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:11:18
[30:00] 19.94% 85722 files 145.370 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:00:49
[40:00] 26.18% 85735 files 190.794 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:10:33
[50:00] 31.56% 85764 files 230.022 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:03:45
[1:00:00] 36.83% 85786 files 268.483 GiB, total 368997 files 728.902 GiB, 0 errors ETA 2:01:39
[1:10:00] 42.17% 85971 files 307.366 GiB, total 368997 files 728.902 GiB, 0 errors ETA 1:41:07
[1:20:00] 47.54% 86062 files 346.539 GiB, total 368997 files 728.902 GiB, 0 errors ETA 1:40:54
[1:30:00] 52.57% 86097 files 383.178 GiB, total 368997 files 728.902 GiB, 0 errors ETA 1:34:53
[1:40:00] 57.75% 123250 files 420.970 GiB, total 368997 files 728.902 GiB, 0 errors ETA 1:16:48
[1:50:00] 62.98% 285948 files 459.041 GiB, total 368997 files 728.902 GiB, 0 errors ETA 56:52
[2:00:00] 69.48% 330022 files 506.419 GiB, total 368997 files 728.902 GiB, 0 errors ETA 50:18
[2:10:00] 74.91% 360398 files 546.010 GiB, total 368997 files 728.902 GiB, 0 errors ETA 45:20
[2:20:00] 80.50% 360521 files 586.746 GiB, total 368997 files 728.902 GiB, 0 errors ETA 34:33
[2:30:00] 94.51% 368984 files 688.859 GiB, total 368997 files 728.902 GiB, 3 errors ETA 8:45
Files: 5 new, 360863 changed, 8126 unmodified
Dirs: 0 new, 52672 changed, 1044 unmodified
Added to the repository: 216.745 MiB (79.914 MiB stored)
processed 368994 files, 728.902 GiB in 2:37:05
snapshot fe64adda saved

I looked in the documentation but could not find a way to get this. Is this possible?

4

@obones welcome to the restic forum.

Can you provide some more details on your environment and how your run restic from the scheduler?

Then, how do you connect to the scheduler output to watch the progress?

Last, what is your goal with restic that you want to monitor its output from a scheduled job?

p.s. I’d consider a normal workflow to first interactively run restic from the commandline and then redirect output logs and/or errors to a logfile for use in a scheduler.

5

Thanks for your reply, here are a few more details

I’m running restic via a command line given to the scheduler, along those lines:

restic -r s3:https://$(b2_bucket_server)/$(BucketName) --cache-dir /home/$(BackupSystemUser)/.cache/restic backup --pack-size 100 --read-concurrency 16 --exclude-caches $(BackupOptions) $(SourceDirs)

The standard output is captured by the scheduler that sends it to me in an email once the command has ended.
I could use its web based interface, but that would show me quite the same thing as in the email.
There is no need for me to watch “live” progress, I really only need the morning email giving me the result of the night’s run.

My need here is to have the output in the email not be too long, while being useful as well. In particular, I like the fact that with the desired output shown above, I would get a good idea where the most time is spent.

Note that I can live with the current state of things, it’s just that currently the email is way longer than I need it to be.

I hope this clarifies things.

6

I am not familiar with the RESTIC_PROGRESS_FPS environment option but I use --verbose=2 to get all the output and then use grep to filter out what I do not want. If you are using a Linux system, you could maybe do something like this.

restic -r s3:https://$(b2_bucket_server)/$(BucketName) --cache-dir /home/$(BackupSystemUser)/.cache/restic backup --pack-size 100 --read-concurrency 16 --exclude-caches $(BackupOptions) $(SourceDirs) | grep -v "^/files/"

That might (I did not test it) remove all the lines that start with /files/. If you back of several top level directories you would have to add one for each.

Here is something else to think about. Is your future backups so large that they require incremental progress updates? The first run may take a long time but after that it only has to save changed files. It looks like your system is processing about 50 GB every 10 minuets, so I would think your incremental backups would need to be much larger then that to make these updates interesting.

7

Thanks, I can indeed filter out the lines with grep, keeping only those starting with \[[0-9]+

You are right that the second backup should be near instant, but it is not, it tells me that lots of files changed:

Files: 24 new, 360870 changed, 8126 unmodified

I have already identified that the content has not changed, it’s only the ctime that did. There is something that changes the file rights on those folders, but I believe it’s too blunt in its way of doing things in that it calls chmod event if the rights are already the expected one.

8

@obones you could consider to have restic ignore ctime changes and see if that speeds up the process if file content and not the latest rights are most important for you:

--ignore-ctime

9

Thanks.
I read about that in the command line help, but it was not clear to me if it would trigger a full content scan or not, which I believe would be unproductive in my case.
But on that page, it makes it quite clear that it would not, so I’ll try it.

10

Sure enough, that --ignore-ctime flag was the one I needed to use, thanks for the pointer