Restic memory use during restore: v0.9.1 vs 0.9.5

Getting Help
1

Hello everyone,

Trying to restore a smallish dirtree (~92GB, ~220K files) from my really large restic backup.

First I tried with restic v0.9.1 which I already had installed in our FreeNAS server (the backup itself was made in another machine running MacOSX – hope this is not a problem):

# mkdir /REDACTED0
# cd !$
# restic --verbose --target=. --include="**/REDACTED1/REDACTED2" \ restore 706e249a`
   repository abb62ab4 opened successfully, password is correct

After about ~1h45, I had to abort the restore because it had sucked almost half of memory in the machine (15 of 32GB), which was impacting its ARC cache and starting to swap:

# ps auwx | grep restic
    root          86219   4.9 45.2 18819824 15134340  5  S+   13:45       10:43.64 restic --verbose --target=. --include=**/REDACTED1/REDACTED2 restore 706e249a
    root          86220   0.1  0.1    82372    26276  5  I    13:45        2:03.66 rclone serve restic --stdio --b2-hard-delete --drive-use-trash=false REDACTED1:REDACTED2 (rclone-v1.42_from_F)
    root          96530   0.0  0.0     6696     2412  8  S+   15:21        0:00.00 grep restic
# top | head -6
    last pid: 96533;  load averages:  0.88,  0.56,  0.53  up 35+02:21:26    15:21:37
    124 processes: 1 running, 122 sleeping, 1 zombie
    Mem: 11G Active, 277M Inact, 3807M Laundry, 15G Wired, 425M Free
    ARC: 11G Total, 1981M MFU, 7471M MRU, 18M Anon, 1210M Header, 791M Other
    7822M Compressed, 9749M Uncompressed, 1.25:1 Ratio
          Swap: 10G Total, 5373M Used, 4867M Free, 52% Inuse

Remembering some IRC talks I had a few months with @fd0, I then upgrade to the latest version (v0.9.5 straight from https://github.com/restic/restic/releases) and tried again; this time, it went apparently much worse, using up no less than 19GB of RAM in little over 15 minutes :

# date; ps auwx | grep restic | grep -v grep; top | head -7 | grep -v '^$'
    Thu May 16 16:05:12 -03 2019
    root         99640  14.3 58.1 23464764 19448112  5  S+   15:49        7:13.23 restic --verbose --target=. --include=**/alex/financeiro restore 706e249a
    root         99641   0.0  0.1    75088    21272  5  S    15:49        0:04.52 rclone serve restic --stdio --b2-hard-delete --drive-use-trash=false REDACTED1:REDACTED2 (rclone-v1.42_from_F)
    last pid:  1577;  load averages:  2.61,  2.34,  1.51  up 35+03:05:01    16:05:12
    126 processes: 1 running, 124 sleeping, 1 zombie
    Mem: 15G Active, 13M Inact, 3349M Laundry, 12G Wired, 582M Free
    ARC: 7939M Total, 3541M MFU, 2435M MRU, 21M Anon, 1150M Header, 793M Other
         4425M Compressed, 5754M Uncompressed, 1.30:1 Ratio
    Swap: 10G Total, 10G Used, 101M Free, 99% Inuse

So it seems that either the latest memory use optimizations in restic restore haven’t helped much (or even made the problem worse) or I am missing something…

Is trying to restore by using restic mount the best way to avoid having to get a machine with oodles of ram in order to restore a backup?

Thanks in advance for your help & suggestions,
– Durval.

2

AFAICT, the major rework of the restorer hasn’t been merged yet and @fd0 is still working on some memory improvements. I think I’ve seen a ‘memory optimization’ branch on Github but it seems to be gone now. :confused:

3

Hello @764287,

Sorry for the delay in responding to you; lots of ‘stuff’ going on here (both restic-related and otherwise).

I’m running this tree (ie PR#2195, “out-of-order-restore-no-progress”) trying to solve that other problem, and it doesn’t seem to do anything memory-wise to restic: In fact, I’m seeing even larger memory usage. As of now, not even 15mins into the restore, it’s using over 36GB(!):

ps auwx | grep restic | grep -v grep 
root             7497 100.0 54.1 40974204 36297964 s008  R+   11:35AM  21:35.51 restic_out-of-order-restore-no-progress --verbose --target=. --include=/alex/financeiro restore 706e249a
root             7498   0.0  0.1  4485680  43192 s008  S    11:35AM   0:00.62 rclone serve restic --stdio --b2-hard-delete --drive-use-trash=false REDACTED1:restic/REDACTED2

Reading the PR, I don’t see any mention of memory optimization; my impression is that its main objective is to optimize restore speed (I guess, even at the expense of using more memory, as seen above).

Do you agree, @764287 , or am I missing something?

I have it right here (got it for restic backup as asked by @fd0, to help him with some memory usage profiles – and I’m in fact still running it for backup). No sense running it for restore, as my main issue with restores right now is time and not memory.

Cheers,
– Durval.

4

Re: the PR memory use, I’ve posted a comment and have been following it up with @ifedorenko on github, if anyone is interested please see there.