Out of Memory running `check`

Running a ‘check’ on one machine is causing:

runtime: out of memory: cannot allocate 541065216-byte block (2180055040 in use)
fatal error: out of memory

From /proc/meminfo
MemTotal: 8088308 kB
MemFree: 3047928 kB
MemAvailable: 7146584 kB

The repo checks fine from other machines.

This is restic v0.11.0 running on a Pi. It’s been running fine for many months. One recent changes I’ve added my MacBook Pro to the repo, and it is running v0.12.1. Could this be the source of the problem? Any particular suggestions to make it work?

Please upgrade to restic 0.15.1 which you can either get by running restic self-update or by downloading it from here: Releases · restic/restic · GitHub

It contains a lot of improvements over restic 0.11.0, so even if the latter has worked for you, it’s a really good idea to upgrade :slight_smile:

Another thing you can do if the problem persists after upgrade to restic 0.15.1 is to run it with
GOGC=20, e.g. GOGC=20 restic ....

To answer your question; Yes, adding more data to the repository naturally means more metadata and more resource usage.

Yes more metadata, but that doesn’t inherently mean vastly more RAM usage. I’m kind of surprised to see it allocating stuff in the multiple GB range. Maybe I don’t understand it well enough, but I’d expect repositories with dozens if not hundreds of hosts, and that the algorithm’s required resources don’t scale linearly.

GOGC=20 helped. Thank you.

You’re welcome. Regarding memory usage and such, it all comes down to what the repository contains, obviously.

They do with respect to memory usage. The point is that restic has to keep an index of all blobs saved in the repository in order to know if (and where) a blob is saved or not. The blob size is limited and the index is saved in memory, so memory usage scales linearly with the repository size.