Restic prune/check reliably OOM-crashing 8GB TrueNAS box

Getting Help
1

This is my first post and I apologize if it’s rambling or otherwise wrong.

restic version:

restic 0.19.1 compiled with go1.24.x on linux/amd64

(Originally on 0.16.4; upgraded mid-troubleshooting — see below.)

Environment variables:

RCLONE_CONFIG=/mnt/tank/backups/secrets/rclone.conf
RESTIC_CACHE_DIR=/mnt/tank/backups/cache/restic

Environment:

  • TrueNAS SCALE, 8GB RAM (Intel i3-1315U)
  • Backend: Backblaze B2 via rclone
  • Repository: 132.87 GiB, 8,187 objects, 244 snapshots (spans two hosts sharing one bucket — see note below)

The command that fails:

restic -r rclone:b2backup:andy-restic-backup/20260109-restic \
       --password-file /mnt/tank/backups/secrets/restic_pw \
       forget \
       --host nas-dxp4800pro \
       --keep-last 4 --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 \
       --prune

What happens:
The kernel OOM-killer fires — not narrowly against restic, but a cascading sweep that killed smbd, dockerd, winbindd, syncthing, even systemd-journal and systemd-logind in one incident, forcing a reboot. Relevant excerpt from journalctl -b -1 -p err:

Jul 05 22:52:19 truenas kernel: Out of memory: Killed process 3034 (cli) total-vm:484616kB, anon-rss:104564kB...
Jul 05 22:52:19 truenas kernel: Out of memory: Killed process 2320721 (syncthing) total-vm:1364756kB...
[... ~30 more processes killed in the same second ...]

Separately, restic check (same repo, no other args besides -r and --password-file) shows the same trajectory — during “check snapshots, trees and blobs,” available memory declines steadily (~30 MiB per snapshot processed in one observed run), clearly heading toward exhaustion before completing all 244. I aborted that run proactively rather than let it crash, so I can’t share a completed failure log for check specifically — only the trend.

Repo detail worth flagging: this repository holds snapshots from two different hosts sharing one B2 bucket — my TrueNAS NAS (nas-dxp4800pro, ~10 snapshots, 4 datasets) and a separate desktop workstation (tuxedo-os, the bulk of the 244 snapshots, backing up /home/andy). I hadn’t fully registered this myself until digging in — relevant to the “why is this repo so expensive for its size” question below.

What I tried on the restic side

  1. Version upgrade (0.16.4 → 0.19.1, which per the changelog includes ~60% prune memory reduction). Helped somewhat — got further before hitting the wall — but same eventual crash.
  2. ZFS ARC cap — reduced arc_max from ~7GiB to 3GiB via a TrueNAS tunable. ARC correctly shrank under pressure as expected, but had no effect on the OOM outcome. Ruled out.
  3. Cache directory — initially suspected --cache-dir/RESTIC_CACHE_DIR defaulting to tmpfs (RAM-backed on TrueNAS SCALE) as a contributor, based on manual ad-hoc testing without the env var set. My actual production script already had RESTIC_CACHE_DIR pointed at a real disk-backed path the whole time (shown above) — so this was a red herring from my own testing methodology, not a factor in the real crashes.
  4. Prune tuning (--max-repack-size 2G, --max-unused 10%) — added to bound per-run repack work. Worth keeping, but based on what I’ve read since, these mainly affect duration/IO, not the peak memory needed for the underlying index-loading/blob-tracking step.

What actually worked: rustic

Tested rustic 0.11.2 against the identical repository, same hardware, no re-initialization — pointed straight at the existing restic-format repo:

rustic --password-file /mnt/tank/backups/secrets/restic_pw \
       --cache-dir /mnt/tank/backups/restic-cache \
       -r rclone:b2backup:andy-restic-backup/20260109-restic \
       check

Completed successfully across all 244 snapshots. Heaviest phase (“checking trees”) took ~13-14 minutes; available memory oscillated down to ~180-200MB at its worst but consistently recovered.

rustic --password-file ... --cache-dir ... -r ... \
       forget --filter-host nas-dxp4800pro \
       --keep-last 4 --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 \
       --prune

Dry-run and real run both completed successfully (316 packs repacked, 8 deleted, 132.2GB remaining), surviving several transient B2 500 errors via normal retry logic. Post-prune check confirmed repo integrity.

Questions

  1. Is this OOM ceiling expected/known for restic at this repo size (130GB / 244 snapshots) on 8GB RAM, or does it point to something misconfigured on my end?
  2. Given the repo spans two hosts sharing dedup space — does splitting by host meaningfully reduce per-run memory, or does cross-host dedup reduce that benefit?
  3. Anyone running rustic long-term against a repo originally created by restic, keeping restic for routine backups and rustic only for check/prune? Any gotchas given they claim full format compatibility?
  4. Is there a known memory ceiling for rustic too at larger scale, or does its architecture genuinely sidestep this class of problem?

Happy to provide further logs/output if useful.

2

Hi @nash, rustic author here :waving_hand:

About your questions:

The memory needed for prune is mainly based on the number of blobs in you repository. As you said, the basic steps in prune are:

  • read the repository index (i.e. information about every blob stored in the repo)
  • check all snapshots/trees for used/unused blobs
  • remove or repack pack files to reduce the number of unused blobs and free space

For the memory consumption the first step is the main contributor.

If the repository contains too many blobs to store in-memory, you’ll receive memory errors - this is true for both restic and rustic. This does not depend on the total repo size and not too much on the number of packs in the repository, but mainly on the number of blobs in the repository. You can run rustic repoinfo to get a detailed view on this.

Given the repo spans two hosts sharing dedup space — does splitting by host meaningfully reduce per-run memory, or does cross-host dedup reduce that benefit?

As written above, if spliting into two repository by host reduces the number of blobs, it reduces the memory usage. The better the deduplication the less additional extra blobs you get from the second host and the less you gain in memory usage.

Anyone running rustic long-term against a repo originally created by restic, keeping restic for routine backups and rustic only for check/prune? Any gotchas given they claim full format compatibility?

I think I would give biased answer here. Just keep in mind that rustic does not implement/use locking for the prune step and you must not you run a resic prune in parallel. If you want some kind of locking, you have to implement this yourself and then you can use --instant-removewhich would make rustic behave like restic (despite of the locking).

Is there a known memory ceiling for rustic too at larger scale, or does its architecture genuinely sidestep this class of problem?

There is in general the same problem with rustic as the algorithm is basically very identical. The differences are how the internal data structures are desigend and how the language operates. restic uses Go (with garbage-collected memory structures) and a self-implemented hashmap (IIRC) for the index. You can try to tune the Go garbage collector (there is some GO_*** env variable) which will likely reduce memory here. rustic uses Rust (which deallocated directly after usage) and some memory-optimized internal data structures which have less memory overhead than a hashmap. I think this makes the memory usage lower than restic`s, but it’s more like a constant factor and this cannot solve the fundamental problem that each blob information has to be stored in-memory to decide what to do.

3

Thanks, this is the clarity I needed — really appreciate you taking the time, especially given who’s answering.

Ran rustic repoinfo as suggested. Results:

| Blob type      |   Count | Total Size |
|----------------|---------|------------|
| Tree           |  106467 |   72.8 GiB |
| Data           | 8172979 |  181.9 GiB |
| Tree to delete |   25937 |  729.0 MiB |
| Data to delete |   19690 |    4.4 GiB |
| Total          | 8325073 |  259.8 GiB |

~8.3 million blobs total, average blob size ~16.5KB — so this is clearly a small-files-dominated repo rather than a large-files one. That tracks: the repo actually spans two hosts (my NAS’s 4 datasets, ~10 snapshots, and my desktop’s /home/nash, the bulk of 244 snapshots) — I’d bet the desktop side (dotfiles, repos, project directories, TiddlyWiki instances) is contributing the majority of that blob count.

Given your answer that splitting reduces memory roughly in proportion to how little dedup benefit is lost, this points pretty clearly at splitting by host as the real fix for me — since a desktop /home and a NAS’s storage pools almost certainly share very little at the file level, so I’d expect to lose little dedup benefit while meaningfully cutting blob count on each side.

Also noted your locking warning re: rustic prune having no built-in locking — appreciated, will make sure restic and rustic are never run concurrently against the same repo, and will plan around that explicitly once/if I settle into a hybrid setup.

This has been a really useful thread to work through — thanks again.

One more thing I’d like to sanity-check before I finalize a plan.

Given the blob-count-driven memory model you described, and that I’m now planning to split this repo by host (NAS vs. desktop) to cut blob count per-repo — is a modest RAM upgrade on the NAS (currently 8GB, could go to 16 or 24GB via a second SO-DIMM) still worth doing as extra headroom, or would the repo split alone likely bring my per-repo blob count comfortably within what 8GB can handle?

Trying to figure out whether to spend on the RAM now, or hold off and see how things look after the split first. If it’s helpful — is there a rough memory-per-blob figure (I saw ~62 bytes/entry plus map/GC overhead mentioned elsewhere for restic) that I could use to estimate this myself for a given blob count, so I’m not just guessing?

4

@nash if you search for OOM in the forum you find other examples.

A common suggestion is to run with a more agressic go garbage collector by setting GOGC=20.

You may give that a try if splitting your repo by host is not enough to prevent OOM.

5

Thanks @GuitarBilly — appreciate the pointer, and wanted to close the loop here rather than leave the thread hanging, since the root cause turned out to be more interesting than I expected.

What actually fixed it: I switched the forget/prune step to rustic, following @alexweiss’s explanation above. Production run on 2026-07-07 via my normal backup_home.sh --backup-and-prune completed cleanly — exit code 0, survived a trough down to 96MB available memory (confirmed via dmesg, no OOM event) where the old restic-based prune would have already been dead. Backup itself still runs through restic; only the memory-heavy forget/prune step moved to rustic. Given rustic’s own warning about no built-in locking on prune, I kept both steps inside the same script/lock so they can never run concurrently against the repo.

The more useful finding: most of those 8.3M blobs shouldn’t have existed at all. Running a file-size histogram against my workstation’s /home turned up 878,877 files under 1KB — 85% of the total file count, but only ~640MB of actual data. Almost all of it traced to /home/nash/.cache/vivaldi: 857,971 files. Browser cache, never meant to be backed up, quietly inflating the blob count for months. Added it to my restic exclusion file; new snapshots won’t carry it forward. Existing snapshots still hold the historical blobs, so the total blob count won’t drop until those age out under my retention policy (--keep-yearly 3, so this will take a while for the oldest ones, but daily/weekly churn should show a steadier decline sooner).

To answer my own question from earlier in the thread: given rustic already handles the current blob count reliably, and the cache exclusion stops the bleeding, neither the RAM upgrade nor the host-based repo split is urgent anymore. Still on my backlog as complementary headroom — the repo split especially, since @alexweiss’s point about proportional-to-dedup-loss savings still applies and NAS-vs-workstation dedup is probably minimal — but nothing’s currently blocking on either.

Lesson for anyone hitting this: before reaching for a bigger hammer (RAM, GC tuning, switching tools), it’s worth running rustic repoinfo or an equivalent blob/file-count breakdown on the source data itself. In my case the “8.3 million blobs” number looked like an inherent property of a small-files-heavy repo, but it was actually one leaked cache directory doing almost all the damage.

Thanks again to both of you — this was a genuinely useful thread to work through.

6

GOGC=20is a pretty tiny hammer :slight_smile:

7

Thanks. I’ll keep an eye on the situation and using GOGC=20 will have priority as an option over installing more RAM.

8

Hi @nash - sorry for the late reply.

Regarding your RAM considerations: You should not do this by calculating theoretical memory usage: Measure it and take into account that it roughly scales with the blob number and depending on how much reserve you like to have. If your reserve is hit, think about buying new memory or other measurements.

One thing especially about rustic prune: Note that this is 2-phase, i.e. the currently marked-as-unused packs are not directly removed. This is what you see in rustic repoinfo as Tree/Data to delete. It will be removed by a subsequent prune run once it stays marked long enough (23h per default).

About your cache files in existing snapshots: You can also remove them using the rewrite command on the affected snapshots, if wanted. (Needs a prune after, of course)

9

@alexweiss

Thank you for your reply.

About your cache files in existing snapshots: You can also remove them using the rewrite command on the affected snapshots, if wanted. (Needs a prune after, of course)

Tried this, and I’m hitting behavior with --glob I can’t make sense of, even after testing several variations in an isolated sandbox repo.

Minimal reproduction:

mkdir -p /tmp/test/data/keep /tmp/test/data/drop
echo "keep me" > /tmp/test/data/keep/file.txt
echo "drop me" > /tmp/test/data/drop/file.txt

rustic -r /tmp/test/repo init --password test
rustic -r /tmp/test/repo --password test backup /tmp/test/data
# snapshot ID: <id>

rustic -r /tmp/test/repo --password test rewrite <id> --glob "!/tmp/test/data/drop/" --all-trees

Expected: drop/file.txt excluded, keep/file.txt retained.

Actual: both files are removed — the rewritten snapshot has 0 files, only the directory skeleton remains.

I’ve tried several variations, all with the same “everything gets wiped” result:

  • The bang pattern alone, with and without **
  • A bare catch-all ** combined with the bang pattern, in both orderings

The only thing that behaves differently is a bare pattern with no bang at all (e.g. --glob "**/drop/**") — that one inverts the problem and keeps only the matched path, dropping everything else.

Is rewrite’s --glob exclude semantics meant to work the same way as backup’s, or is there a different syntax/mental model for tree-rewrite excludes specifically? Version 0.11.2, repo format v2.

Happy to share full terminal output if useful. Also if appropriate we could move the discussion elsewhere e.g. rustic on GitHub?

10

Update — found the cause, wanted to close the loop in case it helps anyone else (or flags something worth fixing).

Root cause was on my end, not rewrite itself: I was passing the exclude pattern inline via --glob "\!/path/", escaping the ! with a backslash to dodge shell history expansion. That backslash-escape doesn’t get stripped inside single quotes the way I assumed — so rustic was actually receiving the literal two-character string \!/path/, not !/path/. Since that doesn’t start with a bare !, it wasn’t recognized as an exclude pattern at all.

That mismatch alone explains one of the two odd results I saw, but it’s worth separately noting the other one, since it surprised me more: a bare pattern with no ! at all doesn’t mean “no effect” — it inverts to become an include-only filter. In my testing, --glob "**/vivaldi/**" (no bang) didn’t leave the tree unchanged — it kept only the files matching that pattern and dropped everything else. So a lone bare glob on rewrite behaves less like “additionally exclude nothing” and more like “keep only this, drop the rest of the tree.” Given my broken bang pattern was silently falling through to be treated as a bare pattern, this is exactly what produced the “everything except the target got deleted” result I was seeing.

Fix for the actual goal (exclude one path, keep everything else): write the pattern to a file and use --glob-file instead of --glob inline — sidesteps the shell escaping issue entirely:

echo '!/home/nash/.cache/vivaldi/' > /tmp/exclude-vivaldi.glob
rustic rewrite <id> --glob-file /tmp/exclude-vivaldi.glob --all-trees

Confirmed working on a real historical snapshot — 880,426 files down to 173,241 after excluding the Vivaldi cache path, all real files verified intact afterward.

Two things that might be worth a docs callout, in case others hit the same traps:

  1. Inline --glob "!..." is fragile from an interactive shell specifically because of how ! interacts with bash history expansion — --glob-file avoids the whole class of problem.
  2. A bare (non-bang) glob pattern on its own isn’t a no-op — it’s an implicit “keep only matches.” Worth being explicit about that in the exclude-patterns doc, since it reads naturally like a “these files, in addition to everything else” pattern rather than “these files, and nothing else.”

Thanks again for the pointer to rewrite — will report back once I run this across the rest of the affected historical snapshots.