Hi All, I have a fairly large repo (~10T) that I backup to and have daily snapshots of. I used to run a script that did the backup and then a check. Recently I have been blowing out my /tmp tmpfs and now my / parition (changing the cache dir) when running the check. The cache is huge, 85-100G. Is t…

[image] hibby50: I have a fairly large repo (~10T) You’re backing up 10 TB. It’s perfectly expected that you need to spend 100 GB for a cache, if you want a cache. If you don’t find this reasonable, simply use the --no-cache option. I’m not sure what other middle ground you’re expecting …

Hi @hibby50 and welcome to the restic community! :slight_smile: So you could try using the restic cache command to clean out old cache data. restic cache --help The "cache" command allows listing and cleaning local cache directories. EXIT STATUS =========== Exit status is 0 if the command was s…

Hey @moritzdietz thank you for the suggestion! I should have been more clear, my issue is not that the cache is accumulating, as a matter of fact after the check is complete it deletes all of its cache. The issue is while the check command is running the cache climbs to 100G by the time it complete…

Ah! Ok, that clears things up. So I have two theories right now. Your restic check command uses a --cache-dir flag which causes the creation of a cache directory You don’t use that flag, but the restic check command has to download and store the blobs somewhere so it uses your users cache dir by d…

Interesting, I am not using any flags for cache, just the check command, so it seems something is not lining up with the documentation. I played around with the --no-cache flag but it makes it so slow. Perhaps I can shift to doing a weekly check vs a check after every nightly run. I’ll attach my sc…

Hahaha :joy: if it is expected that’s fine. It seemed out of place to me and has only recently become an issue after almost 2 years. The bulk of my storage is spinning rust, so I wasn’t sure the implications of having the cache on a slower medium since I already moved it from default RAM to SSD. Al…

If it’s any comfort, I have like 150 GB cache for much smaller repositories (like 50 GB) :smiley: It’s however not expected, it’s due to an issue with file metadata on macOS.

Note that adding compression helps a lot with large caches as all cached files is metadata saved in JSON format. In my experience compression shrinks down the cache to about 1/5 of the size. I have a repo with ~200 GB size and ~100 MB cache size. Scaling this would mean 5GB for your 10TB repo. But …

[image] moritzdietz: By default, the “check” command will always load all data directly from the repository and not use a local cache. That part of the help text wants to say that check by default does not use an already existing cache, but instead creates a new one. So by default there is a …

Aha! That’s very good to know :point_up: Thanks for the link and information Michael

restic forum

Help with large cache

Getting Help

show post in topic

Home
Categories
Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled