Hi everyone,
I am wondering whether there is an existing command that can tell me the file paths to objects stored in a local directory repository that are linked to a particular snapshot?
My use case involves sgi’s data migration facility (DMF). DMF transparently migrates infrequently accessed data to lower-cost storage devices. When you attempt to access files that have been migrated, it transparently fetches the files and brings them back online while the software blocks in the I/O request. While this is fine with small numbers of files, for software that accesses many files, like restic, this makes it prohibitively slow.
DMF does allow you to pre-fetch filenames in a batch, such that all required files can be migrated back online at once. Then access with software is immediate and responsive. Hence, my query about restic being able to list out the repository file paths belonging to a snapshot, such that DMF can fetch them, before running restic for restores.
Hope this makes sense.
Many thanks.
Damo.
Have a look at rustic which IMO exactly supports your requirement as the so-called “cold-storage support”.
Note that there are metadata files which are regularly accessed by restic/rustic (and most of those are cached if you have enabled a cache). So, rustic uses a “hot repo” to store this hot data. In your setting you should ensure that this hot data is never migrated by DMF, e.g. by using the hot/cold repo approach rustic offers.
The actual file contents are saved in the cold repo and may need warm-up before they are accessed.
In rustic, you therefore have the option to warm up needed files in the commands restore, prune and repair index. The warm-up even works when using the dry-run mode.
So, to get all needed data pack for a given snapshot, you can run
rustic restore <SNAP-ID> /restore --dry-run --warm-up-command "echo %id" > files
and you will get the repo pack data file names to warm up (e.g. in /path-to-repo/data/). This works with hot/cold repos, but also with “standard” restic/rustic repos.
Note: rustic always checks if local files in the restore destination exist which already (partially) contain the needed file contents. The already-existing content is skipped in the warm-up list. So make sure that your restore-destination is an empty dir to cover all contents of the specified snapshot!
Thanks Alex
I’m not the sysadmin but a hpc user so I’m not able to formulate my own hot/cold storage rules. But I will give rustic a go with a standard repo and warming it up before restores. Will share what I learn after some testing.
Thanks for your assistance.
Damien.
Happy to get your feedback and don’t hesitate to ask if you have questions. For rustic-specific questions, feel free to use the rustic discussions.
Actually if your storage really transparently warms up the needed files, then the restore --warm-up option of rustic should be the choice for you (again, this also works with a repo where files are transparently migrated) - it accesses all needed files in parallel just before starting the normal restore. I’m using this with OVH cloud archive where trying to access a file also warms it up if it’s not warmed already.