I’m concerned with backing up an APFS or HFS+ disk to a ZFS formatted NAS. I am considering my options, including using restic over sftp.
After some research, I see that on MacOS, standard unix tools like rsync and tar have been updated to handle the complicated metadata structure (Extended Attributes, including variable numbers of variable length forks; Finder data; and other metadata). So they will work if backing up to another Apple formatted disk, but not to “real” unix formats; and the standard unix tools will not be able to handle the output. I also see that Apple’s NFS has been updated with the capability of transferring in the “AppleDouble” format [1] and that their SMB will transfer as something called “Alternate Data Streams” (ADS) [2], both of which can be stored to non-Apple disks. (Though for NFS & SMB, I am not clear what the requirements are on the receiving end. If someone knows, please tell me.)
So my question is, could someone describe how restic backs up APFS/HFS+ files, including what kind of format it uses to store those files in its archive? AppleDouble? ADS? Something else?
Also, if I were to then try to copy that archive to a linux machine and explore it with restic, what would happen? (For example, AppleDouble format is very simple. There are two files, one containing the data fork (filename) and one containing the resource fork and Finder metadata (._filename))
I’m happy to read docs if someone wants to point me to them, but I’m coming up empty when I search them for AFPS, HFS+, MacOS, resource fork, etc.
Thanks!
[1] https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats
[2] https://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_(ADS)
Currently restic has no support of such metadata. Here is list of what’s currently preserved:
https://restic.readthedocs.io/en/latest/manual_rest.html#metadata-handling
See also followed related issue:
I thank you for that very thorough link/bug report. Having read it, I would say that the support is partial.
I’ts missing a couple of pieces, but is actually reasonably complete. I have made a comment there, regarding that the creation time issue should perhaps be moved to a bug report rather than in the “Feature Enhancement” issue that it is in presently.
I am still left with the question I asked. How is this data saved? Is it something that is already a recognizable standard like AppleDouble or Alternate Data Streams? OR is it something else. If I wanted to investigate the various components of a file from a linux system, how would that work? Would I be able to do it? If I tried to restore to a linux FS, would I just get the data fork?
Edit: I took it upon myself to post the creation-time problem as a bug report
Data is saved inside repository in restic internal format. Before encryption every file is represented by small JSON.
One of the coolest thing is that there is good documentation about restic repository format:
https://restic.readthedocs.io/en/stable/100_references.html#trees-and-data
On linux you’ll be able to ‘restore’ snapshot. You’ll get just data fork named with original file name. Alternatively you can just mount restic repository using restic mount and get virtual filesystem where snapshots are represented as directories.
Personally I’ve used restic to backup Mac in the past. But there was no goal to restore bootable system.
@dionorgua - I thank you for that link. I did watch a talk on restic, and have some idea of the deep structure of the archive. That is one level deeper than my question though. I’ll try to be clearer.
Suppose I have a restic backup of an HFS+ disk. I try to restore to some linux FS. I get errors on some files due to some of the metadata that resic has backed up not being restoreable on that linux FS. Can I explore:
Which EA’s are failing and what are their values?
What is the content of the resource fork?
Look at whatever other metadata may be causing problems?
FYI, the only other solution I have found that seems to be cross platform, free (as in freedom) and work with Mac disks is dar. I have not tried it, but some discussion with the developer suggests that they are doing slightly better with this issue than restic has so far. Here is my comment regarding dar on restic's github issues tracker.
This being a subsidiary question, I reworded and moved it here.