Fixing backups in MacOS Mojave/Catalina (Error: Operation not permitted)

Getting Help
1

Hello everyone,

I’ve been using restic for 2-3 years so far but lately I found out that MacOs is blocking it from accessing certain directories on my Mac. The issue is this one https://github.com/restic/restic/issues/2051

I’ve been postponing solving the problem for some time because I’m not super familiar with the way Macs work (I switched recently after years and years of Linux). However now I MUST fix it because my backups are broken and a lot of important stuff is not being backed up.

Does anybody know what is an easy way to allow restic to access sensible directories? I saw some suggestions in the linked issue above but none of it is that clear to me on how to implement it on my Mac. I’m quite worried I might lose important documents.

To give some context, I followed the documentation on the main website and I have a crontab like this:

42 * * * * source /Users/gio/.restic/restic-env; /usr/local/bin/restic backup -q --exclude-file /Users/gio/.restic/exclude.txt /Users/gio; 
10 15 */5 * * source /Users/gio/.restic/restic-env; /usr/local/bin/restic forget -q --prune --keep-hourly 48 --keep-daily 30 --keep-monthly 12
2

I think the context of your question is a bit vague. Are you having a problem with your backups? If yes, which of the proposed solutions in the issue you referred to have you tried?

Restic will output a warning about files that it cannot read, so if you run your backup and don’t get any warnings about files/folders not being read, it should be fine.

3

Are you certain you need to back up the folders it’s denying access to?

The easy way is to avoid protected directories.

macOS is, roughly, putting an extra permission layer between apps/scripts and some kinds of sensitive user/system data. The goal is generally to notify you that a particular application is trying to access a type of data, and obtain affirmative consent for each app that wants access. I think it’s fair-ish to frame this as baby steps towards a more mobile-OS like permission system, where you’d get prompted if an app wanted to access your photos directory, and then again if it wanted to access your contacts. But, it also has rough edges (like cases where the user isn’t automatically prompted to give consent, and you have to go fishing).

If your goal is just backing up important user files, I’d dig into these and make sure you actually need what’s in them–and, if you do, that you know exactly which ones. (I don’t mean this to be dismissive; I don’t have a direct answer to the question myself, because my backups target directories where I store personal files and I haven’t needed to exempt restic from any of the security & privacy restrictions.)

4

the problem is that MacOS requires programs to get explicit permission from the user to access certain directories. And command line programs cannot ask such permissions, they must be somehow wrapped as applications.
The issue I linked is very long and still ongoing, I saw quite a few ideas being proposed but they all seem to be quite “hacky” so I was wondering if there was some official solution.

5

I know, the approach I’m using though is “backup the whole home directory except few subdirectories”.
Like that I was sure that if I created some new subdirectories I didn’t have to add them explicitely every time to the list “to-backup”.
I seem to understand though that there is no official solution yet, that’s bit of a pity I thought I could solve this more easily.

6

There’s been plenty solutions suggested in the GitHub thread. Personally I use Platypus to wrap my backup script in a .app and then give that .app full disk access - with this I haven’t had any problems, but I am not sure that I have tried it on Catalina yet.

7

Hello there,

I have the same issue.
Following advice I wrap my backup script with Platypus and it’s launch using a LauchDaemons plist.
The backup works well but I have error on reading protected files for users (Library, Mail & co).
The app created has full access.
The user running the script is root (I have to dig a little bit this because apparently it’s not the real “root”) and I backup all of /Users
@rawtaz : you seem to have a solution : are you running are root ? or just as a specific user ?

8

I’m only backing up /Users/<username>, with a bunch of exclusions. But I don’t see why the .app, given full disk access rights, shouldn’t be able to access whatever the user has access to. That probably won’t be the entire system files though, of course.

9

I think its related to the user issuing the command.
How are you launching your backup script ? Which user ? Which method ?
So far, I’ve been trying using launchd Daemons as root.
I’m currently testing to run it as a launchd Agent on a per user base.

10

Interesting fact : when I initiate the script of the .app using our MDM (mosyle) it bypass everything and no error.
I’m not an expert on the macos security but it seems that there are different privileges assigned when using root or using MDM.
Will continue investigation.

11

scan: openfile for readdirnames failed: open /Users/user/Library/Application Support/AddressBook/Sources: operation not permitted
scan: openfile for readdirnames failed: open /Users/user/Library/Mail: operation not permitted
error: openfile for readdirnames failed: open /Users/user/Library/Application Support/AddressBook/Sources: operation not permitted
error: openfile for readdirnames failed: open /Users/user/Library/Mail: operation not permitted
error: open /Users/user/Library/Messages/chat.db: operation not permitted

Warning: at least one source file could not be read

Five years later (7 w.r.t Github #2501) MacOS 15.6 (Sequoia) and it is still happening. Now that I use backrest I can’t even play around wrapping the restic binary in some app something.

Price to pay for being in the fruit garden :confused:

Borg works maybe because I am using it via the app Vorta.

12

Yep, FDA is still a thing and you have to work with it. Otherwise you’re out of luck. As you mentioned wrapping… I did troubleshoot this immensely and got it working in the end. Maybe you can even do some trickery with backrest.

(even though this says APFS, it still applies regardless)

13

My question is: which of the “sensitive” MacOS files do people feel the need to back up?

I’m backing up most of my home folder, but I don’t trip this issue because I don’t backup these directories:

Full Disk Access - what is it and what does it do Fileside
Full Disk Access - what is it and what does it do Fileside1243×662 54.2 KB

Is there stuff in there that people need to back up?

14

Indeed FDA is still a thing and the way Apple messed it up (as it does) it has only gotten worse.

I had seen that post of yours. I can’t really think how it can work with backrest. Besides at this point the reason I use backrest is for ease and comfort :slight_smile:

However what still works is: temporarily giving permission to both Terminal.app and /opt/homebrew/Cellar/restic/x.y.z/bin/restic (i.e not the bundled ~/.local/share/backrest/restic unless I specifically want to use that in the terminal as well) and then running backup from terminal which is what I do once in a while just to ensure those ~/Library folders are also backed up. Otherwise I keep those user library folders commended in my restic include file.

15

I’m backing up most of my home folder

And yet I am not even backing up a quarter of my home folder via restic. I only back up what I… well, you said it…

need to

Besides is there even any point in going to “need to” “want to” “should” “ought” “must” “you are holding it wrong” etc? If for nothing else, it’s entirely unnecessary. People are different. That simple.

which of the “sensitive” MacOS files do people feel the need to back up

It came across as rhetorical so I thought you didn’t really ask for an answer, did you? Or if you need to know, maybe try those you listed and see what “trips” your “wires”.

16

My question is: backing up those files might work but has anyone ever been able to actually restore them successfully? Especially in the case of a loss of the whole system disk/device this might become interesting.

I just remember that I had tried porting my Apple Mail data from one machine to another once and it did not work as expected but I can’t recall the exact reasons. Many settings in ~/Library are rather scattered and I think the permissions are complicated as well.

This is one of the reasons I ditched macOS and went to Linux full time. It was with a heavy heart back then but now I’m happy I did. In Linux these things are mostly still pretty basic and manageable.

17

FWIW, my point wasn’t to defend Apple. I agree entirely that they’ve made a bunch of stuff really difficult.

And therefore my question wasn’t rhetorical. I was genuinely curious if some users felt that there was important data in these “sensitive” folders that they really wanted to backup. And as @nicnab says, I also wonder if it’s even possible to restore this data.

In other words, and regardless of how one feels about Apple’s security efforts, for most regular Mac users who want to back up their personal data, does the restrictions on those folders create a practical problem?

1 Like
18

I think it is rather straightforward. restic is not system backup software. It does not support APFS snapshots, resource forks and who knows what others macOS filesystem peculiarities. It does not provide any functionality to support user in restoring full OS. Maybe on Linux it is doable - still requires a lot of system knowledge and manual tasks. But for macOS or Windows it is simply big no go.

It is perfect program to backup plain files user data, Documents, Pictures etc. to the cloud.

It can be valuable part of full backup solution. In case of macOS quite natural would be to use Time Machine for short/medium term full system backup (which will cover all ~/Library directory) and restic for off-site long term backup/archive of user data.

2 Likes
19

Yes, I was able to retrieve a few contacts and few mails (emlx files hunting down in the folders after quite some time). It was not straight forward. Also, I was definitely not able to use it back in the mail account where I had lost the mail (deleted really; by mistake - of my own at the time totally thinking “I don’t need it anymore”). As for contacts - multiple.

Messages? Tried couple of times and while I could query the db file every time, I somehow never find what I was looking for. (Oh, I found a voucher code I had lost by chance and while it didn’t amount to a lot of money but it was nice).

I used to backup message Attachments folders as well but as usual the messy way Apple handles actual deletion the attachments folders have a life and size of their own. So I stopped backing it up.

whole system disk/device this might become interesting.

The thing about Apple’s “sync” based services is that - if you lose something or something went missing - there are not really much (if I can’t see “any”) options to “recover” that you can fall back to (everyone, please, for the love of everything, please do not even mention Time Machine :D).

I couple of times tried to recover contact (yes, that’s an option; probably last 10 daily or weekly snapshots or so; not sure) - there was no contact history there. None! iCloud Drive also has 30 days retention I believe - you can’t change/set it. And if anything is not how it is supposed to be. You better not even try to contact Apple. 1. They hide behind “privacy/encryption; it just works; we have checked there is no problem on our side” and god they start with asking you to reinstall the Mac! Yes, every bl–dy time! Then reinstall the iOS (if it’s an iCloud matter). A lot of you might be from the West where you could just walk to a store and talk to their staff (also called Genius I guess) and your experience might be different but this is my experience.

They don’t hear any reasoning –> they will take your 2 hours to tell you three sentences → they will not let you escalate! They just won’t! Only way is to write to tim cook’s emails and that mostly goes unanswered.

Anyway, rants aside - the thing is those are personal data and Apple’s sync/backup - while actually convenient (when they work) are extremely unreliable in case of recovery if something goes wrong. It becomes a blunt wall. So you/I try to handle that myself.

I have started to use thunderbird now and I will see if I can’t rather try to set the mail back via that instead of via Mial.app.

one of the reasons I ditched macOS and went to Linux full time

That is genuinely a great decision I’d say. I just don’t want to install Linux on a mac and I do wish non-mac laptop ecosystem where I live was more than less than ideal. I came to mac from linux (around 13 years ago).

1 Like
20

Hey I have answered to nicnab’s comment (rant filled).

Yes, it creates a practical problem. The restriction is not granular, users do not really much of a choice. Though I can guess where they are coming from - because those folders involve stuff like “iMessage” et al, they might be squeamish about making it accessible it even under FDA easily.

@kapitainsky Yes, restic is not a system backup software but ~/Library is not a system dir either. Besides there are personal data inside those folders which are not elsewhere and those are not easily accessible to backup programs. Not even in iCloud folder (which is accessible under a separate permission).

All I have to say about TM is - if Time Machine was a viable solution or even worth anything when it comes to efficiency, optimisation, and cloud support then, if not all - almost none of the mac users would even look for tools like restic, borg etc.

So:

  1. restic doesn’t have to become a signed binary or .app or something to be able to access those folders
  2. Yes, there is a need for some people to backup those specific folders - those are not “system/OS” fodlers. I mean is that difficult to accept? :stuck_out_tongue: Are you all non-Mac users assuming since there’s “Library” in it, it’s a system folder?
  3. I faced this specific problem again after long and searched around and after that just necro-bumped this with a comment here hoping maybe something new might have surfaced and if someone would see my comment and knows a solution I might try that. OF course there hasn’t been a solution and it’s fine.