Restic permission denied with SFTP but works with sshfs

❯ restic version
restic 0.17.3 compiled with go1.23.3 on linux/amd64

Hi all, I am trying to use restic over SFTP. To start with, only running snapshots. I am getting Permission denied errors, which I don’t understand as I should have all permissions. Furthermore, mounting the same location with SSHFS and then running restic snapshots with the same options does work! This leaves me a bit confused, I thought ssfhs was just a wrapper over sftp? Are there some sort of permission differences between the two?

First, the output when trying to use sftp:

❯ restic snapshots -r sftp:francis@truenas.home:/mnt/main/backups/restic-1 --password-file password.txt
repository 95891bc5 opened (version 2, compression level auto)
Save(<lock/1161dc57f3>) failed: OpenFile: permission denied
unable to create lock in backend: OpenFile: permission denied

Now when I mount the same path with sshfs:

❯ sshfs francis@truenas.home:/mnt/main/backups/restic-1 ~/mnt/truenas-restic-1/

❯ restic snapshots -r ~/mnt/truenas-restic-1 --password-file password.txt
repository 95891bc5 opened (version 2, compression level auto)
ID        Time                 Host                   Tags        Paths            Size
----------------------------------------------------------------------------------------------
(snapshots redacted)
----------------------------------------------------------------------------------------------
2 snapshots

I also checked that when using plain sftp I am able to do all operations: ls, put, rm, mkdir, rmdir, etc.

Would anyone have any idea what’s going on and why restic with sftp: repo behaves differently to sshfs?

Interestingly, what I understand from the restic error is that it could not create the lock? However when I ls the locks directory in the repo (mounted using sshfs), I do see the lock was created:

❯ restic snapshots -r sftp:francis@truenas.home:/mnt/main/backups/restic-1 --password-file  password.txt
repository 9ce4e894 opened (version 2, compression level auto)
Save(<lock/e6fb028c83>) failed: OpenFile: permission denied
unable to create lock in backend: OpenFile: permission denied

❯ ls ~/mnt/truenas-restic-1/locks/
e6fb028c8376142ff377fb5e1b8fee90267746e81da5dd6ebb959efc7fb7654e-restic-temp-88098707c19bfe77f25bce23c70c7c7c

Could this be a bug in restic? Is there a way to get more verbose output from restic? Even using --verbose=2 or -vv doesn’t give me anymore output.

1 Like

Can you run ls -l restic-1 and ls -l restic-1/locks (on the truenas.home end) and share the result ?

2 Likes

@auxym, please have a look here to read about more extensive logging:

## Debug Logs

2 Likes
(ssh'd on the remote machine)

francis@truenas:~$ ls -l /mnt/main/backups/restic-1/
total 111
-rwxrwx---   1 francis francis 155 Mar 25 17:12 config
drwxrwx--- 258 francis francis 258 Mar 25 17:12 data
drwxrwx---   2 francis francis  11 Mar 25 22:55 index
drwxrwx---   2 francis francis   3 Mar 25 17:12 keys
drwxrwx---   2 francis francis   9 Mar 25 22:55 locks
drwxrwx---   2 francis francis   3 Mar 25 22:55 snapshots

francis@truenas:~$ ls -l /mnt/main/backups/restic-1/locks/
total 4
-rwxrwx--- 1 francis francis 0 Mar 25 17:36 38d10377329c42525ab31666a320931bd20f02f72d7d0ddfe763f8a7ec21f013-restic-temp-aa90150a4510a4c1f71f6706d6959c55
-rwxrwx--- 1 francis francis 0 Mar 25 17:36 4f3cd333c59a228a1ada45f12adf7f08dccaa485c6c8fc21343ca1b7da80660b-restic-temp-4044dc731b055a781e0081907e34c8d4
-rwxrwx--- 1 francis francis 0 Mar 25 17:34 55e16be028fe0b444fa207056ec139c7c3cd96e67b1b53de306567dce44c9f45-restic-temp-ce18b2a94418da8c0dfedbfc4b9029ce
-rwxrwx--- 1 francis francis 0 Mar 25 17:34 a301fda6c7c0aeb6ec08a138430b0deeb489d3f047566a7bb17ef94377e0b81a-restic-temp-3f9096a4934d8677e8f8f1e4310c3508
-rwxrwx--- 1 francis francis 0 Mar 25 17:35 a58e532d987f31a2eada5a1046bf3926e1b39e9558a461f1ba54e70d3d29b730-restic-temp-ac15a8d8cda906a7556c7c6926844804
-rwxrwx--- 1 francis francis 0 Mar 25 17:34 ad7ff6c490eb26e1c4ebc0a38518c25f6ad54f5c4ba91761f8db1089d79490ae-restic-temp-7bb1fc56ec40e7c4bb92229f14bcaed9
-rwxrwx--- 1 francis francis 0 Mar 25 17:15 e6fb028c8376142ff377fb5e1b8fee90267746e81da5dd6ebb959efc7fb7654e-restic-temp-88098707c19bfe77f25bce23c70c7c7c

All those locks are from me trying to run restic snapshots with sftp: as above.

Here is the log result from running restic snapshots -r sftp:... as above:

2025/03/26 20:24:32 restic/main.go:137	main.main	1	main []string{"restic", "snapshots", "-r", "sftp:francis@truenas.home:/mnt/main/backups/restic-1", "--password-file", "/home/francis/.config/resticprofile/password.txt"}
2025/03/26 20:24:32 restic/main.go:138	main.main	1	restic 0.17.3 compiled with go1.23.3 on linux/amd64
2025/03/26 20:24:32 restic/global.go:583	main.innerOpen	1	parsing location sftp:francis@truenas.home:/mnt/main/backups/restic-1
2025/03/26 20:24:32 restic/global.go:578	main.parseConfig	1	opening sftp repository at &sftp.Config{User:"francis", Host:"truenas.home", Port:"", Path:"/mnt/main/backups/restic-1", Layout:"", Command:"", Args:"", Connections:0x5}
2025/03/26 20:24:32 sftp/sftp.go:143	sftp.Open	1	open backend with config sftp.Config{User:"francis", Host:"truenas.home", Port:"", Path:"/mnt/main/backups/restic-1", Layout:"", Command:"", Args:"", Connections:0x5}
2025/03/26 20:24:32 sftp/sftp.go:61	sftp.startClient	1	start client ssh [truenas.home -l francis -s sftp]
2025/03/26 20:24:33 layout/layout.go:147	layout.ParseLayout	1	parse layout string "" for backend at /mnt/main/backups/restic-1
2025/03/26 20:24:33 layout/layout.go:103	layout.DetectLayout	1	detect layout at /mnt/main/backups/restic-1
2025/03/26 20:24:33 layout/layout.go:121	layout.DetectLayout	1	found default layout at /mnt/main/backups/restic-1
2025/03/26 20:24:33 layout/layout.go:175	layout.ParseLayout	1	layout detected: <DefaultLayout>
2025/03/26 20:24:33 sftp/sftp.go:161	sftp.open	1	layout: <DefaultLayout>
2025/03/26 20:24:33 sftp/sftp.go:165	sftp.open	1	using (0o660 file, 0o770 dir) permissions
2025/03/26 20:24:33 logger/log.go:52	logger.(*Backend).Stat	1	Stat(<config/>)
2025/03/26 20:24:33 sema/semaphore.go:27	sema.semaphore.GetToken	1	acquired token
2025/03/26 20:24:33 logger/log.go:54	logger.(*Backend).Stat	1	  stat err <nil>
2025/03/26 20:24:33 logger/log.go:24	logger.(*Backend).IsNotExist	1	IsNotExist(<nil>, <nil>, false)
2025/03/26 20:24:33 index/index.go:375	index.(*Index).Finalize	1	finalizing index
2025/03/26 20:24:33 logger/log.go:59	logger.(*Backend).List	1	List(key)
2025/03/26 20:24:33 sftp/sftp.go:531	sftp.(*SFTP).List	1	send 0695ac45ef6ced0403c86b6acf5898d11da4155065863e984d8f57d81e6b9b37
2025/03/26 20:24:33 repository/key.go:145	repository.SearchKey.func1	1	trying key "0695ac45ef6ced0403c86b6acf5898d11da4155065863e984d8f57d81e6b9b37"
2025/03/26 20:24:33 logger/log.go:45	logger.(*Backend).Load	1	Load(<key/0695ac45ef>, length 0, offset 0)
2025/03/26 20:24:33 sema/semaphore.go:27	sema.semaphore.GetToken	1	acquired token
2025/03/26 20:24:33 logger/log.go:47	logger.(*Backend).Load	1	  load err <nil>
2025/03/26 20:24:34 repository/key.go:158	repository.SearchKey.func1	1	successfully opened key 0695ac45ef6ced0403c86b6acf5898d11da4155065863e984d8f57d81e6b9b37
2025/03/26 20:24:34 logger/log.go:61	logger.(*Backend).List	1	  list err context canceled
2025/03/26 20:24:34 repository/repository.go:163	repository.(*Repository).LoadUnpacked	1	load config with id 0000000000000000000000000000000000000000000000000000000000000000
2025/03/26 20:24:34 logger/log.go:45	logger.(*Backend).Load	1	Load(<config/0000000000>, length 0, offset 0)
2025/03/26 20:24:34 sema/semaphore.go:27	sema.semaphore.GetToken	1	acquired token
2025/03/26 20:24:34 logger/log.go:47	logger.(*Backend).Load	1	  load err <nil>
2025/03/26 20:24:34 cache/cache.go:99	cache.New	1	using cache dir /home/francis/.cache/restic/9ce4e894892d99952bab6055d8a41a5c29a3328f57b5384ecbac0cedef4314e6
2025/03/26 20:24:34 repository/repository.go:151	repository.(*Repository).UseCache	1	using cache
2025/03/26 20:24:34 cache/cache.go:223	cache.OlderThan	1	0 old cache dirs found
2025/03/26 20:24:34 logger/log.go:59	logger.(*Backend).List	51	List(lock)
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send 4f3cd333c59a228a1ada45f12adf7f08dccaa485c6c8fc21343ca1b7da80660b-restic-temp-4044dc731b055a781e0081907e34c8d4
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse 4f3cd333c59a228a1ada45f12adf7f08dccaa485c6c8fc21343ca1b7da80660b-restic-temp-4044dc731b055a781e0081907e34c8d4 as an ID
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send 38d10377329c42525ab31666a320931bd20f02f72d7d0ddfe763f8a7ec21f013-restic-temp-aa90150a4510a4c1f71f6706d6959c55
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse 38d10377329c42525ab31666a320931bd20f02f72d7d0ddfe763f8a7ec21f013-restic-temp-aa90150a4510a4c1f71f6706d6959c55 as an ID
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send ad7ff6c490eb26e1c4ebc0a38518c25f6ad54f5c4ba91761f8db1089d79490ae-restic-temp-7bb1fc56ec40e7c4bb92229f14bcaed9
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse ad7ff6c490eb26e1c4ebc0a38518c25f6ad54f5c4ba91761f8db1089d79490ae-restic-temp-7bb1fc56ec40e7c4bb92229f14bcaed9 as an ID
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send e6fb028c8376142ff377fb5e1b8fee90267746e81da5dd6ebb959efc7fb7654e-restic-temp-88098707c19bfe77f25bce23c70c7c7c
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse e6fb028c8376142ff377fb5e1b8fee90267746e81da5dd6ebb959efc7fb7654e-restic-temp-88098707c19bfe77f25bce23c70c7c7c as an ID
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send a58e532d987f31a2eada5a1046bf3926e1b39e9558a461f1ba54e70d3d29b730-restic-temp-ac15a8d8cda906a7556c7c6926844804
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse a58e532d987f31a2eada5a1046bf3926e1b39e9558a461f1ba54e70d3d29b730-restic-temp-ac15a8d8cda906a7556c7c6926844804 as an ID
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send 55e16be028fe0b444fa207056ec139c7c3cd96e67b1b53de306567dce44c9f45-restic-temp-ce18b2a94418da8c0dfedbfc4b9029ce
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse 55e16be028fe0b444fa207056ec139c7c3cd96e67b1b53de306567dce44c9f45-restic-temp-ce18b2a94418da8c0dfedbfc4b9029ce as an ID
2025/03/26 20:24:34 sftp/sftp.go:531	sftp.(*SFTP).List	51	send a301fda6c7c0aeb6ec08a138430b0deeb489d3f047566a7bb17ef94377e0b81a-restic-temp-3f9096a4934d8677e8f8f1e4310c3508
2025/03/26 20:24:34 repository/repository.go:824	repository.(*Repository).List.func1	51	unable to parse a301fda6c7c0aeb6ec08a138430b0deeb489d3f047566a7bb17ef94377e0b81a-restic-temp-3f9096a4934d8677e8f8f1e4310c3508 as an ID
2025/03/26 20:24:34 logger/log.go:61	logger.(*Backend).List	51	  list err <nil>
2025/03/26 20:24:34 restic/json.go:25	restic.SaveJSONUnpacked	1	save new blob lock
2025/03/26 20:24:34 logger/log.go:30	logger.(*Backend).Save	1	Save(<lock/8cad350591>, 160)
2025/03/26 20:24:34 logger/log.go:32	logger.(*Backend).Save	1	  save err OpenFile: permission denied
2025/03/26 20:24:34 retry/backend_retry.go:170	retry.(*Backend).Save.func1	1	Save(<lock/8cad350591>) failed with error: OpenFile: permission denied
2025/03/26 20:24:34 restic/global.go:254	main.Warnf	1	Save(<lock/8cad350591>) failed: OpenFile: permission denied
2025/03/26 20:24:34 repository/repository.go:479	repository.(*Repository).SaveUnpacked	1	error saving blob <lock/8cad350591>: OpenFile: permission denied
2025/03/26 20:24:34 restic/cleanup.go:39	main.Exit	1	exiting with status code 1

I don’t immediately see much in there except the same Permission Denied error. I’m not sure if these multiple “unable to parse” errors are meaningful?

OK I mostly figured it out. The short version is: change the ACL mode on the Truenas dataset from Restricted to Passthrough. I can now run restic snapshots without an error (and restic backup, too!).

From my understanding (Who can explain aclmode = discard, passthrough and restricted with Example? | TrueNAS Community), Restricted mode basically rejects any attempt to chmod a file or directory. I assume restic must do some chmodding? And this blocked it.

1 Like

Yes, restic restricts the file permissions to only allow the owner read/write access to the data in the repository. Restic 0.18.0 (should be released any minute now) will provide better error messages for the SFTP backend.

1 Like