(ssh_fx_permission_denied)

Getting Help
1

Hello everyone,

I’ve been dealing with some permission-related issues with my Restic backup repository over SFTP and wanted to ask for your help.

• Version: restic 0.9.4 compiled with go1.11.6 on linux/amd64.

1- Initial Problem:

I encountered the following error in my backup logs:

Load(<index/1c73fc4299>, 0, 0) returned error, retrying after 583.982418ms: sftp: "Permission denied" (SSH_FX_PERMISSION_DENIED)

2- I compared the index file permissions between two installations (A and B):

Installation A (working):
-r-------- 1 root root [size] [date] [index files]
-r-------- 1 root root [size] [date] [index file]

Installation B (problematic) → Had mixed ownership:
-r-------- 1 root root [size] [date] [index file]
-r-------- 1 userX userX [size] [date] [index file]
-r-------- 1 userX userX [size] [date] [index file]
-r-------- 1 userX userX [size] [date] [index file]

Since the most recent index file was owned by root:root, I changed ownership to userX:userX, since the previous ones in this installation had been working.

3- Then I ran restic rebuild-index, which resulted in new errors, but this time in data:

``Load(<data/81b447e5b9>, 591, 13778406) returned error, retrying after 409.551505ms: sftp: “Permission denied” (SSH_FX_PERMISSION_DENIED)`

After rebuild-index, I noticed:

  • Old index files disappeared
  • Only one new index remained, owned by root:root:

4- Afterwards , running restic check showed:

created new cache in /tmp/restic-check-cache-720761760
create exclusive lock for repository
load indexes
check all packs
pack 81b447e5: not referenced in any index
pack d79761a5: not referenced in any index
2 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree f003cd40:
  tree f003cd4010c50c0d818d47e65c5f5f4f3519284f7930e0d5153d8d91602ff1c2 not found in repository

5- As suggested I ran restic prune, but got more permission errors in data again:

Load(<data/81b447e5b9>, 591, 13778406) returned error, retrying after 376.970741ms: sftp: "Permission denied" (SSH_FX_PERMISSION_DENIED)

But then, again comparing both installations A and B, they have the same permissions in data, e.g.

Installation A:
/data/00 →
-r-------- 1 userX userX [size] [date] [index file]
-r-------- 1 userX userX [size] [date] [index file]

Installation B:
/data/00 →
-r-------- 1 userX userX [size] [date] [index file]
-r-------- 1 userX userX [size] [date] [index file]

I’m hesitant to make any further changes as I don’t want to risk damaging anything or make irreversible changes to the repository. Any guidance would be appreciated.

Thank you all in advance for your help!

2

Hi @mb19 !

It seems to me like you are connecting over SFTP as different users (“root” and “userX”) at different times. As a start, I recommend that you chown all files to “userX:userX” and then make sure that you are always using only that user from now on.

3

Hello!
Thanks for the suggestion, but I have already set the owners of index (mentioned in step 2), however, the errors persisted, and in data, the owners didn’t change

4

It is good that you have fixed the ownership inconsistencies. What I want to emphasize is the importance of using correct user when connecting to the repository.

As to the other problems/symptoms, I rely on others being able to better help you than I am capable of. :slight_smile:

5

With the latest restic being v0.17.4 you should seriously consider upgrading your vintage version from 2019… It is 6 years old. There were 20+ restic releases since then not only bringing new features but fixing multiple bugs. Definitely SFTP you are using has been improved a lot. Check release notes yourself:)

6

All files and folders in a repository are supposed to be owned by the user which you’re using to access the repository via sftp.

As this seems to be a permission problem, we need to know more about the server setup besides that there’s sftp involved somewhere.

7

Hello!

You were all right.
It turns out that in some of the directories under ‘data’, some of the files had incorrect owners as ‘root:root’

I saw this by running ls -lR | grep -v 'nameX'

Now that I have changed them, the problem I have is that when I run the check, this happens:

created new cache in /tmp/restic-check-cache-158160092
create exclusive lock for repository
load indexes
check all packs
pack 81b447e5: not referenced in any index
pack d79761a5: not referenced in any index
2 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree a7b2b1ad:
  tree a7b2b1ad0d5b4f194ef9ece9b9f5c3357e36fe298d2732b5afab62e8841dce24 not found in repository
error for tree bedd3196:
  tree bedd3196afc98ba781d21ae5c143cf73ee67eac69586fd8c7ac1f2d43ce8b430 not found in repository
error for tree 2e2502da:
  tree 2e2502daee4fa9ff2095e5092dbbb360cce68d4770661cf386b9a517ace3d02e not found in repository
error for tree 09e35c16:
  tree 09e35c1694a61b1f92f80d97743233da108c26c01c9ab419f3f5e09c22213142 not found in repository
error for tree 3b397ae5:

And it suggests the prune, but the prune doesn’t solve the issue either:

repository 765a2f59 opened successfully, password is correct
counting files in repo
building new index for repo
[0:20] 100.00%  695 / 695 packs
repository contains 695 packs (6720 blobs) with 3.213 GiB
processed 6720 blobs: 2324 duplicate blobs, 868.409 MiB duplicate
load all snapshots
find data that is still in use for 40 snapshots
tree 09e35c1694a61b1f92f80d97743233da108c26c01c9ab419f3f5e09c22213142 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        github.com/restic/restic/internal/repository/repository.go:653
github.com/restic/restic/internal/restic.FindUsedBlobs
        github.com/restic/restic/internal/restic/find.go:11
main.pruneRepository
        github.com/restic/restic/cmd/restic/cmd_prune.go:191
main.runPrune
        github.com/restic/restic/cmd/restic/cmd_prune.go:85
main.glob..func18
        github.com/restic/restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra/command.go:800
main.main
        github.com/restic/restic/cmd/restic/main.go:86
runtime.main
        /usr/lib/go-1.11/src/runtime/proc.go:201
runtime.goexit
        /usr/lib/go-1.11/src/runtime/asm_amd64.s:1333

Regarding the update, for now, it’s not an option :disappointed:

8

Rebuilding the repository index might help (restic rebuild-index, but create a backup of the index folder first). But if that doesn’t solve the problem then the only options are removing ALL damaged snapshots (see Route 2 in Recover from broken pack file · Issue #828 · restic/restic · GitHub) or using a newer restic version for repairing the repository. Honestly, trying to fix the repository with your ancient restic version is a waste of time, so I’m not going to provide additional help for that route.

Restic is just a single binary. Just download it from Github, unpack it and mark it as executable. Then follow the Troubleshooting — restic 0.17.3 documentation steps.

As long as you don’t upgrade the repository format version, both restic 0.9.4 and 0.17.3 will be able to work on the same repository. But I can highly recommend using the latest restic version, which will be much faster, much more reliable and so on.