Stop restic replacing ssh target FQDN with IP

Getting Help
1

I recently made some changes to SSH and my hosts and switched my backup sftp destination from an IP to a .local fqdn. Once that happened, I started getting errors

subprocess ssh: Permission denied, please try again.
subprocess ssh: Permission denied, please try again.
subprocess ssh: stratacast@192.168.10.45: Permission denied (publickey,password).
Fatal: unable to open repository at sftp:192.168.10.45:/usr/home/stratacast/rstc-desktop: unable to start the sftp session, error: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

I realized that restic was replacing the fqdn with the IP address during the SSH process. With no entry for 192.168.10.45 in my ssh config, I would get this error. Can I tell restic to NOT replace my SSH target’s domain name with its IP address?

2

Can you please paste the whole command and output to see what happened? What version of restic? Any other hints at your configuration? restic doesn’t do anything different than any other commands as far as I know.

1 Like
3 
RESTIC_PASSWORD_FILE="$HOME/.local/bin/.restic_pass"
export RESTIC_PASSWORD_FILE
RESTIC_REPOSITORY="sftp:homebackup.local:/home/stratacast/rstc-desktop"
export RESTIC_REPOSITORY

restic --compression=auto --exclude-file $excludes backup $HOME /var/www/htdocs

This is the output of a successful run when it from the command line, with the IP or FQDN:

repository b69b9215 opened (version 2, compression level auto)
using parent snapshot 50b42516
[0:00] 100.00%  37 / 37 index files loaded

Files:          98 new,   100 changed, 343023 unmodified
Dirs:            6 new,   198 changed, 71031 unmodified
Added to the repository: 50.680 MiB (15.387 MiB stored)

processed 343221 files, 192.572 GiB in 0:16
snapshot 72328adf saved

The output is the same if I specify the IP for the cron job. The log message in my first post is the output I get when I run this script via cron and the FQDN is specified.

My ssh config:

Host homebackup.local
        User            stratacast
        Port            22
        IdentityFile    /home/stratacast/.ssh/homeBackup

Host 192.168.10.45
        HostName        192.168.10.45
        User            stratacast
        Port            22
        IdentityFile    /home/stratacast/.ssh/homeBackup

Restic version: 0.16.1

The ssh keys are stored in my ssh-agent. It is clear restic is gaining access to the key, since it is working when I specify my target with the IP, not the fqdn. The odd part is, that restic is for some reason substituting the target only when run as a cron job. Maybe something in the ssh lib that it uses? I’m also leaving room for it being something with the environment in cron doing something, but based on the ssh subprocess messages, my guess is for some reason restic is passing the resolved name to ssh, not what it is receiving in in the -r arg

4

If you can successfully run this on an interactive shell yourself, can it be that the cron job does not use your ssh_config file to map the the FQDN or IP correctly?

Maybe you can add the --verbose option in your cron job to generate more verbose output to troubleshoot this further.

5

Hm looks ok to me. Is there a reason why you don’t say HostName homebackup.local in the first ssh config entry? Not sure what that would do but maybe worth a try? And yes what @moritzdietz said sounds like an option too. Some things are different when you run things via cron.

6

restic just passes the specified hostname to ssh without further processing.

As you see the problems only in the cronjob, this means that the cronjob still uses the old configuration with the IP as target. So please check what exactly is run in the cronjob, and whether the RESTIC_REPOSITORY environment variable might be set in some unexpected place (e.g. run grep for /etc and other places where the configuration might be stored.

7

I’ve learned that specifying HostName isn’t necessary when the entry is the name of the host itself. I’ve tried with/without with the same results

8

I think you are onto something. A piece of information I forgot to add to that is in my ssh_config is I am using the control master config

Host *
        IdentitiesOnly          yes
        ObscureKeystrokeTiming  yes
        ControlPath             ~/.ssh/controlmasters/%r@%h%p
        ControlMaster           auto
        ControlPersist          60m

So I think any time at all that it works, I would have had the control master entry in place already. If I SSH in to 192.168.10.45 ahead of time, everything works because that channel is already in place. So I think there’s 2 problems:

  1. why does the hostname change to the IP at ssh login
  2. why does it appear that restic is not reading from ~/.ssh/config?

I am using SSH certificates, does restic not support that?

9

Restic just runs ssh to open the connection to the target. Thus, everything supported by ssh should work. If you’re interested in the exact command that restic uses to start ssh, then set the environment variable DEBUG_LOG=restic.log before running restic. Then search for start client in that log.

1 Like
10

That’s good to know. I found the flags for restic to modify the ssh command and get some debugging rolling on the command itself. I’ll report back with my findings. Maybe there’s some security feature between running a cron job as my user vs running it as myself interactively. I’ll run it this way and the DEBUG_LOG flag you mentioned. Thank you!

11

Thank you all for steering me in the right direction. I didn’t know that restic uses the installed SSH binary. My keys either may not have been previously password protected like I thought and when I regenerated my key, I switched to the SSH certificate method. It seems that both the Hostname change and certificate change were red herrings, and the real culprit was that cron didn’t have access to my ssh agent. The output of ssh -vv helped

subprocess ssh: debug1: read_passphrase: can't open /dev/tty: Device not configured
subprocess ssh: debug2: no passphrase given, try next key

I set my ssh-agent to bind to a custom path that won’t change, then in my script that runs restic I added

SSH_AUTH_SOCK=/home/stratacast/.ssh/myagent
export SSH_AUTH_SOCK

Now I can ssh with the FQDN, and I confirmed it doesn’t translate to the IP since it shows up as the FQDN in my control master path.

3 Likes