Fatal: unable to save snapshot: sftp: no space left on device

Getting Help
1

Hi restic users,

I’m running into an interesting issue when I’m backing up my machine to an SFTP location on my local network, I’m using restic 0.16.4 compiled with go1.22.2 on linux/amd64 (Ubuntu).

The command run and the error look like follows:

sudo restic -r sftp:nas:/restic --verbose --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} backup /
open repository
restic@<redacted>'s password: 
enter password for repository: 
repository 42e7b8f4 opened (version 2, compression level auto)
lock repository
no parent snapshot found, will read all files
load index files
[0:02] 100.00%  22 / 22 index files loaded
start scan on [/]
start backup on [/]
error: no result
error: no result
error: no result
scan finished in 27.601s: 1436676 files, 309.476 GiB
Fatal: unable to save snapshot: sftp: no space left on device

I am using config in .ssh for the machine looking like

Host nas
	HostName <redacted>
	User restic
	Port 2222
	ServerAliveInterval 60
	ServerAliveCountMax 240

The connection definitely works fine, interesting is I ran the command initially and it would backup and upload all files, only finishing the backup would fail. If I now rerun the command it reproducibly gives the output noted above.

Manually logging in with the correct user (restic) on the machine via SFTP it looks like this:

# sftp nas
sftp> df -h
    Size     Used    Avail   (root)    %Capacity
   7.0TB    3.5TB    3.5TB    3.5TB          50%
sftp> cd /restic/
sftp> pwd
Remote working directory: /restic
sftp> ls
config     data       index      keys       locks      snapshots
sftp> df -h
    Size     Used    Avail   (root)    %Capacity
   7.0TB    3.5TB    3.5TB    3.5TB          50%

The permissions for the user are also fine, I can see the uploaded files as well as the lockfile being created when I re-run the backup command as lined out on the very top and space is available. (I actually already played around with different “sftp roots” for the user already since at / the NAS would report a different availability than in the restic folder. But that is definitely not the case anymore).

I have also checked

and checked for enough space on my machine as well as for the .cache folders:

root@lynx2:~/.cache# df -h /root/.cache
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p6   98G   61G   32G  66% /
root@lynx2:~/.cache# df -i /root/.cache
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/nvme0n1p6 6467216 1271625 5195591   20% /
root@lynx2:~/.cache#

Another observation I made is that the amount of GiB scanned ever so slightly increases after subsequent runs. Is there a way to increase verbosity further when running restic?

Any further hints would be appreciated, thank you.
-f

2

Okay, so my intuition suggests to me that /restic is on your nas’ root filesystem, and that is what is running out of space, even though it is inconsistent with the information you have provided above. I’m wondering whether, through trial and error, trying to troubleshoot the problem, something is now configured differently to how you documented it above.

If you haven’t already, sometimes it helps to step back, and start over from the very beginning with setting things up, and copy and paste your steps as you go. In my experience this often unearths the mystery.

If that gives you no joy, then here is what I suggest you try next.

  1. Take a backup of a smaller folder/directory on your computer to see if that backup actually completes successfully.
  2. If you can, login to your nas, and using root, see where sftpd is writing to, using lsof. This will help you to confirm it is writing where intended.
  3. Again if you can login, monitor disk space usage during your backup using df -k and pay attention to the usage up to failure.
  4. One more thing. Without knowing what sort of nas system you have, is it possible that storage quotas are enabled on the filesystem you are writing to? It could be that your restic user account has maxed out its available quota. I believe some filesystem (i.e. zfs) have fs specific quota semantics that can be configured too. If your nas is a proprietary one, take a look in the UI to see if there are user quotas enabled.

See how you go.

Damo.

1 Like
3

Hi Damo and thanks for your reply.

You got it all right with your suggestions and in the end I also figured it out simultaneously: There was a quota applied to the “shared folder” on the volume (it’s a Synology) and since it uses btrfs underneath (depending on your config) it was not the most straight forward thing to figure that part out on the CLI what effectively prohibits the writes. But this command will show it to you if you ssh into and run it as root:

ash-4.4# btrfs qgroup show -pcref /volume1/backup-linux/
qgroupid         rfer         excl     max_rfer     max_excl parent  child 
--------         ----         ----     --------     -------- ------  ----- 
0/588       499.96GiB        0.00B    500.00GiB         none ---     ---

(For completeness, I learned you can also see that on the UI via [Control Panel > Shared Folder])

Thanks again, this thread can be closed and may it be of any help to the next person running into the same or a similar issue.

Good to see that restic/internal/backend/sftp/sftp.go at master · restic/restic · GitHub triggers correctly :slight_smile: