Can't browse mounted repo

Hi, I’m trying to mount a repo in order to browse it, I’ve mounted it with the command:

restic -r repo_name mount mnt_ssh
enter password for repository: 
repository bc991fef opened successfully, password is correct
created new cache in /home/marco/.cache/restic
Now serving the repository at mnt_ssh
Use another terminal or tool to browse the contents of this folder.
When finished, quit with Ctrl-c here or umount the mountpoint.

then if I try to browse the mounted dir I see the four directories: hosts ids snapshots tags, but I can’t browse the snapshot itself, because I get an input/output error:

❯ ls mnt_ssh/snapshots/
2022-04-08T03:00:01+02:00  2022-04-22T03:00:02+02:00  latest
2022-04-15T03:00:01+02:00  2022-04-29T03:00:01+02:00

❯ ls mnt_ssh/snapshots/latest
ls: lettura della directory 'mnt_ssh/snapshots/latest': Errore di input/output

❯ ls mnt_ssh/snapshots/2022-04-29T03:00:01+02:00
ls: lettura della directory 'mnt_ssh/snapshots/2022-04-29T03:00:01+02:00': Errore di input/output

What should I do?

Thanks!
Marco

Hm, that’s odd. You did let restic run after the mount, right? Is the user running restic the same as the user browsing the snapshots?

Yes I let restic run after the mount, the user is the same (two different machines). I’ve tried also by root, but nothing.

Which restic version are you using? And which operating system?

I’m using Archlinux, Restic 0.13. The repo has been created with Restic Arm 0.12 (rpi).

Please show us the output of the (regular Linux command) mount before you mount the repository.

Please also try mounting it at /tmp/foobar or similar subdirectory under /tmp, but of course after first running mkdir /tmp/foobar (or similar, correspondingly).

❯ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=3999344k,nr_inodes=999836,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on / type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14820)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=4009568k,nr_inodes=1048576,inode64)
/dev/sda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=801912k,nr_inodes=200478,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Same error mounting under /tmp:

❯ ls /tmp/foobar/snapshots/2022-04-29T03:00:01+02:00
ls: lettura della directory '/tmp/foobar/snapshots/2022-04-29T03:00:01+02:00': Errore di input/output

After mounting restic repo mount command adds:

restic on /tmp/foobar type fuse (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Which FUSE do you have installed?

Also, can you please try this with e.g. restic 0.12 and 0.11 as well to see if it makes a difference?

Finally, have you tried FUSE mounting something else than a restic repository on this system, and if so was it successful?

Hi, I’ve tried both Restic 0.12.1 and 0.11.0 but there’s no difference, same error of input/ourput.

On my Archlinux machine I’ve installed:

❯ pacman -Q fuse
fuse2 2.9.9-4
❯ pacman -Q fuse3
fuse3 3.10.5-1

If I mount for example a resource with sshfs it works fine, so FUSE is working fine.

Are you able to access any other snapshots in fuse mount? Does check complain about problems?

I just tried essentially the same steps (also on Arch) and that worked for me. What strikes me as odd is, that restic reports that it created a new cache. Is that message still shown if you run the mount command again? If yes please move the cache somewhere else and try whether that resolves the problem.

Hi Michael, I’ve tried another restic repo, done with a simple script I’ve written that makes a backup each week of the / of the machine. I’m using it for the two restic repo I’ve currently.

#!/bin/bash
restic -r /srv/restic-repo backup / --password-file /root/restic_passwd --exclude-file /root/exclude.dir
chown -R marco:marco /srv/restic-repo   # needed for Syncthing
restic -r /srv/restic-repo forget --password-file /root/restic_passwd --keep-daily 4 --prune

but with check both repos obtain some errors like:

pack a3621d6d: not referenced in any index
6402 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree c43a986d:
  id c43a986d8f0001fec580f07e4379e3eba11339c1f1a5d02a955c992ee3f1a252 not found in repository
error for tree fc0cbfb0:
  id fc0cbfb0a95c1c2c6fb461bc5016e05c946133f55268f81d9f0f67a5a3df15c2 not found in repository
error for tree 88d7631a:
  id 88d7631a98e433ad2214fa5bfc9a6792b0f4fc10bc2ffdc79686d0b75fa46e6e not found in repository
error for tree 457a9045:
  id 457a9045ac81bf216cf76cc82c72a812df3d8400b11702b91d7875194f485a3b not found in repository
[0:00] 100.00%  4 / 4 snapshots...
Fatal: repository contains errors

I’ve already tried to run a rebuild-index, but the errors persist.

The errors reported by check are likely the cause of the EIO errors thrown by mount.
Which errors does check report after running rebuild-index? The log above looks like the output before running rebuild-index.

pack a3621d6d: not referenced in any index

That warning shows that your repository has lost data, which is usually caused by the data storage loosing data (as you’re using a recent restic version and the snapshots are recent, this is the most likely cause). After fixing that problem, you can repair the repository by following the steps at Recover from broken pack file · Issue #828 · restic/restic · GitHub . However, that will likely lead to loosing some snapshots.

I’ve deleted the repo and rebuilt it again, now I can mount it correctly, thanks!

I haven’t understood why this data storage problem is happening, which could be the cause?

The cause depends a lot on where the repository is stored. Judging from the commands the repository is stored locally? Which filesystem is used there? Did anything unexpected happen? Maybe filesystem problems or a system crash? Or did restic fail?