Can't browse mounted repo

Getting Help
1

Hi, I’m trying to mount a repo in order to browse it, I’ve mounted it with the command:

restic -r repo_name mount mnt_ssh
enter password for repository: 
repository bc991fef opened successfully, password is correct
created new cache in /home/marco/.cache/restic
Now serving the repository at mnt_ssh
Use another terminal or tool to browse the contents of this folder.
When finished, quit with Ctrl-c here or umount the mountpoint.

then if I try to browse the mounted dir I see the four directories: hosts ids snapshots tags, but I can’t browse the snapshot itself, because I get an input/output error:

❯ ls mnt_ssh/snapshots/
2022-04-08T03:00:01+02:00  2022-04-22T03:00:02+02:00  latest
2022-04-15T03:00:01+02:00  2022-04-29T03:00:01+02:00

❯ ls mnt_ssh/snapshots/latest
ls: lettura della directory 'mnt_ssh/snapshots/latest': Errore di input/output

❯ ls mnt_ssh/snapshots/2022-04-29T03:00:01+02:00
ls: lettura della directory 'mnt_ssh/snapshots/2022-04-29T03:00:01+02:00': Errore di input/output

What should I do?

Thanks!
Marco

2

Hm, that’s odd. You did let restic run after the mount, right? Is the user running restic the same as the user browsing the snapshots?

3

Yes I let restic run after the mount, the user is the same (two different machines). I’ve tried also by root, but nothing.

4

Which restic version are you using? And which operating system?

5

I’m using Archlinux, Restic 0.13. The repo has been created with Restic Arm 0.12 (rpi).

6

Please show us the output of the (regular Linux command) mount before you mount the repository.

Please also try mounting it at /tmp/foobar or similar subdirectory under /tmp, but of course after first running mkdir /tmp/foobar (or similar, correspondingly).

7 
❯ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=3999344k,nr_inodes=999836,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on / type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14820)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=4009568k,nr_inodes=1048576,inode64)
/dev/sda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=801912k,nr_inodes=200478,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Same error mounting under /tmp:

❯ ls /tmp/foobar/snapshots/2022-04-29T03:00:01+02:00
ls: lettura della directory '/tmp/foobar/snapshots/2022-04-29T03:00:01+02:00': Errore di input/output

After mounting restic repo mount command adds:

restic on /tmp/foobar type fuse (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)
8

Which FUSE do you have installed?

Also, can you please try this with e.g. restic 0.12 and 0.11 as well to see if it makes a difference?

Finally, have you tried FUSE mounting something else than a restic repository on this system, and if so was it successful?

9

Hi, I’ve tried both Restic 0.12.1 and 0.11.0 but there’s no difference, same error of input/ourput.

On my Archlinux machine I’ve installed:

❯ pacman -Q fuse
fuse2 2.9.9-4
❯ pacman -Q fuse3
fuse3 3.10.5-1

If I mount for example a resource with sshfs it works fine, so FUSE is working fine.

10

Are you able to access any other snapshots in fuse mount? Does check complain about problems?

I just tried essentially the same steps (also on Arch) and that worked for me. What strikes me as odd is, that restic reports that it created a new cache. Is that message still shown if you run the mount command again? If yes please move the cache somewhere else and try whether that resolves the problem.

11

Hi Michael, I’ve tried another restic repo, done with a simple script I’ve written that makes a backup each week of the / of the machine. I’m using it for the two restic repo I’ve currently.

#!/bin/bash
restic -r /srv/restic-repo backup / --password-file /root/restic_passwd --exclude-file /root/exclude.dir
chown -R marco:marco /srv/restic-repo   # needed for Syncthing
restic -r /srv/restic-repo forget --password-file /root/restic_passwd --keep-daily 4 --prune

but with check both repos obtain some errors like:

pack a3621d6d: not referenced in any index
6402 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree c43a986d:
  id c43a986d8f0001fec580f07e4379e3eba11339c1f1a5d02a955c992ee3f1a252 not found in repository
error for tree fc0cbfb0:
  id fc0cbfb0a95c1c2c6fb461bc5016e05c946133f55268f81d9f0f67a5a3df15c2 not found in repository
error for tree 88d7631a:
  id 88d7631a98e433ad2214fa5bfc9a6792b0f4fc10bc2ffdc79686d0b75fa46e6e not found in repository
error for tree 457a9045:
  id 457a9045ac81bf216cf76cc82c72a812df3d8400b11702b91d7875194f485a3b not found in repository
[0:00] 100.00%  4 / 4 snapshots...
Fatal: repository contains errors

I’ve already tried to run a rebuild-index, but the errors persist.

12

The errors reported by check are likely the cause of the EIO errors thrown by mount.
Which errors does check report after running rebuild-index? The log above looks like the output before running rebuild-index.

pack a3621d6d: not referenced in any index

That warning shows that your repository has lost data, which is usually caused by the data storage loosing data (as you’re using a recent restic version and the snapshots are recent, this is the most likely cause). After fixing that problem, you can repair the repository by following the steps at Recover from broken pack file · Issue #828 · restic/restic · GitHub . However, that will likely lead to loosing some snapshots.

13

I’ve deleted the repo and rebuilt it again, now I can mount it correctly, thanks!

I haven’t understood why this data storage problem is happening, which could be the cause?

14

The cause depends a lot on where the repository is stored. Judging from the commands the repository is stored locally? Which filesystem is used there? Did anything unexpected happen? Maybe filesystem problems or a system crash? Or did restic fail?

15

Hi, sorry for the delay for the answer, I deleted the repository and after the new rebuilding it worked fine, but after a month I’ve seen today that the error appears again.

I’ve setup a weekly backup and a pruning rule to keep the latest 4 snapshots, so last week the fifth weekly backup has pruned the oldest snapshot but something has corrupted the repository itself again.

I’m using a simple script in crontab scheduled each week with this command to clean up the snapshots:

restic -r /srv/restic-repo forget --password-file /dir/restic_passwd --keep-last 4 --prune

Is it wrong?

Thanks again!

16

The command is ok.

What does check report? Which error did forget --prune report?

Judging from the commands the repository is stored locally? Which filesystem is used there? Did anything unexpected happen? Maybe filesystem problems or a system crash? Or did restic fail?

17

Yes the repo is stored locally and then synched to another host through Syncthing. I’m using ext4 as FS.

The check reports these errors:

pack 9979efe9: not referenced in any index
pack f26c124f: not referenced in any index
pack 6685dba1: not referenced in any index
pack 9c48b3bb: not referenced in any index
pack 7317babc: not referenced in any index
pack 6381f466: not referenced in any index
pack 6b6749f4: not referenced in any index
pack 925b8698: not referenced in any index
pack 12d0425a: not referenced in any index
pack bd8e660b: not referenced in any index
pack 0cabbe2a: not referenced in any index
pack 40919e5d: not referenced in any index
pack 4744c542: not referenced in any index
pack aa0d343f: not referenced in any index
pack d57e17bc: not referenced in any index
pack 3a97b038: not referenced in any index
pack 66db1de1: not referenced in any index
pack aed0c0db: not referenced in any index
pack 0bfe39c6: not referenced in any index
pack fc9740a1: not referenced in any index
pack 89d02443: not referenced in any index
pack 9f26c37e: not referenced in any index
pack d45c6998: not referenced in any index
pack e3213c94: not referenced in any index
pack 0e086db1: not referenced in any index
pack 7d754b69: not referenced in any index
pack b0656b09: not referenced in any index
pack 6be24b28: not referenced in any index
pack 4ddc7953: not referenced in any index
pack cc5650e3: not referenced in any index
pack 434b23c8: not referenced in any index
pack ee0ad5f3: not referenced in any index
pack 395cb7a3: not referenced in any index
pack 60b7d5a5: not referenced in any index
pack 3bf7f870: not referenced in any index
pack 652aa869: not referenced in any index
pack 79f341c7: not referenced in any index
pack 12ee0054: not referenced in any index
pack a5f7dd90: not referenced in any index
pack 5bb0e364: not referenced in any index
pack b3663fc7: not referenced in any index
pack f30684a7: not referenced in any index
pack fb682b9d: not referenced in any index
pack 308386fb: not referenced in any index
6268 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
error for tree c31695af:ots...
  id c31695afc367187ce194127001481d07107320c7760ac2286ed473ec16d1f307 not found in repository
error for tree 0af51dae:
  id 0af51dae7652457de458dd2761fa9b105962c5cf4471a4230d11bffc04011fc1 not found in repository
error for tree 3fa81711:
  id 3fa81711c39ac1a04b1cb80c6f5d84fd08058e7954728596118f2c4d511a9484 not found in repository
error for tree cea3e202:
  id cea3e2024664b40e3f7169c1d0cebcb671765b8d69341d7a345d7df1e1582ec5 not found in repository
[0:00] 100.00%  4 / 4 snapshots...
Fatal: repository contains errors
18

Did any of the backup run report an error or were all of the successful? Running restic rebuild-index it could be possible to fix most of the problems, but the main question is why your repository is loosing data. The damage sort of looks like something (not restic!) deleted files at least in the index/ folder, but maybe also somewhere else.

Are you running check on the host that created the backup or on the sync target?

19

Hi, I’ve found that on the local machine the check finishes succesfully, while on the other host with the repo synced through Synchthing the repo fails the check process.

20

There are a few things you can check: do the repositories on both hosts contain the same files? Does the file content of each files match it’s sha256 hash?