Restic copy of repo crashed - now root dir is full - what to do?

1

At this time, if I attempt a reboot, it will fail. Here is how I got here - I am trying to copy a repo from my USB backup HD to a second USB HD.

CLI output:

tomc@sys76:~/programs/restic-backup/restic-backup-scripts$ sudo restic -p /home/tomc> /Dropbox/ssap init --repo /media/tomc/THD3/music init --repo2 /media/tomc/seagate-4-b/sys76-backup/music --copy-chunker-params
enter password for secondary repository:
repository c6af3af6 opened successfully, password is correct
created restic repository 49012a320e at /media/tomc/THD3/music

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
tomc@sys76:~/programs/restic-backup/restic-backup-scripts$ sudo restic -p /home/tomc/Dropbox/ssap -r /media/tomc/seagate-4-b/sys76-backup/music copy -p /home/tomc/Dropbox/ssap --repo2 /media/tomc/THD3/music
enter password for destination repository:
repository c6af3af6 opened successfully, password is correct
repository 49012a32 opened successfully, password is correct
created new cache in /root/.cache/restic

snapshot b18ba100 of [/home/tomc/Music] at 2021-10-23 09:23:59.13324564 -0700 PDT)
copy started, this may take a while…
LoadBlob(669a261f0710569658f579b8bdd4a55027fa11e3f542162ee7bab37d8f719e56) returned error decrypting blob 669a261f0710569658f579b8bdd4a55027fa11e3f542162ee7bab37d8f719e56 failed: ciphertext verification failed
tomc@sys76:

I have 5 repos, so I decided to move on to the next, the ask in this forum what to do about the above error. Bad idea; it didn’t go well…

tomc@sys76:~/programs/restic-backup/restic-backup-scripts$ sudo restic -p /home/tomc/Dropbox/ssap init --repo /media/tomc/THD3/pictures init --repo2 /media/tomc/seagate-4-b/sys76-backup/pictures --copy-chunker-params
[sudo] password for tomc:
enter password for secondary repository:
repository 029582b9 opened successfully, password is correct
created restic repository 892bb91347 at /media/tomc/THD3/pictures

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
tomc@sys76:~/programs/restic-backup/restic-backup-scripts$ sudo restic -p /home/tomc/Dropbox/ssap -r /media/tomc/seagate-4-b/sys76-backup/pictures copy -p /home/tomc/Dropbox/ssap --repo2 /media/tomc/THD3/pictures
enter password for destination repository:
repository 029582b9 opened successfully, password is correct
repository 892bb913 opened successfully, password is correct
created new cache in /root/.cache/restic

snapshot ec883add of [/home/tomc/Pictures] at 2021-11-07 09:09:51.832951885 -0800 PST)
copy started, this may take a while…
SaveBlob(a27da695cfb60276f322f2519b059e183f9150c5317594e0e669b46ac1ba4fdf) returned error write /media/tomc/THD3/pictures/data/58/5856397f3727ce9a85c6be507b6b9e5c64b73f635bb16cccc869f7149768a9c2: no space left on device
tomc@sys76:~/programs/restic-backup/restic-backup-scripts$ dfc
FILESYSTEM (=) USED FREE (-) %USED AVAILABLE TOTAL MOUNTED ON
udev [--------------------] 0.0% 7.7G 7.7G /dev
tmpfs [=-------------------] 0.1% 1.5G 1.5G /run
/dev/nvme0n1p2 [====================] 100.0% 0.0B 58.5G /
tmpfs [=-------------------] 0.9% 7.7G 7.7G /dev/shm
tmpfs [=-------------------] 0.1% 5.0M 5.0M /run/lock
tmpfs [--------------------] 0.0% 7.7G 7.7G /sys/fs/cgroup
/dev/loop2 [====================] 100.0% 0.0B 144.6M /snap/chromium/1810
/dev/loop1 [====================] 100.0% 0.0B 144.6M /snap/chromium/1801
/dev/loop3 [====================] 100.0% 0.0B 55.5M /snap/core18/2128
/dev/loop0 [====================] 100.0% 0.0B 128.0K /snap/bare/5
/dev/loop4 [====================] 100.0% 0.0B 55.5M /snap/core18/2246
/dev/nvme0n1p1 [==========----------] 47.0% 271.0M 511.0M /boot/efi
/dev/loop5 [====================] 100.0% 0.0B 164.9M +nap/gnome-3-28-1804/161
/dev/loop6 [====================] 100.0% 0.0B 65.2M +/gtk-common-themes/1519
/dev/loop7 [====================] 100.0% 0.0B 170.1M /snap/signal-desktop/378
/dev/nvme0n1p4 [===========---------] 54.1% 175.9G 383.5G /home
/dev/loop8 [====================] 100.0% 0.0B 170.0M /snap/signal-desktop/377
/dev/loop9 [====================] 100.0% 0.0B 32.5M /snap/snapd/13640
/dev/loop10 [====================] 100.0% 0.0B 42.2M /snap/snapd/13831
tmpfs [=-------------------] 0.0% 1.5G 1.5G /run/user/1000
/dev/sdb1 [==------------------] 5.1% 434.1G 457.4G /media/tomc/THD31
/dev/sda1 [====================] 95.4% 8.7G 191.2G /media/tomc/seagate-4-a
/dev/sda2 [===========---------] 54.7% 327.9G 723.6G /media/tomc/seagate-4-b
tomc@sys76:~/programs/restic-backup/restic-backup-scripts$

Note two things:

“/dev/sdb1 [==------------------] 5.1% 434.1G 457.4G /media/tomc/THD31” - My target USB HD mountpoint shows an almost full drive. That’s absurd. All 5 of my repos combined should take up about half that drive.

" /dev/nvme0n1p2 [====================] 100.0% 0.0B 58.5G /" - That is a 58G root dir that is now full.

This exact problem happened earlier today. I tried to reboot, thinking that the junk (I assume) in root might get automatically cleared. Nope. I couldn’t boot. So, I thought I needed a larger root dir, and I took some real time to run gparted and made it almost 50% bigger. Now here I am again with exactly the same problem.

FIRST - how do I get my root dir fixed? With a partition of this size, the OS should normally take up less than half the partition.

SECOND - what is going on with the problem encountered when I tried to copy /media/tomc/seagate-4-b/sys76-backup/music? and what is proper way to recover from such an error?

I am well and truly stuck at this point. All suggestions are welcome!

==== UPDATE - about 1 hour later ====

It occurred to me to LOOK at my root dir to see what might be there. An attempt to clear restic cache had accomplished nothing, so I had assumed earlier that it would be some sort of fragment of something or other.

Wrong. I found a file “THD3” occupying all the free space. Deleting that fixed my full root dir problem.

That deals with my FIRST question above.

Now… I’m left with the SECOND.

I will hope for an answer to that query. While waiting, I’m going to run my weekly maintenance script - clean cache, remove locks, prune repos, check with --read-data.

2

Why is this both a file taking up space, and it appears to be a directory as well?

Also curious why you’re running restic in sudo. Seems for “user level” backup of media that wouldn’t be necessary, and might also prevent things from having root access where you might not want it to.

Wondering also if messing around with something Dropbox is controlling is a good idea. (I’m a bit gunshy with Dropbox, so this is probably an irrational line of inquiry.)

3

The repository /media/tomc/seagate-4-b/sys76-backup/music seems to be slightly damaged. Please run restic check --read-data to get a full assessment of the damage. The repair steps will then probably be something like those outlined in Recover from broken pack file · Issue #828 · restic/restic · GitHub .