Sorry for the length of this. I have done my best to provide key information while reducing it to a minimum.
I use restic twice daily to make backups of my personal account on a recently purchased System76 laptop.
System software:
Operating System: Kubuntu 20.04
KDE Plasma Version: 5.18.5
KDE Frameworks Version: 5.68.0
Qt Version: 5.12.8
Kernel Version: 5.11.0-41-generic
OS Type: 64-bit
System hardware:
Processors: 8 × 11th Gen Intel® Core™ i7-1165G7 @ 2.80GHz
Memory: 15.5 GiB of RAM
System storage - 500 GB PCIe Gen4 SSD
Backup USB storage - Seagate 1TB HD
I am having a strange problem that I’ve seen before. It happened last night: My backup with restic was clearly in trouble, so I terminated it with Ctrl+C. I then found several other running programs to be locked up. I did a hard reboot and couldn’t get past the account sign-in screen. I fired up my Kubuntu ISO DVD, looked at the backup USB storage and system storage, and couldn’t see any problem I could recognize. Needing a working computer today, I reloaded the OS (/home is in a separate partition), updated it, and took off.
As last night’s backup has not completed, I reran it rather soon this afternoon:
- In a Konsole terminal, I issued ‘‘cd /home/tomc/bkup && sudo sh backup-all3 >> /home/tomc/programs/restic-backup/restic-backup-scripts/backup-all3.log 2 >&1’’
‘‘backup-all3’’ is a script that calls 7 other scripts in succession, each backing up a portion of my user account. The last 3 scripts called are:
bk-hid3a
bk-hid3b
bk-hid3c
This afternoon, ‘‘backup-all3’’ ran to completion, but the last two components of the ‘‘backup-all3’’ script - bk-hid3b & bk-hid3c - appeared to do nothing.
- Looking at my ‘‘backup-all3.log’’: All goes well until bk-hid3b is invoked, at which point restic appears to be doing nothing.
Here are the relevant lines extracted from the collection of scripts in ‘‘backup-all3’’:
- The backup script run before bk-hid3a: restic -p /home/tomc/Dropbox/ssap -r /media/tomc/seagate-4-b/sys76-backup/remainders backup /home/tomc/.thunderbird /home/tomc/CmapToolsLogs /home/tomc/Desktop /home/tomc/Documents /home/tomc/Downloads /home/tomc/Library /home/tomc/My_Cmaps /home/tomc/PDF /home/tomc/programs /home/tomc/R /home/tomc/software_archive /home/tomc/Videos
- Invocation of bk-hid3a: restic -p /home/tomc/Dropbox/ssap -r /media/tomc/seagate-4-b/sys76-backup/bk-hid3a backup /home/tomc/.recoll /home/tomc/.local
- Invocation of bk-hid3b: restic -p /home/tomc/Dropbox/ssap -r /media/tomc/seagate-4-b/sys76-backup/bk-hid3b backup /home/tomc/.config /home/tomc/.mozilla /home/tomc/.rustup /home/tomc/snap /home/tomc/.atom /home/tomc/.var /home/tomc/.googleearth /home/tomc/digikam_db /home/tomc/.vscode-oss /home/tomc/.cargo /home/tomc/.kde /home/tomc/gems /home/tomc/.kite
- Invocation of bk-hid3c: restic -p /home/tomc/Dropbox/ssap -r /media/tomc/seagate-4-b/sys76-backup/bk-hid3c backup /home/tomc/ --exclude-file=/home/tomc/programs/restic-backup/restic-backup-scripts/restic-excludes-hiddensPlus.txt
Here is the end of the backup-all3.log:
.
.
.
----------------------------------------------------------
||||| bk-rem3 - typical run time <= 0:05:00 ||||||||||||||
Mon Dec 13 13:57:06 PST 2021
=== remainders - restic backup - ver. 4: 2021-10-18 for Sys76 ===
no parent snapshot found, will read all files
Files: 81366 new, 0 changed, 0 unmodified
Dirs: 7323 new, 0 changed, 0 unmodified
Added to the repo: 136.691 MiB
processed 81366 files, 45.604 GiB in 1:46
snapshot 7bc596f0 saved
=== END restic backup of remainders ===
----------------------------------------------------------
||||| bk-hid3a - typical run time <= 0:20:00 ||||||
Mon Dec 13 13:58:53 PST 2021
=== hiddensPlus-a - restic backup - ver. 5: 2021-12-11 for Sys76 ===
no parent snapshot found, will read all files
Files: 46761 new, 0 changed, 0 unmodified
Dirs: 3995 new, 0 changed, 0 unmodified
Added to the repo: 2.436 GiB
processed 46761 files, 29.873 GiB in 1:36
snapshot 768c67e6 saved
=== END restic backup bk-hid3a ===
----------------------------------------------------------
||||| bk-hid3b - typical run time <= 0:07:00 ||||||
Mon Dec 13 14:02:11 PST 2021
=== hiddensPlus-b - restic backup - ver. 5: 2021-12-11 for Sys76 ===
=== END restic backup bk-hid3b ===
----------------------------------------------------------
||||| bk-hid3c - typical run time <= 0:00:01 ||||||
Mon Dec 13 14:02:11 PST 2021
=== hiddensPlus-c - restic backup - ver. 5: 2021-12-11 for Sys76 ===
=== END restic backup bk-hid3c ===
||||||||||||||||||||||||||||||||||||||||||||||||
Mon Dec 13 14:02:11 PST 2021
>>>>> END restic backup of all repositories <<<<<
================================================
================================================
Here is my Konsole terminal output:
tomc@tomc-Galago-Pro:~/bkup$ cd /home/tomc/bkup && sudo sh backup-all3 >> /home/tomc/programs/restic-backup/restic-backup-scripts/backup-all3.log 2 >&1
Save(<data/030ba8658e>) returned error, retrying after 552.330144ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/03/030ba8658ea002a590f6bc42ba8b5e3b51d542c3b30cdc246422bb5e55d00aab: input/output error
Save(<data/52f46dc508>) returned error, retrying after 582.280027ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/52/52f46dc508ad085dea0acd818389898583b9e231905e26f3e304ff6b5ec8ca2f: input/output error
Save(<data/ec9af993ed>) returned error, retrying after 468.857094ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/ec/ec9af993ed69ed5b5f01dd71d35ef2ae58813cbadcb36e35b8590095e27ca68a: input/output error
Save(<data/5290ca1f01>) returned error, retrying after 462.318748ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/52/5290ca1f01eca0a13c6588813b9c288298fb3c666646718081973bdc0515df22: input/output error
Save(<data/f8c6065abf>) returned error, retrying after 720.254544ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/f8/f8c6065abf11fad6721ffc39942cc2100d25c40941cc16e750849e5d85bc7f27: input/output error
Save(<data/252092b3ee>) returned error, retrying after 593.411537ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/25/252092b3eef1856720c4e815affcae8a03b9eda98dbc09beff37a4fb69f69371: input/output error
Save(<data/5ad6d29bf0>) returned error, retrying after 282.818509ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/5a/5ad6d29bf0a1e40262b2f8a642668b9b75486db03c5baf106c97ca6dcfb30ff8: read-only file system
Save(<data/ccef269659>) returned error, retrying after 328.259627ms: sync /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/data/cc/ccef2696598ea96b5b06350d5ab0bde00e24bda01b1737532dc283d2b379dbbe: read-only file system
Remove(<lock/b7d774c990>) returned error, retrying after 298.484759ms: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 600.683895ms: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 1.142114207s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 2.216767434s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 1.807980427s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 3.343745266s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 4.659096946s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 8.277195667s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 10.034154425s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 15.244749861s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
error while unlocking: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directoryRemove(<lock/b7d774c990>) returned error, retrying after 589.542338ms: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 538.914789ms: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 791.085236ms: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 1.452720516s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 2.71014173s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 5.173209677s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 4.517033471s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 6.809451431s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 16.05103845s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Remove(<lock/b7d774c990>) returned error, retrying after 13.581705599s: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
error in cleanup handler: chmod /media/tomc/seagate-4-b/sys76-backup/bk-hid3a/locks/b7d774c99008c7fe761ef7ead3b2cc6785e7c1abb73831a85b30a533e0ed7896: no such file or directory
Fatal: unable to open config file: stat /media/tomc/seagate-4-b/sys76-backup/bk-hid3b/config: no such file or directory
Is there a repository at the following location?
/media/tomc/seagate-4-b/sys76-backup/bk-hid3b
bk-hid3b: 10: a: not found
Fatal: unable to open config file: stat /media/tomc/seagate-4-b/sys76-backup/bk-hid3c/config: no such file or directory
Is there a repository at the following location?
/media/tomc/seagate-4-b/sys76-backup/bk-hid3c
bk-hid3c: 9: a: not found
tomc@tomc-Galago-Pro:~/bkup$
My Backup USB storage HD is partitioned thus (KDS partition manager output):
##|v1|## partition table of /dev/sda
# on Monday, 13 December 2021 20:22:43 PST
type: "msdos"
align: "sector"
# number start end type roles label flags
1;2048;409602047;ext4;primary;"seagate-4-a";"" [195 GiB]
2;409602048;1953519615;ext4;primary;"seagate-4-b";"" [736 GiB]
Before the backup run there were these mountpoints:
/media/tomc/seagate-4-a
/media/tomc/seagate-4-b
My backup target repositories are in seagate-4-b.
After today’s backup run I see these:
tomc@tomc-Galago-Pro:/media/tomc$ ls -l
.
.
drwx------ 3 root root 4096 Dec 13 13:59 seagate-4-b
drwxr-xr-x 5 root root 4096 Nov 8 20:23 seagate-4-b1
tomc@tomc-Galago-Pro:/media/tomc$
seagate-4-b contains ONLY the backup produced today by bk-hid3a - Its size is 58 GiB, and it’s 24% used
seagate-4-b1 mystifies me - I didn’t manually create it - is size is 723 GiB, and it’s 46% used; it contains:
root@tomc-Galago-Pro:/media/tomc/seagate-4-b1/sys76-backup# ls -l
total 28
drwx------ 7 root root 4096 Dec 11 16:30 bk-hid3a
drwx------ 7 root root 4096 Dec 11 16:30 bk-hid3b
drwx------ 7 root root 4096 Dec 11 16:30 bk-hid3c
drwxr-xr-x 7 root root 4096 Dec 11 12:19 dropbox
drwx------ 7 root root 4096 Oct 13 23:35 music
drwx------ 7 root root 4096 Oct 13 23:36 pictures
drwx------ 7 root root 4096 Nov 3 18:28 remainders
root@tomc-Galago-Pro:/media/tomc/seagate-4-b1/sys76-backup#
That list is apparently alphabetical by repository name. All repositories EXCEPT the “bk-hid…” one contain legitimate updates from today’s backup run. The “bk-hid…” repositories are not updated.
QUESTION #1: How did the new seagate-4-b1 mountpoint get created? Why are some of my repositories moved to it? My repositories, originally created in one place, are now in two. I have no assurance that if I edit the scripts to point to the new locations that this won’t happen again…
QUESTION #2: How do I fix this?
This problem has happened on at least 2 prior occasions. This time I want to actually understand the problem and address it for future runs. But I need expert help.
=====
Note that -
- any references to “Dropbox” above are merely to a directory formerly connected to the Dropbox service; and
- I “sudo” my meta-script merely because the first element updates restic. Pursuant to a suggestion made to me on an earlier thread, I will in the future isolate that usage so it pertains only to the restic update.