I’m a long-time restic user and first time poster on this Discourse.
I recently migrated to a new workstation and used this as an opportunity to validate my backup process. I completed one final backup on the old machine and then restored from that snapshot on the new one. Everything was perfect until I invoked my backup script from the new machine and realized that a specific file that I expected to exist was missing.
My backup script first invokes restic --exclude-caches --verbose backup --exclude "$HOME/.cache" --exclude "$HOME/rpmbuild" "$HOME" and then uses rclone to persist the data to the cloud: rclone copy --immutable --progress --stats-one-line "$RESTIC_REPOSITORY" <<<CLOUD PATH>>>.
The backup step completed OK but the rclone failed. I quickly realized rclone was missing its ~/.config/rclone/rclone.conf configuration. I confirmed that rclone.conf wasn’t included in the snapshot but it does exist on the old machine:
> ls -laZ ~/.config/rclone
total 12
drwxrwxr-x. 2 bfallik bfallik unconfined_u:object_r:config_home_t:s0 4096 Aug 23 2020 .
drwx------. 34 bfallik bfallik unconfined_u:object_r:config_home_t:s0 4096 Sep 17 16:34 ..
-rw-------. 1 bfallik bfallik unconfined_u:object_r:config_home_t:s0 106 Aug 23 2020 rclone.conf
I can’t figure out why restic skipped rclone.conf and I’m a bit worried that some other files were missed during the backup+restore cycle. Both old and new PCs use restic version 0.12.0.
Any ideas for how I can debug this further? Thanks in advance!
Do you have a file named CACHEDIR.TAG in $HOME/.config/ or any of its subdirectories? You can check using find $HOME/.config -name CACHEDIR.TAG.
Another thing to check is of course the verbose output you get from the backup run. Make sure to look for restic saying it can’t read a file or similar.
I can re-run it but not against an identical configuration since I already moved the repo disk to the new workstation. Will see if I can reproduce the problem with -vvv and --dry-run and report back. Thx.
You don’t need the repo to do a dry run, as long as the old system is what it was it should be the same regardless where restic thinks it would store and access the repo.
When I run a test backup on the old machine the file ~/.config/rclone/rclone.conf is added to the repo so I’m dumbfounded why the file isn’t being backed up in my usual repo.
Differences for this test
backed up into an empty restic init repo
replaced --verbose with -vvv
only passed the dir ~/.config to backup instead of ~
Otherwise everything was the same as prior runs.
Could there be some configuration within the repo itself that is preventing this file from being archived?
No, not at all. Please run the “real” or “original” backup on the old host, but with -vvv and --dry-run after upgrading to 0.12.1. Please also show the output of the snapshots command, if you don’t mind.
FYI I don’t think --dry-run was included in 0.12.1. I had to download a more recent binary from https://beta.restic.net/ to find that flag.
❯ ./restic version
restic 0.12.1-dev (compiled manually) compiled with go1.17 on linux/amd64
Anyway, back on topic. I couldn’t easily re-run against the real repo on the old machine since I’ve already moved the repo disk. Instead I re-ran my backup script against the real repo on the new machine. Here’s how I invoked restic:
> ./restic --exclude-caches -vvv backup --dry-run --exclude "$HOME/.cache" --exclude "$HOME/rpmbuild" "$HOME" 1>restic.log 2>&1
...
> grep rclone restic.log
new /home/bfallik/.config/rclone/rclone.conf, saved in 0.000s (0 B added)
new /home/bfallik/.config/rclone/, saved in 0.000s (0 B added, 500 B metadata)
> ls -l /home/bfallik/.config/rclone/rclone.conf
-rw-------. 1 bfallik bfallik 106 Dec 14 20:37 /home/bfallik/.config/rclone/rclone.conf
It seems odd that restic seems to think that the file is new but then adds 0 B for it.
Here’s a list of the snapshots (I manually edited the hostnames for anonymity):
This can happen if restic didn’t find the right parent snapshot but the file contents has been saved before in some snapshot (it’s just one example, I reproduced it that way just now). I don’t see why restic wouldn’t find the correct parent snapshot since you use the same path all the time, but can you please complement/edit your post above so that all the output except the lines listing new/modified/ for each file is printed?
Also, let’s rewind a bit, can you show us the output of ./restic ls <theSnapshotThatYouRestored> | grep rclone? Which snapshot ID was that?
To your first, here is the log output with new/modified/unmodified info removed:
> grep -v new restic.log | grep -v unchanged | grep -v modified
open repository
lock repository
load index files
using parent snapshot 28bc65d1
start scan on [/home/bfallik]
start backup on [/home/bfallik]
scan finished in 4.425s: 116070 files, 14.648 GiB
Would add to the repo: 950.007 MiB
processed 116070 files, 14.662 GiB in 0:15
When I originally encountered this problem I used this simple script to mount the repo:
#!/bin/bash
set -e
set -o allexport
# shellcheck source=/dev/null
source ~/.bfallik-restic-repo.env
set +o allexport
TEMPDIR=$(mktemp -d --tmpdir restic-mount.XXXXXXXXXX)
function cleanup {
rmdir -v "${TEMPDIR}"
}
trap cleanup EXIT
restic mount "${TEMPDIR}"
I then looked for the data under the latest snapshot path.
ls /tmp/restic-mount.9Ne7BQXM3R/snapshots/latest/home/bfallik/.config/rclone
ls: cannot access '/tmp/restic-mount.9Ne7BQXM3R/snapshots/latest/home/bfallik/.config/rclone': No such file or directory
Prior in this thread I repeated this process with restic ls <snapshot> and that also failed to find the file using a specific snapshot ID.
Good question! Running the wildcard ls on the snapshots tree turned up that all of the snapshots but the most recent (from my new machine) contain the .config/rclone directory. This result helps clarify one detail that was incorrect in my original post. It seems that restic didn’t restore this file when I first transitioned to the new machine. The file not existing in the latest snapshot was a red herring since I did re-run the backup script once and that created a new snapshot (without rclone.conf, because that hadn’t been restored) before failing at the rclone step.
I don’t have great records from when I initiated the restore operation. It may just be the case that I forgot to restore this directory myself and didn’t realize that when debugging why rclone failed and also got confused because the file was missing from the most recent snapshot.
At this point I’m willing to chalk this up to operator error (PEBKAC!). Thanks for helping me investigate, @MichaelEischer and @rawtaz !
Ah, that explains a lot I think your analysis makes sense - with what you said above there’s IMO no sign that restic hasn’t backed that file up, and as you say you probably just missed it when you were using the mount to grab your files from the repository.
I think your analysis makes sense - with what you said above there’s IMO no sign that restic hasn’t backed that file up, and as you say you probably just missed it when you were using the mount to grab your files from the repository.