Full Server Restore on Ubuntu 20.04

Hello Restic-Forum,

I have been reading quite a few times posts in this Forum.
There are problems that I am currently facing. As stated, I am running Ubuntu 20.04.

After the server crashed, I wanted to make a Restore from Restic. I install a clean copy of Ubuntu (same Version) aswell as Restic. I input the credentials as I did previous and am able to see the snapshots.

Aside a few folders (found in this forum) I usually backup by
restic backup --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} /

So after the server started with the copy, I do restic restore ABCD1234 --target /
The server ignores a few files and then crashes completely and is not able to run again.

Please, has anyone an idea of how to restore this backup? There is only one drive mounted (Server). Access via SSH.

Kind regards and thanks for the help in advance.
Lukas

I restored a full Ubuntu 20.04 a while ago for testing, and this is what I excluded:

  • /etc/ssh to keep current SSH host keys
  • /boot if UUIDs of partitions (LVM) are not identical (probably the case when re-installing)
  • /etc/default/grub see above
  • /etc/fstab if UUIDs and/or partitions are not exactly the same
  • /etc/network/interfaces (Ubuntu seems to work around here to prevent accidents)
  • /etc/shadow if you don’t want to get passwords mixed up (take care to stay in sync with /etc/group and /etc/passwd!)
  • and possibly more…
    • /etc/machine-id could be another file to exclude, not sure though.

root ~# restic restore -t / -e /etc/ssh -e /boot -e /etc/default/grub -e /etc/fstab latest

Hello wn1337, thanks for your response.
I sadly still have problems. I read about offline restore, which I could not find more information on.
I tried using your info provided and got following output. After a few seconds the server outputs “Remote side unexpectedly closed network connection”.

ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/dbus-daemon: open /usr/bin/dbus-daemon: text file bu                                                                                                                                                             sy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/system.jour                                                                                                                                                             nal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/system.journal: no s                                                                                                                                                             uch file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/system.jour                                                                                                                                                             nal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/system.journal: no s                                                                                                                                                             uch file or directory
ignoring error for /usr/bin/bash: open /usr/bin/bash: text file busy
ignoring error for /usr/bin/bash: open /usr/bin/bash: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/restic: open /usr/bin/restic: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/python3.8: open /usr/bin/python3.8: text file busy
ignoring error for /usr/bin/perl: open /usr/bin/perl: text file busy
ignoring error for /usr/bin/perl: open /usr/bin/perl: text file busy
ignoring error for /usr/bin/perl: open /usr/bin/perl: text file busy
ignoring error for /usr/bin/perl: open /usr/bin/perl: text file busy
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.j                                                                                                                                                             ournal: open /var/log/journal/3571fb8951f2400b9db3be002797f0e7/user-2001.journal                                                                                                                                                             : no such file or directory
ignoring error for /usr/lib/accountsservice/accounts-daemon: open /usr/lib/accou                                                                                                                                                             ntsservice/accounts-daemon: text file busy
ignoring error for /usr/lib/policykit-1/polkitd: open /usr/lib/policykit-1/polki                                                                                                                                                             td: text file busy
ignoring error for /usr/lib/systemd/systemd: open /usr/lib/systemd/systemd: text                                                                                                                                                              file busy
ignoring error for /usr/lib/systemd/systemd: open /usr/lib/systemd/systemd: text                                                                                                                                                              file busy
ignoring error for /usr/lib/systemd/systemd: open /usr/lib/systemd/systemd: text                                                                                                                                                              file busy
ignoring error for /usr/lib/systemd/systemd-journald: open /usr/lib/systemd/syst                                                                                                                                                             emd-journald: text file busy
ignoring error for /usr/lib/systemd/systemd-logind: open /usr/lib/systemd/system                                                                                                                                                             d-logind: text file busy
ignoring error for /usr/lib/systemd/systemd-networkd: open /usr/lib/systemd/syst                                                                                                                                                             emd-networkd: text file busy
ignoring error for /usr/lib/systemd/systemd-networkd: open /usr/lib/systemd/syst                                                                                                                                                             emd-networkd: text file busy
ignoring error for /usr/lib/systemd/systemd-networkd: open /usr/lib/systemd/syst                                                                                                                                                             emd-networkd: text file busy
ignoring error for /usr/lib/systemd/systemd-networkd: open /usr/lib/systemd/syst                                                                                                                                                             emd-networkd: text file busy
ignoring error for /usr/lib/systemd/systemd-resolved: open /usr/lib/systemd/syst                                                                                                                                                             emd-resolved: text file busy
ignoring error for /usr/lib/systemd/systemd-timesyncd: open /usr/lib/systemd/sys                                                                                                                                                             temd-timesyncd: text file busy
ignoring error for /usr/lib/systemd/systemd-udevd: open /usr/lib/systemd/systemd                                                                                                                                                             -udevd: text file busy
ignoring error for /usr/lib/systemd/systemd-udevd: open /usr/lib/systemd/systemd                                                                                                                                                             -udevd: text file busy
ignoring error for /usr/lib/udisks2/udisksd: open /usr/lib/udisks2/udisksd: text                                                                                                                                                              file busy

Do you know what might help here? I am very much struggling in this matter.
Many regards,
Lukas

** Adding to that, I did try it with a tutorial listed on here, but the rescue system is not able to securely download the restic package neither execute it. Another offline use of the backup wasn’t found.

You have to boot from a live system, mount the partition of the current server installation and then restore to it. While the system is running it won’t be possible to replace everything.

Which rescue system did you use?

Hello @MichaelEischer,
I am using the provided GRML Rescue-Boot.
From a clean installation of Ubuntu 20.04, the server installation is already mounted, would you be able to clarify?

That remark referred to the live system (GRML in your case). That is you’ll have to run GRML, mount the partition to restore data to, then install restic by e.g. installing it via the package manager or downloading it from Github. And afterwards restore the data to the mounted partition. If any of these steps doesn’t work, we’ll need more details about the specific problem.

Hello,
I checked with fdisk -l what Partitions are mounted, and happily, the partition was correctly there.
The usual way of apt-get did not work I did get the files from restic via git clone. Also go could be installed correctly with version.
Sadly after trying to run build.go I get the error " ./build.go:181 undefined os.ReadFile".
The files are there, go is running, another way of installing is not possible as it seems. Apt cannot update because it declares the source as insecure.

So, I cannot install the usual way because of the insecure source and with go, the installer build.go throws an error. :exploding_head:

Hello @MichaelEischer,
I figured out a way to solve that. I booted from a newer version of GRML which then worked.
After solving a few other problems, I am standing in front of my (hopefully) last problem, the automatic backup.

By now I put the Crontab Value as following.

*/5 * * * * /usr/bin/restic backup --force --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} / >> /var/log/syslog 2>&1

I added --force after I always got the Error “using parent snapshot …”
Now, I get “no parent snapshot found, will read all files”. Which seems like it should run. I wait a few minutes, but when I check restic snapshots, I do not get any new snapshot from the Cron Backup.

All the Environmental Strings are located in /etc/environment, and because the using parent snapshot could read the last made Snapshots ID the connection seems to work.

Do you have any idea on this last bit?

Many regards,
Lukas

** The connection works. It uses the parent snapshot.
Cron-Log

Load(<lock/b699f83434>, 0, 0) returned error, retrying after 7.649837917s: The specified key does not exist.
Load(<lock/b699f83434>, 0, 0) returned error, retrying after 15.394871241s: The specified key does not exist.
using parent snapshot dedc30df
error: read /proc/1/attr/apparmor/exec: invalid argument
error: read /proc/1/attr/apparmor/prev: invalid argument
error: read /proc/1/attr/exec: invalid argument
error: read /proc/1/attr/fscreate: invalid argument
error: read /proc/1/attr/keycreate: invalid argument
error: read /proc/1/attr/prev: invalid argument
error: read /proc/1/attr/smack/current: invalid argument
error: read /proc/1/attr/sockcreate: invalid argument
error: read /proc/1/task/1/attr/smack/current: invalid argument
error: read /proc/1/task/1/attr/apparmor/exec: invalid argument
error: read /proc/1/task/1/attr/apparmor/prev: invalid argument
error: read /proc/1/task/1/attr/exec: invalid argument
error: read /proc/1/task/1/attr/fscreate: invalid argument
error: read /proc/1/task/1/attr/keycreate: invalid argument
error: read /proc/1/task/1/attr/prev: invalid argument
error: read /proc/1/task/1/attr/sockcreate: invalid argument

Even though proc is excluded. It does not make any sense for me.

I don’t see a reason why you’d need to pass --force to restic, quite the contrary: it will just slow down you backups. The “using parent snapshot” is just an information, not an error.

Did the backup run complete? Does the command work if you run it manually?

Running the backup every five minutes seems a bit excessive. If a backup run is slower than usual, there is also nothing which prevents multiple concurrent backups from piling up.

Btw, directly appending to the syslog is a very bad idea, that will garble the log file if multiple process want to add entries to the syslog. The output from restic should end up in a separate logfile.

Or even better: please take a look around the forum on how to setup restic as a systemd service. That would fix both previous problems, although sending an email if the backup fails might be more complex.

Regarding the read errors, my guess is that the --exclude={/dev,/proc,...} syntax is not expanded by cron as you’d expect. The simplest way is to just create an exclude file and then specify that on the command line. That way you avoid problems with the parameter interpolation.

Please just create a wrapper script and set the environment variables there. There’s not point in polluting the environment across the whole system.

That error shouldn’t occur unless there’s more than one restic process running at the same time.

Hello Michael,
thank you for your response. I will take a look into building a wrapper and making restic act as systemd.

This was for testing purposes, I did plan (if it works in the end), making a backup once every hour and one daily or something along this.

Also I removed the --force flag.

This is the main question I am standing in front. If I run exactly that command, it works perfectly. I can see it in the restic snapshots list and could work with it. With the cron I only get the “using parent snapshot” as information along with the last snapshot ID and thats it. I am quite unsure if there is someting I am missing. I do not get any other error aside the information on the parent.

I cannot see it in the snapshots list.

I think you have to fix the logging of the cron job first. Without the log output, there’s no way of telling what goes wrong.

Since I was just going through my backup in order to install the Restic-Solution on a new server here is the solution for any future readers that worked for me, for the last 4+ months.

0 2 * * * /usr/bin/restic backup -r REPOSITORY -e /dev -e /media -e /mnt -e /proc -e /run -e /sys -e /tmp -e /var/tmp / >> /var/log/restic.log 2>&1