I have a personal server with SSD Nodes and recently decided I’d like to move the instance to another DC. They stated they can spin up the instance but that all the data will be erased. The volume cannot be moved to the new DC.
I am currently backing up with Restic but am only doing specific directory/docker volume backups. I admittedly don’t have a thorough restore plan. I just wanted to make sure the critical data is replicated.
I’d like to rework my strategy so that I am able to do a full system restore in case anything went wrong, or in this case if I have to simply move the server.
I was going to just naively do a full system backup (something like restic backup /) but after reading this thread I realized that doing such a backup while it is running may not be ideal… but to my knowledge I don’t have the ability to download snapshots or even attach another instance to the volume to create one. Although even if I could I’d like to be able to do this without shutting down the server.
This is admittedly out of my comfort zone so I would love to learn more and do this the right way!
Is Restic the right tool for this? What is the best practice for this sort of thing?
After reading that thread I mentioned above and this one it sounds like one strategy is to backup the full system while excluding all the directories that are not necessary for a replication or that will cause issues, but I honestly do not know enough about Linux to confidently identify each one.
I then realized there are certain actions required when restoring that I am unfamiliar with. In the first thread the OP describes mounting a hard disk, restic restoring to it, and running chroot then getting it ready to boot from… but I believe I am booting from the only volume mounted on this server.
I guess my question there is; what is the technique for a full system backup while on the same volume?
I am also treating this as a bit of a research project so feel free to be a bit brief and just send me materials! I can ask questions later
Not necessarily. It depends on the services that are running on the machine:
Typically most services don’t write to the disk constantly, except database servers obviously. And by that I mean any kind of database (time series, sql, no-sql, etc.)
So in a nutshell you want to exclude all files from your database software (assuming you have one) and use their native tool instead (mysqldump for example). Then you do backup these exported files.
Apart from that I do a backup at the root on all my machines.
My exclusion file looks like this:
So your backup script will first go through and dump all the databases, but you still end up backing up the live db as well? Hoping it will come up fine in the restore, but if not you manually restore from the dump?
Yes, my backup script does a dump of all my databases into a backup folder on the machine, but then I don’t backup the database files. Depending on the usage of your DB there are too many chances it will be corrupt. I suppose you can replay the journal (if the DB supports it) but I don’t bother.
So for mysql you can see I skip /var/lib/mysql altogether.
The restore is done manually, I never scripted it. But it would just mean I’ll do a cat dump.sql | mysql for mysql.
So just make sure you can successfully do a backup/restore of all your database servers and you’re good to go.
Oh it’s because I don’t save any data in docker volumes. But that’s my use case only. If you do have data in volumes you should keep the folder in the backup.
After a restore I just pull the images and restart the containers no problem.
I don’t deviate much from a stock OS install (Debian, ubuntu, etc.) and I keep it up to date.
So the restore is usually quite easy, I restart a VPS from a stock OS of the same version so I can get into it. Although some provider will give you a virtual KVM access so you can reinstall yourself, I’m struggling to get something working on all VPS providers. Some don’t even give you the option anyway.
Once logged into my fresh install I just restore everything at the root. After that you just need to make sure your kernel is properly linked to the boot by doing an update-grub2 then you can reboot.
After that you can reimport all your data from the database backup files. That’s it really.
I encourage you to try on a virtual machine. Either in the cloud or with Virtual Box so you you know how to do it, and make sure it works
Assuming the new machine has the same or bigger storage device.
In order of preference for this.
If your provider has snapshot functionality that can move snapshots between regions, use that.
Boot the original machine and the new machine with Clonezilla. Set the new machine up as a receiver, and configure the old machine to send its image to the new server.
