Lack of Documentation on how to do a full system back up

randUser · May 15, 2018, 8:18pm

I want to perform a full backup of my system but there is hardly any information on how to do so using restic. The only clue I got was from the two blocks of code below which I found on this website and the official documentation.

                              Questions

I would like someone to point me in the right direction by answering the following questions.

What does the below flag --one-file-system mean in this context and where can I find documentation about it? I’ve tried ./restic help but its not there.
su -c 'restic backup --one-file-system --exclude /dev/* --exclude /lost+found/* --exclude /media/* --exclude /mnt/* --exclude /proc/* --exclude /sys/* --exclude /tmp/* --exclude /var/cache/apt/* /
Other than the values for --exclude, I don’t understand the rest of the syntax below. How do you read it? Its different to the usual restic -r backup .

root@a3e580b6369d:/# sudo -u restic /opt/restic/bin/restic --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} -r /tmp backup /

This code is from the Official Documentation showing how to use restic to backup the whole system but there is no explanation on what the command means.

If I wanted to do a full system backup that is equivalent to creating a system image will the following code work or is there something more elegant?
restic backup / --exclude={/dev/, /media/, /mnt/, /proc/, /run/, /sys/, /tmp/, /var/tmp}

Explanation: This reads as restic backup root where all my data is stored and exclude the following directories inside braces ({ }).

                             Suggestions

I think the documentation needs improving, restic works really well so far but sometimes code is just dumped in the documentation without any explanation of the syntax. A big incentive of backing up is to do a full system back up and restore so why is information about this largely missing from the official docs.

Lack of documentation increases the risk of making a mistake which would be very costly. Not everyone has the resources to fully test a full system back up of massive amounts of data so providing proper documentation is key. If it wasn’t for the massive red tape for opening non-development related issues on Github, I would have posted this there.

matt · May 15, 2018, 10:32pm

Don’t you have to pause or turn off the system to truly create an image? Otherwise files will be changing as the backup or imaging process runs, meaning the image won’t be consistent to a certain point in time; it’ll be a motion blur of states between start and end.

What exactly are you trying to do? A backup or make a disk image?

randUser · May 16, 2018, 10:43am

I am trying to do a system backup thesame way you do it in windows. I have a dualboot system, basically I want to backup my entire linux logical volume so that if Linux fails to start I can easily restore it back to the state it was in before failure. I have already created a system image for Windows and I didn’t need to pause or turn off the system. It backed up my whole Windows C Drive, well over 100 GB. Is restic designed to be able to do this?. The idea is that when if I need to re-install windows I can easily restore my previous operating system back to the way it was.

To answer your last question, I mean a system image not a disk image. I don’t want to just back up my files and documents. I have a lot of applications, configurations and software installed so I have to back up my whole system.

zcalusic · May 16, 2018, 12:10pm

@randUser, here’s a hint how to do it a bit more elegant, at least how I do the same task for several machines:

create an exclude file, and list all excludes in there, like this:

/dev/*
/proc/*
/run/*
/tmp/*
...

then run backup like this: restic backup --exclude-file=/usr/local/etc/restic-exclude.txt /

Some observations:

having all excludes in one editable place makes maintenance easier in the long run (and especially if you have several machines to backup, if they’re similar you can share the exclude file among them)
using /dev/* instead of only /dev is good for disaster recovery, /dev, /tmp (and many others) need to exist after recovery, even if they’re empty (yeah, I’ve tested that in practice, it worked well for me)
the current stable version of restic still has some bugs, where it will traverse folders excluded like the above and report errors, they can be ignored but will spam through email reports and require attention where it shouldn’t, hopefully this will be fixed in the future releases (from my report this night: error for /proc/28220: lstat /proc/28220: no such file or directory, and then a dozen of those)
haven’t used --one-file-system myself, even if it’s a neat feature that keeps backup contained within one file system, but for disaster recovery is not so good for the reasons mentioned above, and also many times I have another mount points I want to backup, too (YMMV, of course)

Hope it helps!

randUser · May 16, 2018, 1:42pm

Thanks zcalusic.

Are those 5 lines all that are needed in the exclude file? What is the … for and does the file name have to match restic-exclude exactly?
Can you explain your second bullet point? How is /dev/* different from /dev/? Both are directories but what difference does the wildcard ( * ) make?
I’m running the unstable version of restic (0.8.3-266). Are you referring to these bugs: #549, #1494? If so, do you know if they have been fixed in the version I am using?
For disaster recovery you say --one-file-system is not so good, can you elaborate?
Lastly, what is the advantage of excluding all those files, is it because they are always the same for every system so it would be redundant? I know temp would be empty on a new system but I am not sure about the rest. In otherwords if I wanted to do a full system backup then why should I exclude those directories specifically? Would that not mean that I would lose my configuration and some of my data?

randUser · May 16, 2018, 7:22pm

I managed to do a full system backup, thanks. Although I got some unexpected output. The size of root is 37.934 GiB but restic shows that it only added 27.856 GiB. I don’t think the directories in the exclude-file add up to 10 GiB so why did it add 10 GiB less than what is expected?

Files:       745608 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added:      27.856 GiB

processed 745608 files, 37.934 GiB in 1:17:34
snapshot 14c2i80 saved

jedi453 · May 16, 2018, 8:25pm

Hey @randUser,

There’s a few things that I think haven’t been made completely clear yet ( It’s likely the online documentation better describes them, but I’ll try to summarize some major points here ).

Assuming you’re using Linux, and the latest Restic Beta,

The --one-file-system option does a lot of the exclusions you’re trying to do manually. Any filesystem which isn’t specifically mounted as part of the root partition will be excluded, including virtual file systems. This means that virtual filesystems like /proc /sys /dev /tmp and /run will be excluded automatically without the need for an exclude file or directive ( So long as you specify the --one-file-system directive. ) However! If you’re backing up to somewhere on your root partition, you will need to exclude that manually!
This is just a clarification of the above, but it’s likely your distro uses separate filesystems or mountpoints for important directories, like possibly /boot and /home. These would not be backed up with the --one-file-system flag, unless you specifically back them up manually in a separate backup ( I’ve tested this, it works fine for me ), or you manually re-add them with --include directives. ( I haven’t tested this, but adding something like --include /home should work, if /home is a separate mountpoint. ( to get an idea of your separate mountpoints, run grep '^/' /etc/fstab ) If you’re not including a major filesystem, this is likely the reason the backup seems too small.
Restic Does Deduplication. It’s possible it only added 27.856 GiB of unique data, while while you have about 10GB of data that was able to be deduplicated.
If you want to backup your full installation, you’ll have to use sudo or do it from a root shell. This allows reading all the necessary files from all users and keep all correct permissions. You’d also need to restore from a root shell as well. WARNING! Always be very careful when using root! Root can do whatever it wants to your system, so make sure you type exactly what you mean! I recommend using a script for this that only root has the permission to read/write/execute. Once you’re satisfied the script is correct, double then triple-check it! If you haven’t been doing this, it could result in restic missing files that you don’t have permission to view as your normal user.

I’ve tested backing up my Arch and Ubuntu installs multiple times and reinstalling them onto other HDDs. I’m not sure how it would work in non-Linux OSes, but in Linux there are a few things you’ll need to note to do a full restore in the event of a disk failure or complete wipe:

There are important files on Linux that tell the computer where the partitions are it needs to boot. For example the file /etc/fstab requires updating to allow your system to boot again. Any file that specifically identifies the disks or partitions used will need to be updated. If you’re using encryption, often the file /etc/crypttab will need to be updated. The bootloader configuration may also need to be updated. For Grub2, you’ll likely find the base config in /etc/default/grub. See here for some background info on restoring from a backup like this.
The bootloader will need to be reconfigured and reinstalled. To do this you’ll need to boot off a liveCD and mount all the important partitions, then chroot into the install. Some good pointers for doing this can be found here. Once you’ve chrooted, you’ll need to reinstall the bootloader on the new HDD and then re-configure it. Assuming your new install drive is /dev/sdX and your distro is using GRUB2 ( It wont be! You’ll need to replace /dev/sdX! Don’t just blindly copy these! ):

grub-install /dev/sdX
grub-mkconfig -o /boot/grub/grub.cfg   # For Arch - Other more manual distros may be similar
update-grub   # For Ubuntu  - Mint and Debian may be similar

One thing that’s really cool about restic is that if you’d like you can take all my suggestions without creating a new restic repository. Restic will handle the dirty work, and your backup should basically only grow by any new files added, plus some small metadata.
I hope this clears things up somewhat.

Good luck! I think you’ll find Restic does an excellent job of doing what you’re trying to accomplish!
jedi453

fd0 · May 16, 2018, 8:30pm

Interesting, documentation is an area we can definitely improve. Thanks for the feedback!

In the help for the restic backup command, like this:

$ restic backup --help
The "backup" command creates a new snapshot and saves the files and directories
given as the arguments.
Usage:
  restic backup [flags] FILE/DIR [FILE/DIR] ...
[...]
  -x, --one-file-system                  exclude other file systems
[...]

Each command has a dedicated help for the flags that are specific for this command. restic help also tells you this in the last line of the output

$ restic help
[...]
Use "restic [command] --help" for more information about a command.

The option --one-file-system means that restic should only archive files as long as they are stored in the same file system. If you run restic backup --one-file-system /home, and the directory /home/foo is a different file system, restic will archive everything under /home except the directory /home/foo.

What @zcalusic was getting at was that with --one-file-system, the directory /home/foo would not be included in the snapshot, so when you restore the snapshot, before anything could be mounted there, you’d need to create the directory manually yourself. On the other hand, running restic backup --exclude "/home/user/*" /home (mind the quotes!) would do almost the same: Archive all below /home, exclude the contents of /home/user, but include the directory /home/user itself so you don’t need to recreate it after restore.

True, it’s not explained, but most of it is shell magic: The shell will expand restic backup --exclude={/foo,/bar} /home into restic backup --exclude=/foo --exclude=/bar /home. You can see this by inserting echo before the command (so it is just printed back to you). Does that explain a bit better what’s going on? Most people running restic right now seem to be experienced users and admins, who understand what’s going on here.

Because it hasn’t been written (yet). Are you maybe interested in helping us? Are you up for a pull request?

stevesbrain · May 17, 2018, 6:18am

You should probably be doing LVM snapshots, then either backing up the LVM snapshot at a block level, or, backing up the contents of the LVM snapshot. This ensures transactionally consistent data

formorer · May 17, 2018, 6:34am

no it doesn’t. an lvm snapshot is an open filesystem with running
applications where you don’t have any idea what the state is. Usually a blank
lvm snapshot is from all solutions the worst one.

Database will need repair, the filesystem could be broken and so on.

Alex

stevesbrain · May 17, 2018, 6:37am

Thanks for weighing in. What I mean to say is that this is a “point in time” snapshot. You won’t have part of your logs from one point in time, with a DB or files from another.

Yes, there will be certain databases that won’t cope nicely with that alone - this is why we’re all scripting regular dumps of our DB’s before backing up our FS’s, no?

As for filesystem broken, I’m yet to see this occur, and have restored via this method on many occasions. Is this something you’ve witnessed happen?

randUser · May 17, 2018, 10:23am

Thanks @jedi453. I finally understand what --one-file-system does.

I am using fedora to be precise and yes I am using one of the latest versions of the beta i.e. version 266. I am not backing up on my root system, instead I am backing up to the cloud.
When setting up my distro, I set up only the root because I was not aware that you could have separate filesystems or mountpoints for important directories like /home.
I ran grep '^' /etc/fstab and the output showed this:

/dev/mapper/fedora-root / ext4 defaults 1 1
UUID=cc4e252f-a31f-2e13-b51c-1735ge3cd17e /boot ext4 defaults 1 2
/dev/mapper/fedora-swap swap swap defaults 0 0

It seems I don’t have a major file system such as /home that is mounted to my system.

My understanding of Restic’s deduplication algorithm is that when backing up data pack a to an existing repository with data pack b it compares the timestamp of thesame files present in both data pack b with data pack a and only adds files from data pack a if timestamp data pack a > timestamp data pack b. Based on this reasoning in my case there would be no need for restic to add only 27.856 GiB of unique data out of the total 37.934 GiB because the only other files I had were two 10 MB files named test.bin from the restic tutorial in the official documentation.
Thanks for the heads up. I already figured this out when I tried to back up with a normal user and restic was denied permission in every single directory it tried to access. I Ctrl + C'ed the backup, ran restic check, restic check --read-data and then restic prune to fix the interrupted upload by getting rid of what it left behind in my repository. I am already using an exclude-file as a script rather than doing it manually in the shell. Is this the script you are referring to? If not can you give an example?

About your tips on a full restore:
Let me get this right, so /etc/fstab tells the computer how to find the partitions it needs to boot?
When you say:

“the file /etc/fstab requires updating to allow your system to boot again. Any file that specifically identifies the disks or partitions used will need to be updated.”

Do you mean that I will need to update /etc/fstab when restoring my system? Can I update it from the backup done by restic to the cloud? I’m not sure if restoring from the cloud using restic can be used to reinstall fedora. Will it work thesame as restoring and reinstalling from a local external hard drive?

“If you’re using encryption, often the file /etc/crypttab will need to be updated. The bootloader configuration may also need to be updated. For Grub2, you’ll likely find the base config in /etc/default/grub.”

How do I check if I’m using encryption, is this thesame as using root (i.e. su) which needs me to type in a password. What am I supposed to be doing with all these directories e.g. /etc/crypttab and /etc/default/grub? Again do you mind using “update” in a specifc context so I can get a better idea of what it means?

“The bootloader will need to be reconfigured and reinstalled. To do this you’ll need to boot off a liveCD and mount all the important partitions, then chroot into the install. Some good pointers for doing this can be found here. Once you’ve chrooted, you’ll need to reinstall the bootloader on the new HDD and then re-configure it. Assuming your new install drive is /dev/sdX and your distro is using GRUB2 ( It wont be! You’ll need to replace /dev/sdX! Don’t just blindly copy these! ):”

My system is a laptop and I don’t have a CD drive, can I use a different media and which liveCD do you recommend? I have Grub which is used to boot up my system but I’m not sure if it is Grub or Grub2 that I have. It might be important to mention that I have windows too because this is a dualboot system. Does this change anything? You mentioned you have backed up and reinstalled Arch and Ubuntu many times, have you done this on a dualboot system before? The motivation for this backup is that because Windows shipped with my laptop, it is sitting on too much hardrive space which fedora really needs.

Yes it was very helpful, thanks for your response.

randUser · May 17, 2018, 11:20am

Thanks @fd0. Its reassuring to have the developers respond directly to users.

From what you say is it right to think that --one-file-system is the same as doing --exclude= “/home/foo” because both ways would exclude the contents of foo and also the foo directory itself which means you have to re-create it? Also why are the quotes necessary in the shell but not in the script? I thought a string literal is only necessary in the shell when there are spaces between characters.

I actually re-read the code on that last page and its actually fairly straight forward. Can’t believe I missed it.

root@a3e580b6369d:/# sudo -u restic /opt/restic/bin/restic --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} -r /tmp backup /

I’m assuming you wrote this code… Your code reads as: user restic call the restic binary located in bin, exclude the files in braces ({and}), then in the repository at tmp back up root. Is this correct? I know the path /opt/restic/bin/restic is thesame as ./restic when you cd /opt/restic/bin/restic but I am not sure why you are using the path /opt/restic/bin/restic to call the restic binary even after explicitly using the restic command beforehand.

Thanks for your explanation. Working with restic is a lot clearer now and hopefully I can expand it further to do more such as automating back ups.

There is actually a big possibility that I’m going to have to re-install my OS so your invitation has come at the right time and is welcomed. If I go ahead with it I plan to document my backup and restoration via restic so I would be happy for a pull request. I would like to know, in this pull request, how would I contribute?

randUser · May 17, 2018, 11:31am

Thanks @stevesbrain. I haven’t heard of LVM snapshoots before. It sounds like an interesting solution to do a backup and restore. I see that you mentioned it for databases but does it work with local storage because my system is a laptop and can it work with restic?

formorer · May 17, 2018, 12:08pm

You have databases all over your desktop, like the sqlites your browser uses.

Alex

randUser · May 17, 2018, 1:05pm

Thanks for confirming that @formorer.

jedi453 · May 17, 2018, 2:13pm

Hey @randUser,

First, let me clarify a few things I was wrong or too vague on:

The command I gave to list filesystems grep '^/' /etc/fstab was incorrect. ( Sorry about that… XD ). You’ll want to do this instead: grep -v '^\#' /etc/fstab . It wont make it look as neat, but it should show you everything better. It’s still possible you missed a filesystem if you relied on my last command… Again, sorry for the confusion.
When I said LiveCD, I just meant a live Linux environment of the same distro and version as the install. It can be from a USB drive as well.
As I understand it, Restic does deduplication differently than you explained. The backend deduplication is done by looking at groups of 64 bytes, computing a fingerprint on them, and looking up if that fingerprint is already used. If there’s a stored set of 64 bytes that matches, it just references the previous backup of those bytes, if not, it makes a new index and stores those 64 bytes in it. I could be incorrect, but it’s very possible that if it works this way it reduced storage requirements significantly.

I think I’ve overcomplicated the restore process, here’s a simpler version that should make things significantly easier. I haven’t tested it myself, but everything I know about Linux says it should work fine.

To prepare for HDD/SSD failure or any unforeseeable problems, you should always keep a good copy of the Fedora LiveUSB of the same version you used to install. For example, if you’re using Fedora 28, make a Fedora 28 LiveUSB ( see here ). Also, you’ll want to make sure you can access your cloud storage again if you start from scratch, this will likely involve remembering the user/password you used to set it up. Additionally, you’ll need to remember the Restic backup password you’ve been using. I would also keep the same version of Restic you used to backup compiled for the same Linux Architecture on a flash drive or something somewhere, just incase that beta isn’t directly available from the internet anymore.

The basic process ( I haven’t tested yet, but should work fine ) is installing the same exact version of Fedora exactly as you did before (Same HDD partitioning, same user, same password… As same as a new install can be…), but from the LiveUSB. Once you’ve completed the installation, you’ll need to stay in the LiveUSB environment if it supports this, if not, reboot back into the LiveUSB. Then in the live environment, you’ll install the same version of Restic you used for backups, and configure it exactly as you did before, with whatever you did to get the cloud storage working. You’ll have to mount your new (re-)install somewhere on the live environment’s root ( For simplicity, you could just make a folder /mnt/restore as root, and mount the HDD to that. It doesn’t have to be there, you’ll just have to note where exactly it is and change the commands you run to reflect that. ). Then as the root user, you’ll restic restore with the -t /mnt/restore (changed appropriately) option with a few more --exclude directives I’ll cover below. ( You’ll need to make sure you don’t overwrite those important install-identifying files I mentioned in my last post ( For example, /etc/fstab, /etc/crypttab, /etc/default/grub, /boot/grub2/grub.cfg and /boot/grub/grub.cfg ).

Here’s a basic rundown of the process ( WARNING! This assumes you’re using BASH as a shell, and assumes you’re using GRUB2 as a bootloader (The default since Fedora 16) (NOTE! All these commands are run on the same instance of the same shell in sequential order, don’t expect to be able to close the shell or do other work then resume where you left off! ):

Boot from your LiveUSB of the same Linux Distro+Version as you were running when you last backed up.
Install The Distro onto your HDD, exactly as you did before ( You’ll want to make sure you have the same username and password, and same partitioning scheme as you did before. ) Make sure not to overwrite whatever you’re not replacing ( eg Windows ), sounds like you already did this fine, but never hurts to be extra careful.
Once the install is finished, either continue running in the live environment, or reboot into the live environment again and mount the new (re-)installation somewhere, it’s very important you remember/write down exactly where (case sensitive). For this example, I’ll be assuming /mnt/restore, so you’ll have to replace that with wherever it actually is during your restore.
Open a terminal and open a root shell: sudo -i ( WARNING 2.0! Root can do whatever it wants to the system! Be very careful here, and make sure your read and enter everything correctly! If unsure, don’t do it! This will grant all subsequent commands entered root access! )
Run the following commands ( WARNING 3.0! DON’T JUST COPY AND PASTE THIS, REVIEW EACH COMMAND AND THE COMMENTS I’ve included! Change /mnt/restore if needed ):

# You're kind of on your own for this next command.  Just whatever you set the repository as, you'll need to export the Environment Variable RESTIC_REPOSITORY as...
export RESTIC_REPOSITORY="CHANGE_ME!"
# Then Restore to the mounted location ( Change as appropriate! ) I'm using the "latest" snapshot.
# You can run "restic snapshots" to get an idea of which one you want to use, then replaced "latest" with the snapshot ID.
restic restore latest -t /mnt/restore --exclude '/etc/fstab' --exclude '/etc/crypttab' --exclude '/boot/grub/grub.cfg' --exclude '/etc/default/grub' --exclude '/boot/grub2/grub.cfg'

# Now we'll need to "chroot" into the new restored installation. 
# First we have to mount important virtual filesystems:
# Again, Change all instanced of "/mnt/restore" as appropriate!
mount --bind /dev /mnt/restore/dev
mount -t sysfs sys /mnt/restore/sys
mount -t proc proc /mnt/restore/proc
mount -t devpts devpts /mnt/restore/dev/pts
mount -t tmpfs tmp /mnt/restore/tmp
mount --rbind /run /mnt/restore/run
cp /etc/resolv.conf /mnt/restore/etc/

# Now the Actual chroot! Again, making adjustments as needed! 
# ( I'm assuming you're using bash, however, you may need to change that! )
chroot /mnt/restore /bin/bash

# Now all the commands you run from here will be
#  as if you're running them on the new installation, until you "exit" (below).
# You'll need to reinstall the bootloader
# Again, this assumes you're using GRUB2! 
# If at all unsure, see here: https://fedoraproject.org/wiki/GRUB_2
# First well need to see which HDD the root partition is on:
lsblk
# Look for a line ending in a '/', This should be the root directory.  
# If this line is part of a tree, you'll need to look up until the first line you see like "sdb" or "sd(some letter, no number)". 
# Make sure you choose the first line before the tree, or if not in a tree, the first line before the root mount in that format.
# I'm assuming it's "sdX" ( IT WONT BE!  CHANGE AS NEEDED! )
grub2-install /dev/sdX
grub2-mkconfig -o /boot/grub2/grub.cfg

Exit the chroot ( the re-installed Linux’s shell ): exit
Sync disk writes, Do this several times, especially for SSDs / SSHDs

sync
sync
sync
sync
sync
sync
sync # Why, Yes! I am paranoid!
sync; sleep 30s; sync # Very Paranoid...  This will take at least 30 seconds... Give it time.

Reboot out of the LiveCD: reboot
Boot into the new installation

This process should work fine, however I haven’t tested it myself. An important rule of backups is that nothing is really backed up until you’ve tried a restore… So if you have a spare USB HDD or Flash Drive that is large enough and you can wipe, try doing the process on that… If you don’t, you could try doing the process in a virtual machine ( if you have the space. ) If it doesn’t work, at least you’ll know before it’s needed!

Good luck, I hope this is what you needed,
jedi453

Edit: I just noticed an important typo. It should have read restic restore latest -t /mnt/restore ... ( not /tmp/restore ). Again, this should all be replaced by location you’ve mounted the new installation to.

randUser · May 17, 2018, 6:04pm

Thanks a lot. I’m going to print out these instructions and follow them step by step. There are some things I want to clarify to make sure I pull this off without any hiccups:

Just to mention, the new command you provided: grep -v '^\#' /etc/fstab didn’t make any difference, it is thesame as before. So there was nothing wrong with the previous command you provided
The Fedora LiveUSB link you provided can I get fedora 27 on that. I haven’t upgraded to fedora 28 because I plan to re-install fedora.
Your understanding sounds good. I believe those groups are called packs but I might be wrong on that. Maybe @fd0 can confirm how Restic’s deduplication actually works?

About your instructions:

When making the fedora LiveUSB do I select fedora workstation (the one I use) or make my own .iso file? I’m asking because both options are given in the link you sent (fedora).

I mentioned earlier that the reason for my back is memory issues caused by windows taking up too much space and leaving fedora with too little. Because you mentioned needing exactly the same version of fedora, I have to ask, if I delete my Linux partition (boot and swap), and use the extra space from windows to create a larger unallocated space then will these instructions still be valid? I would still install the exact version of fedora as before but this time there would be unallocated space next to fedora unlike what it is now as shown below:

 df -h
 Filesystem               Size  Used Avail Use% Mounted on
 devtmpfs                 3.9G     0  3.9G   0% /dev
 tmpfs                    3.9G  192M  3.7G   5% /dev/shm
 tmpfs                    3.9G  2.0M  3.9G   1% /run
 tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
 /dev/mapper/fedora-root   43G   32G  8.1G  80% /
 tmpfs                    3.9G   14M  3.9G   1% /tmp
 /dev/sda5                976M  196M  713M  22% /boot
 tmpfs                    789M   16K  789M   1% /run/user/42
 tmpfs                    789M   11M  778M   2% /run/user/1000
 tmpfs                    789M     0  789M   0% /run/user/0


 fdisk -l
 Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disklabel type: dos
 Disk identifier: 0xb72b0508

 Device     Boot     Start       End   Sectors   Size Id Type
 /dev/sda1  *         2048   1026047   1024000   500M  7 HPFS/NTFS/exFAT
 /dev/sda2         1026048 395909025 394882978 188.3G  7 HPFS/NTFS/exFAT
 /dev/sda3       498311168 500113407   1802240   880M 27 Hidden NTFS WinRE
 /dev/sda4       395909120 498311167 102402048  48.8G  5 Extended
 /dev/sda5       395911168 398008319   2097152     1G 83 Linux
 /dev/sda6       398010368 498311167 100300800  47.8G 8e Linux LVM

 Partition table entries are not in disk order.


 Disk /dev/mapper/fedora-root: 43 GiB, 46103789568 bytes, 90046464 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes


 Disk /dev/mapper/fedora-swap: 4.9 GiB, 5247074304 bytes, 10248192 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes

Is the live USB installation assuming GRUB is still installed or do I have to install it again before installing fedora from the LiveUSB? I’m not sure if the LiveUSB makes this unnecessary and if so, why?
I ran a script in the terminal called bootinfoscript, found here: bootinfoscript
and it confirmed that I have GRUB2 (see below) and I use bash so set up is good.

Boot Info Script 0.61 [1 April 2012]

============================= Boot Info Summary: ===============================

=> Grub2 (v1.99) is installed in the MBR of /dev/sda and looks at sector 1 of
the same hard drive for core.img, but core.img can not be found at this
location.

sda1: __________________________________________________________________________
```
 File system:       ntfs
 Boot sector type:  Windows Vista/7: NTFS
 Boot sector info:  No errors found in the Boot Parameter Block.
 Operating System:  
 Boot files:        /bootmgr /Boot/BCD
```
sda2: __________________________________________________________________________
```
 File system:       ntfs
 Boot sector type:  Windows Vista/7: NTFS
 Boot sector info:  No errors found in the Boot Parameter Block.
 Operating System:  
 Boot files:        /bootmgr /Windows/System32/winload.exe
```
sda3: __________________________________________________________________________
```
 File system:       ntfs
 Boot sector type:  Windows Vista/7: NTFS
 Boot sector info:  No errors found in the Boot Parameter Block.
 Operating System:  
 Boot files:        
```
sda4: __________________________________________________________________________
```
 File system:       Extended Partition
 Boot sector type:  -
 Boot sector info: 
```
sda5: __________________________________________________________________________
```
 File system:       ext4
 Boot sector type:  -
 Boot sector info: 
 Operating System:  
 Boot files:        /grub2/grub.cfg
```
sda6: __________________________________________________________________________
```
 File system:       LVM2_member
 Boot sector type:  -
 Boot sector info: 
```
fedora-swap’: __________________________________________________________________
```
 File system:       
 Boot sector type:  Unknown
 Boot sector info: 
 Mounting failed:   mount: /tmp/BootInfo-uFhqaEk8/LVM/fedora-swap': unknown filesystem type ''.
```
fedora-root’: __________________________________________________________________
```
 File system:       
 Boot sector type:  Unknown
 Boot sector info: 
 Mounting failed:   mount: /tmp/BootInfo-uFhqaEk8/LVM/fedora-swap': unknown filesystem type ''.
```
mount: /tmp/BootInfo-uFhqaEk8/LVM/fedora-root’: unknown filesystem type ‘’.

============================ Drive/Partition Info: =============================

Drive: sda _____________________________________________________________________
Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos

Partition Boot Start Sector End Sector # of Sectors Id System

/dev/sda1 * 2,048 1,026,047 1,024,000 7 NTFS / exFAT / HPFS
/dev/sda2 1,026,048 395,909,025 394,882,978 7 NTFS / exFAT / HPFS
/dev/sda3 498,311,168 500,113,407 1,802,240 27 Hidden NTFS (Recovery Environment)
/dev/sda4 395,909,120 498,311,167 102,402,048 5 Extended
/dev/sda5 395,911,168 398,008,319 2,097,152 83 Linux
/dev/sda6 398,010,368 498,311,167 100,300,800 8e Linux LVM

“blkid” output: ________________________________________________________________

Device UUID TYPE LABEL

/dev/mapper/fedora-root ec219f79-0450-44e5-8b80-c58d72f39b77 ext4
/dev/mapper/fedora-swap 8be9f36d-da25-49fd-ad4d-405b863d2a8a swap
/dev/sda1 7CD2B75DD2B719FA ntfs System Reserved
/dev/sda2 1CE6DE12E6DDEC50 ntfs
/dev/sda3 4810536510535950 ntfs
/dev/sda5 aa8d191d-c06d-4d09-a37a-2843fd9bc09c ext4
/dev/sda6 tA2zmI-JNef-SD3O-Z9BC-v1Mn-JjQQ-Fq6Iig LVM2_member

========================= “ls -R /dev/mapper/” output: =========================

/dev/mapper:
control
fedora-root
fedora-swap

================================ Mount points: =================================

Device Mount_Point Type Options

/dev/mapper/fedora-root / ext4 (rw,relatime,seclabel,data=ordered)
/dev/sda5 /boot ext4 (rw,relatime,seclabel,data=ordered)
If /mnt/restore is the name of the new installation to mount, in my case is this the name of the fedora version that is created in the LiveUSB?
Should these two commands you listed after mount be greyed out like they are in your code? I’m being careful so as not to misinterpret anything: mount --bind /dev /mnt/restore/dev
mount --rbind /run /mnt/restore/run
For your last paragraph, do you mean to check if restic backed up my system properly by testing it on an external hard drive or do you mean testing the whole fedora re-installation process you outlined?
By the way do you have any experience with resizing windows partitions?

Thanks for the excellent breakdown. Its very coherent. The task no longer seems insurmountable.

randUser · May 17, 2018, 6:26pm

Hey @fd0 what are your thoughts on why restic added 27.856 GiB to my Cloud based repository when it processed 37.934 GiB of data? The repository only contained two snapshots each with only one file that contained data from /dev/urandom.

restic (beta version 266) output:

Files:       745608 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added:      27.856 GiB

processed 745608 files, 37.934 GiB in 1:17:34
snapshot 14c2i80 saved

jedi453 · May 18, 2018, 2:14am

Hey @randUser,

I’ll have to look over my post more tomorrow, I just noticed an important typo which I’ve corrected in the original.

Important: From what you’ve posted, your “/boot” directory is a separate partition, you’ll need to back it up and restore it separately. I’ve also included some important notes at the end of this post, definitely make sure you read the end too!

I’ll have to see if I have time tomorrow to elaborate more, but to some of your points:

It’s likely what printed is the full file, the command you responded with ( grep '^' /etc/fstab ) will just print the whole file, the command I gave grep '^/' /etc/fstab will print only lines starting with a forward slash ( note the missing / in your version. ) Definitely be careful typing in everything I send ( after reviewing it yourself, very important! Luckily this command would not have modified anything though… ) Your output of that command does show a separate /boot partition, which you’ll have to backup separately then mount ( in something like /mnt/restore/boot ) when you do the restore. This is likely the default for Fedora 27.
Yeah, just whichever Distro and version of the distro you used to install originally. The LiveUSB version of that is what you want. You mention you used “workstation” ( I’m not familiar with Fedora ), so you’ll want the LiveUSB version of “Fedora Workstation 27”.
Certainly take my explanation with a grain of salt. I honestly have very little understanding of Restic’s internal workings. I just saw something like this in a video from a conference on youtube from several years ago. Even if it was true in the past, it’s very possible it isn’t anymore. Unfortunately I was overzealous in an attempt to explain that the deduplication could easily result in a lower backup size than the original storage used. Also note that Restic does a file-based backup, not an image-based one, so the size of the partition is irrelevant, only the disk space taken up by files matters.
You’ll want Exactly what you used to install, except as a live version. If that’s “Fedora Workstation 27”, that’s what you want, just as an installable LiveUSB.
Assuming I’ve got the instructions correct ( again I’ll have to look into this more when I have more time ), the restore process should not depend on the setup of the harddrive, you could even install it to another drive or possibly another computer ( with similar hardware ). I suggest you try doing this with a spare USB drive if you have one, to test the backup worked correctly. Your command df -h shows that /boot is a separate partition and will not be backed up with --one-file-system, it also shows that only 32G of space was used by your root partition, so about 28 GB of backup is possible.
GRUB will be installed by default on the new install, but even if it weren’t, the restore would “reinstall” it.
Sounds good!
/mnt/restore isn’t the name of the new installation, it’s just the location you’re mounting the new installation at. It’s a little complicated, but in Linux (unlike Windows), when you mount something it shows up as a directory (folder), not a separate drive letter or separate root. I didn’t give instructions for this though. From the LiveUSB environment, you’ll have to mkdir /mnt/restore as root, then you’ll have to mount the new partition or LVM device.
Those commands shouldn’t be grayed out, I don’t know how to avoid it though, it’s the forum software trying to format the code. Only the Comments ( Everything after a “#” by itself ) should be grayed out and not entered. This is just because I don’t know how to tell the forum it’s a BASH script, not some other language.
The idea was to install the same version of Fedora Workstation 27 you’re running now freshly into either a virtual machine or external HDD you can completely wipe/format. This would allow you to make tests without risking formatting your main storage ( You’d have to be very careful though, a virtual machine would likely be a better place to start, you could even do it in windows with software like VirtualBox ). Then you would try restoring to this throw-away install, just to verify the restoration process as outlined would work.
I don’t have experience with resizing Windows partitions, however GParted should be capable of this, just make sure you back everything up first!

ANOTHER WARNING! These posts have been my attempt to greatly simplify what is in some sense a complex process. It’s very possible to mess this up and wipe your current install completely ( both Windows and Linux ) don’t do this unless you feel very comfortable with it, and I take no responsibility if it doesn’t work. I’m just a random person on a forum trying to help out…

Another Note: Restic is really meant more for ongoing and long term incremental backups. If I’m understanding your use-case correctly, it’s overkill for the task you’re trying to accomplish… It would be a good backup solution for if you want to backup your stuff daily into snapshots you could restore, but for a one-off backup that only is made to immediately restore after redoing your partition setup, it’s more than you need… Here’s what’s probably a better backup solution for a one-time backup. You’ll want to use similar commands to those specified, but with bsdtar not tar.

Good luck with this! Again please be careful, I’m just trying to help, Don’t go through with it if you’re not sure of yourself! Again I can’t take responsibility if it doesn’t work or it breaks something!

Good luck,
jedi453