Backup best practice - review my scripts?

Getting Help
1

Hi,

I run a daily incremental backup on my OMV5 NAS (which stores documents, photos and videos), and a weekly prune / check. I wanted to check whether I’m missing anything in my scripts (or I’m doing things that are not necessary). Any feedback would be really appreciated!

Daily backup

#!/bin/bash

SHAREDFOLDERS=/srv/dev-disk-by-label-NASHD

# Check cache directory exists

[ ! -d "/var/cache/restic" ] && mkdir -p "/var/cache/restic"

# START

date "+BACKUP using restic starting on %c%n"

# TEST network connection

printf "TEST network "

if ping -q -c 1 -W 1 8.8.8.8 >/dev/null; then
	printf "[OK]\n"
else
	>&2 date "+%n%H:%M %c%n"
	>&2 printf "[DOWN, exiting]\n" 
	exit 1
fi

# Environment variables needed for all tasks

source "$SHAREDFOLDERS"/Config/scripts/restic/config/variables.sh

# REMOVE STALE LOCKS --------------

date "+%n--- Removing stale locks at %H:%M"
if ! restic unlock; then
	>&2 date "+%n%H:%M %c%n"
	>&2 printf "FAILED: restic unlock\n\n"
	exit 1	
fi

# CHECK BACKUP INTEGRITY

date "+%n--- Starting DOWNLOAD INTEGRITY CHECK at %H:%M"
if ! restic --cache-dir /var/cache/restic check --read-data-subset=$((1 + RANDOM % 128))/128; then
	>&2 date "+%n%H:%M %c%n"
	>&2 printf "FAILED: restic download integrity check\n\n"
	exit 1
fi

# BACKING UP ----------------------

# Backup SHARED FOLDERS

date "+%n--- Starting SHARED FOLDERS backup at %H:%M"
if ! restic --cache-dir /var/cache/restic backup --tag sharedfolders ""$SHAREDFOLDERS"/Archive/" ""$SHAREDFOLDERS"/Photos/" ""$SHAREDFOLDERS"/Config/" --exclude=".*"; then
	>&2 date "+%n%H:%M %c%n"
	>&2 printf "FAILED: restic backup sharedfolders\n\n"
	exit 1
fi

# Re-checking backup integrity

date "+%n--- Starting DOWNLOAD RE-CHECK at %H:%M"
if ! restic --cache-dir /var/cache/restic check --read-data-subset=$((1 + RANDOM % 128))/128; then
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "FAILED: restic download re-check (recheck)\n"
	exit 1
fi

date "+%n--- Finished at %H:%M%n"

Weekly prune / check

#!/bin/bash

SHAREDFOLDERS=/srv/dev-disk-by-label-NASHD

# Check cache directory exists

[ ! -d "/var/cache/restic" ] && mkdir -p "/var/cache/restic"

# START

date "+CHECK, FORGET and PRUNE backup using restic on %c%n"

# TEST network connection

printf "TEST network "

if ping -q -c 1 -W 1 8.8.8.8 >/dev/null; then
  printf "[OK]\n"
else
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "[DOWN, exiting]\n"
	exit 1
fi

# Environment variables needed for all tasks

source "$SHAREDFOLDERS"/Config/scripts/restic/config/variables.sh

# Removing stale locks

date "+%n--- Removing stale locks at %H:%M"
if ! restic unlock; then
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "FAILED: restic unlock\n"
	exit 1	
fi

date "+%n--- Starting DOWNLOAD INTEGRITY CHECK at %H:%M"
if ! restic --cache-dir /var/cache/restic check --read-data-subset=$((1 + RANDOM % 128))/128; then
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "FAILED: restic download subset check\n"
	exit 1
fi

# Forgetting old snapshots

date "+%n--- Starting FORGET at %H:%M%n"
if ! restic --cache-dir /var/cache/restic forget --keep-daily 7 --keep-weekly 6 --keep-monthly 12; then
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "FAILED: restic forget\n"
	exit 1
fi

# Pruning repository

date "+%n--- Starting PRUNE at %H:%M"
if ! restic --cache-dir /var/cache/restic prune; then
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "FAILED: restic prune\n"
	exit 1
fi

# Re-checking backup integrity

date "+%n--- Starting DOWNLOAD RE-CHECK at %H:%M"
if ! restic --cache-dir /var/cache/restic check --read-data-subset=$((1 + RANDOM % 128))/128; then
	>&2 date "+%n--- Finished at %H:%M"
	>&2 printf "FAILED: restic download re-check (recheck)\n"
	exit 1
fi

date "+%n--- Finished at %H:%M"
2

My 2 cents:

  • why remove the stale locks? I heard there have been problems but those should be removed in the latest beta builds. Better use a beta version if you encounter problems with locks. I would never automatically run restic unlock
  • I would not run check with every backup. But this is my personal opinion :smile:
  • In fact, checking before and after the backup is overkill. If you are usure, run check without --read-data before your backup.
  • If you need/want to run check with --read-data, I would read all data in the weekly check+prune or use --read-data-subset as intended, i.e. x/52 where x is the week of the year or x/4 where x is the week of the month, etc.
  • no need to separate forget and prune. forget --prune does the job!
  • Again, I would not check before and after the pruning, but given that prune needs special care I can understand people who want to be sure. However, no need to run a check --read-data twice, IMO this suffices once.
1 Like
3

Thanks! Super useful.

I’ve had problem with stale locks before, and I’m a bit reluctant to use betas for my backup, but I’ll definitely try taking it off when the next version is released.

Gabe

4

I can heavily suggest to try out the next versions. There are already lots of optimizations which have been merged and some more on the way.
Those optimizations include a noticable reduction of memory requirements for large repositories, and speed optimizations.