Automating restic backups on Debian based Linux laptops

Here is a Bash shell script that I have tested successfully on Linux for automating restic based backups using either cron or anacron.

#!/usr/bin/env bash

This script should be run via cron

set -o errexit

Default values for options

restic_config_dir=“${HOME}/restic-config”
log_dir=“${HOME}/.restic-log”
interval=$((0)) # Default interval is take backup immediately
delay_value=$((0)) # Default delay is no delay
backup_tag=“none” # Default backup tag
profile_name=“” # Set profile to null
backup_output=“”

Help message function

help_message() {
echo “Usage: $(basename $0) [-c RESTIC_CONFIG_DIR] [-l LOG_DIR] [-i INTERVAL] [-d DELAY] [-t backup identifier hourly, daily, weekly or monthly] [-p backup profile name] [-h]”
echo “Options:”
echo " -c RESTIC_CONFIG_DIR Path to the directory containing restic configurations root. Defaults to $HOME/restic-config"
echo " -l LOG_DIR Path to the directory where log files would be stored. Defaults to $HOME/.restic-log"
echo " -i INTERVAL Interval in minutes (m), hours (h), or days (d). Defaults to 0 immedfiate"
echo " -d DELAY Delay in seconds (s), minutes (m), or hours (h). Defaults to 0 no delay"
echo " -t FREQUENCY Tag to add to backups such as hourly daily weekly monthly or whatever you choose here."
echo " Defaults to the hostname-machine-id if not provided"
echo " -p PROFILE restic backup profile name. Mandatory option"
echo " -h Show this help message and exit"
}

Parse command line options

while getopts “:c:l:i:d:t:p:h” opt; do
case $opt in
c)
restic_config_dir=“$OPTARG”
;;
l)
log_dir=“$OPTARG”
;;
i)
interval_unit=“${OPTARG: -1}” # Get the last character of the option argument
interval_value=“${OPTARG%?}” # Remove the last character from the option argument
if ! [[ “$interval_value” =~ ^[[:digit:]]+$ ]]; then # check if a non negative integer
echo “Error: Backup interval ${interval_value} must be a valid integer greater than or equal to 0.”
exit 1
fi
case “$interval_unit” in
m) interval=$((interval_value * 60)) ;; # Minutes to seconds
h) interval=$((interval_value * 3600)) ;; # Hours to seconds
d) interval=$((interval_value * 86400)) ;; # Days to seconds
*)
echo “Invalid interval unit: $interval_unit” >&2
exit 1
;;
esac
;;
d)
delay_unit=“${OPTARG: -1}” # Get the last character of the option argument
delay_value=“${OPTARG%?}” # Remove the last character from the option argument
if ! [[ “$delay_value” =~ ^[[:digit:]]+$ ]]; then # Check if a non negative integer
log “Error: delay value ${delay_value} must be a valid integer greater than or equal to 0.”
exit 1
fi
case “$delay_unit” in
s|m|h) delay=“${delay_value}${delay_unit}” ;; # sleep duration
*)
echo “Invalid delay unit: $delay_unit” >&2
exit 1
;;
esac
;;
h)
help_message
exit 0
;;
t)
backup_tag=“$OPTARG”
;;
p)
profile_name=“$OPTARG”
;;
?)
echo “Invalid option: -$OPTARG” >&2
exit 1
;;
:slight_smile:
echo “Option -$OPTARG requires an argument.” >&2
exit 1
;;
esac
done

shift $((OPTIND-1))

Exit with help message if -p profile_name option not provided

if [ -z “${profile_name}” ]; then
echo “Error: profile_name mandatory command line option missing!”
help_message
exit 1
fi

set backup-tag and files_from restic command line parameter from inputs

backup_tag=“${profile_name}-${backup_tag}”
files_from=“${restic_config_dir}/${profile_name}/${profile_name}-restic-backup.files”

Check if restic --files-from file is present

if [ ! -f “$files_from” ]; then
echo “Error: --files-from file $files_from does not exist! exiting!”
exit 1
fi

Creates the log directory if it does not exist

if ! mkdir -p “$log_dir”; then
echo “Unable to create restic log directory ${log_dir}! Terminating” >&2
exit 1
fi

Create the local cache directory for last run detection if it does not exist

if ! mkdir -p “${HOME}/.cache/restic-backup”; then
echo “Unable to create restic cache directory ${HOME}/.cache/restic-backup ! Terminating” >&2
exit 1
fi

cache file from which restic determines last successful backup run if it exists

restic_cache_file=“${HOME}/.cache/restic-backup/${backup_tag}.restic-backup-last-run-timestamp”

Create restic-backup-last-run-timestamp file if it does not exist

if ! touch “${restic_cache_file}”; then
echo “You do not have write access to ${restic_cache_file}”
exit 1
fi

. /etc/profile
. ~/.profile
. ~/.bashrc

set -euo pipefail

readonly log_file=“${log_dir}/restic.log”
readonly restic_host=“$(hostname)-$(/usr/bin/cat /etc/machine-id)” # Generate repeatable consistent unique id for this host
readonly ping_host=“amazonaws.com” # ping this host to check if internet is accessible
readonly restic_env_file=“${restic_config_dir}/${profile_name}/.${profile_name}.restic.env” # profile specific environment file with Aamazon S3 credentials

Check if we can write to the log file

if ! touch “$log_file”; then
echo “You do not have write access to ${log_file}”
exit 1
fi

Logs a message with a timestamp

log() {
local message=$1
printf ‘%s %s\n’ “$(date ‘+%Y-%m-%d %H:%M:%S’):” “$message” >> “$log_file”
}

Start of backup job

log “Start of backup job”

if ! restic_binary=$(command -v restic); then
log “restic binary not found! Terminating”
exit 1
fi

if ! source “${restic_env_file}”; then
log “Failed to source restic environment file ${restic_env_file}”
exit 1
fi

Check internet connectivity before starting the backup

if ! ping -q -c 1 -W 2 “$ping_host” >/dev/null; then
log “No internet access detected. Backup aborted.”
exit 1
fi

Log input options

if [ “$interval” -eq 0 ]; then
log “No interval specified. Backing up immediately”
else
log “Backup interval specified is ${interval_value}${interval_unit}”
fi

if [ “$delay_value” -eq 0 ]; then
log “Delay option is 0 hence will not sleep”
else
log “Sleep ${delay} before start of backup”
sleep “${delay}”
log “Resuming after sleep”
fi

log “Host ID $restic_host”
log “profile_name: ${profile_name}”
log “config directory: ${restic_config_dir}”
log “log directory: ${log_dir}”
log “Backup tag is ${backup_tag}”

Get last backup time from cache file

read last_backup_snapid last_backup <<< $(/usr/bin/cat “${restic_cache_file}” | awk ‘{print $1 " " $2}’)
last_backup=${last_backup:-0} # set last_backup to 0 if unset or null

Check if last_backup is a valid integer greater than or equal to zero

if ! [[ $last_backup =~ ^[[:digit:]]+$ ]]; then

This is the first time this script is being run

if ! backup_output=$(“$restic_binary” snapshots --latest 1 -c -q --host “$restic_host” --tag “$backup_tag” | grep “$restic_host” | tail -1); then
log “Failed to get last backup timestamp”
exit 1
fi
last_backup_snapid=$(echo “${backup_output}” | awk ‘{print $1}’)
last_backup_human=$(echo “${backup_output}” | awk ‘{print $2 " " $3}’)
last_backup=$(date -d “$last_backup_human” +%s)
else
last_backup_human=$(date -d “@$last_backup” ‘+%Y-%m-%d %H:%M:%S’)
fi
log “Last backup snap ID ${last_backup_snapid} for backup tag ${backup_tag} taken at ${last_backup_human}”

Check if it’s time to take a new backup

current_time=$(date +%s)
time_diff=$((current_time - last_backup))
if [ “$time_diff” -gt “$interval” ]; then

Take a new backup

log “Backup is older than interval specified. New Backup started”
if “$restic_binary” backup --files-from “$files_from” --tag “$backup_tag” --host “$restic_host” -q >> “$log_file” 2>&1; then
backup_output=$(“$restic_binary” snapshots --latest 1 -c -q --host “$restic_host” --tag “$backup_tag” | grep “$restic_host” | tail -1)
last_backup_snapid=$(echo “${backup_output}” | awk ‘{print $1}’)
last_backup_human=$(echo “${backup_output}” | awk ‘{print $2 " " $3}’)
last_backup=$(date -d “$last_backup_human” +%s)
log “Backup succeeded with snap ID ${last_backup_snapid} for backup tag ${backup_tag} at ${last_backup_human}”
else
log “Backup failed with exit code $?”
fi
else
log “Last backup was taken sooner than the interval specified”
log “Not taking any backup”
fi

write last backup timestamp to cache file

echo “${last_backup_snapid} ${last_backup}” > “$restic_cache_file”

End of backup job

log “End of backup job”

###END OF SCRIPT#####

2 Likes

It would be great if you could format your post such that it becomes more readable. You can put ~~~ on its own line right before and after every code snippet, that way the forum will display the code better.

1 Like

Sorry, did not know how to paste formatted code. Doing so now :relieved:


#!/usr/bin/env bash

# This script should be run via cron 

set -o errexit

# Default values for options
restic_config_dir="${HOME}/restic-config"
log_dir="${HOME}/.restic-log"
interval=$((0))  # Default interval is take backup immediately
delay_value=$((0))     # Default delay is no delay
backup_tag="none"     # Default backup tag
profile_name=""   # Set profile to null 
backup_output=""

# Help message function
help_message() {
  echo "Usage: $(basename $0) [-c RESTIC_CONFIG_DIR] [-l LOG_DIR] [-i INTERVAL] [-d DELAY] [-t backup identifier hourly, daily, weekly or monthly] [-p backup profile name] [-h]"
  echo "Options:"
  echo "  -c RESTIC_CONFIG_DIR   Path to the directory containing restic configurations root. Defaults to \$HOME/restic-config"
  echo "  -l LOG_DIR             Path to the directory where log files would be stored. Defaults to \$HOME/.restic-log"
  echo "  -i INTERVAL            Interval in minutes (m), hours (h), or days (d). Defaults to 0 immedfiate"
  echo "  -d DELAY               Delay in seconds (s), minutes (m), or hours (h). Defaults to 0 no delay"
  echo "  -t FREQUENCY           Tag to add to backups such as hourly daily weekly monthly or whatever you choose here."
  echo "                         Defaults to the hostname-machine-id if not provided"
  echo "  -p PROFILE             restic backup profile name.  Mandatory option"
  echo "  -h                     Show this help message and exit"
}

# Parse command line options
while getopts ":c:l:i:d:t:p:h" opt; do
  case $opt in
    c)
      restic_config_dir="$OPTARG"
      ;;
    l)
      log_dir="$OPTARG"
      ;;
    i)
      interval_unit="${OPTARG: -1}"  # Get the last character of the option argument
      interval_value="${OPTARG%?}"   # Remove the last character from the option argument
      if ! [[ "$interval_value" =~ ^[[:digit:]]+$ ]]; then  # check if a non negative integer
        echo "Error: Backup interval ${interval_value} must be a valid integer greater than or equal to 0."
        exit 1
      fi
      case "$interval_unit" in
        m) interval=$((interval_value * 60)) ;;  # Minutes to seconds
        h) interval=$((interval_value * 3600)) ;;  # Hours to seconds
        d) interval=$((interval_value * 86400)) ;;  # Days to seconds
        *)
          echo "Invalid interval unit: $interval_unit" >&2
          exit 1
          ;;
      esac
      ;;
    d)
      delay_unit="${OPTARG: -1}"  # Get the last character of the option argument
      delay_value="${OPTARG%?}"   # Remove the last character from the option argument
      if ! [[ "$delay_value" =~ ^[[:digit:]]+$ ]]; then   # Check if a non negative integer
        log "Error: delay value ${delay_value} must be a valid integer greater than or equal to 0."
        exit 1
      fi
      case "$delay_unit" in
        s|m|h) delay="${delay_value}${delay_unit}" ;;  # sleep duration
        *)
          echo "Invalid delay unit: $delay_unit" >&2
          exit 1
          ;;
      esac
      ;;      
    h)
      help_message
      exit 0
      ;;
    t)
      backup_tag="$OPTARG"
      ;;
    p)
      profile_name="$OPTARG"
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
    :)
      echo "Option -$OPTARG requires an argument." >&2
      exit 1
      ;;
  esac
done

shift $((OPTIND-1))

# Exit with help message if -p profile_name option not provided
if [ -z "${profile_name}" ]; then
    echo "Error: profile_name mandatory command line option missing!"
    help_message
    exit 1
fi

# set backup-tag and files_from restic command line parameter from inputs
backup_tag="${profile_name}-${backup_tag}"
files_from="${restic_config_dir}/${profile_name}/${profile_name}-restic-backup.files"

# Check if restic --files-from file is present
if [ ! -f "$files_from" ]; then
    echo "Error: --files-from file $files_from does not exist! exiting!"
    exit 1
fi

# Creates the log directory if it does not exist
if ! mkdir -p "$log_dir"; then
  echo "Unable to create restic log directory ${log_dir}! Terminating" >&2
  exit 1
fi

# Create the local cache directory for last run detection if it does not exist
if ! mkdir -p "${HOME}/.cache/restic-backup"; then
  echo "Unable to create restic cache directory ${HOME}/.cache/restic-backup ! Terminating" >&2
  exit 1
fi

# cache file from which restic determines last successful backup run if it exists
restic_cache_file="${HOME}/.cache/restic-backup/${backup_tag}.restic-backup-last-run-timestamp"

# Create restic-backup-last-run-timestamp file if it does not exist
if ! touch "${restic_cache_file}"; then
  echo "You do not have write access to ${restic_cache_file}"
  exit 1
fi

. /etc/profile
. ~/.profile
. ~/.bashrc

set -euo pipefail

readonly log_file="${log_dir}/restic.log"
readonly restic_host="$(hostname)-$(/usr/bin/cat /etc/machine-id)" # Generate repeatable consistent unique id for this host
readonly ping_host="amazonaws.com"                                 # ping this host to check if internet is accessible
readonly restic_env_file="${restic_config_dir}/${profile_name}/.${profile_name}.restic.env"   # profile specific environment file with Aamazon S3 credentials             


# Check if we can write to the log file
if ! touch "$log_file"; then
  echo "You do not have write access to ${log_file}"
  exit 1
fi

# Logs a message with a timestamp
log() {
  local message=$1
  printf '%s %s\n' "$(date '+%Y-%m-%d %H:%M:%S'):" "$message" >> "$log_file"
}

# Start of backup job
log "Start of backup job"

if ! restic_binary=$(command -v restic); then
  log "restic binary not found! Terminating"
  exit 1
fi

if ! source "${restic_env_file}"; then
  log "Failed to source restic environment file ${restic_env_file}"
  exit 1
fi   

# Check internet connectivity before starting the backup
if ! ping -q -c 1 -W 2 "$ping_host" >/dev/null; then
  log "No internet access detected. Backup aborted."
  exit 1
fi

# Log input options
if [ "$interval" -eq 0 ]; then
  log "No interval specified. Backing up immediately"
elserr
  log "Backup interval specified is ${interval_value}${interval_unit}"
fi 

if [ "$delay_value" -eq 0 ]; then
  log "Delay option is 0 hence will not sleep"
else
  log "Sleep ${delay} before start of backup"
  sleep "${delay}"
  log "Resuming after sleep"
fi

log "Host ID $restic_host"
log "profile_name: ${profile_name}"
log "config directory: ${restic_config_dir}"
log "log directory: ${log_dir}"
log "Backup tag is ${backup_tag}"

# Get last backup time from cache file
read last_backup_snapid last_backup <<< $(/usr/bin/cat "${restic_cache_file}" | awk '{print $1 " " $2}')
last_backup=${last_backup:-0}      # set last_backup to 0 if unset or null

# Check if last_backup is a valid integer greater than or equal to zero
if ! [[ $last_backup =~ ^[[:digit:]]+$ ]]; then
  # This is the first time this script is being run
  if ! backup_output=$("$restic_binary" snapshots --latest 1 -c -q --host "$restic_host" --tag "$backup_tag" | grep "$restic_host" | tail -1); then
    log "Failed to get last backup timestamp"
    exit 1
  fi
  last_backup_snapid=$(echo "${backup_output}" | awk '{print $1}')
  last_backup_human=$(echo "${backup_output}" | awk '{print $2 " " $3}')
  last_backup=$(date -d "$last_backup_human" +%s)
else
  last_backup_human=$(date -d "@$last_backup" '+%Y-%m-%d %H:%M:%S')
fi
log "Last backup snap ID ${last_backup_snapid} for backup tag ${backup_tag} taken at ${last_backup_human}"


# Check if it's time to take a new backup
current_time=$(date +%s)
time_diff=$((current_time - last_backup))
if [ "$time_diff" -gt "$interval" ]; then
  # Take a new backup
  log "Backup is older than interval specified. New Backup started"
  if "$restic_binary" backup --files-from "$files_from" --tag "$backup_tag" --host "$restic_host" -q >> "$log_file" 2>&1; then
    backup_output=$("$restic_binary" snapshots --latest 1 -c -q --host "$restic_host" --tag "$backup_tag" | grep "$restic_host" | tail -1)
    last_backup_snapid=$(echo "${backup_output}" | awk '{print $1}')
    last_backup_human=$(echo "${backup_output}" | awk '{print $2 " " $3}')
    last_backup=$(date -d "$last_backup_human" +%s)
    log "Backup succeeded with snap ID ${last_backup_snapid} for backup tag ${backup_tag} at ${last_backup_human}"
  else
    log "Backup failed with exit code $?"
  fi
else
  log "Last backup was taken sooner than the interval specified"
  log "Not taking any backup"
fi

# write last backup timestamp to cache file
echo "${last_backup_snapid} ${last_backup}" > "$restic_cache_file"

# End of backup job
log "End of backup job"

2 Likes
@page { size: 21cm 29.7cm; margin: 2cm } p { line-height: 115%; margin-bottom: 0.25cm; background: transparent } a:link { color: #000080; text-decoration: underline }

Documentation for restic-backup.sh bash script



restic-backup.sh is a bash shell script that can backup the local directories on a Debian style Linux machine to an Amazon Web Service S3 bucket.


Brief usage:


Usage: restic-backup.sh [-c RESTIC_CONFIG_DIR] [-l LOG_DIR] [-i INTERVAL] [-d DELAY] [-t backup identifier daily weekly or monthly] [-p backup profile name] [-h]

Options:

-c RESTIC_CONFIG_DIR Path to the directory containing restic configurations

-l LOG_DIR Path to the directory where log files should be stored

-i INTERVAL Interval in minutes (m), hours (h), or days (d). Defaults to 0 immediate

-d DELAY Delay in seconds (s), minutes (m), or hours (h). Defaults to 0 no delay

-t FREQUENCY Tag to add to backups such as hourly daily weekly monthly or whatever you choose here.

Defaults to the hostname-machine-id if not provided

-p PROFILE restic backup profile name. Mandatory option

-h Show this help message and exit



This script is designed to run through cron or anacron on a periodic basis to take a backup of the system to Amazon S3.


The tool used for taking the backup is restic an open source command line backup program written in Go and ported to amd64 Linux, Windows and MacOS. Some of the features of restic is capability to take backups to most cloud providers such as AWS and Azure and others.


The tool restic has some very attractive features:


  • Open source community developed

  • Free to use

  • Encrypted backups using secure symmetric cyphers to most cloud providers and local REST servers and folders

  • Uses a content based de-duplication to detect changes and generally saving only the changed part in subsequent backups using a hash algorithm to detect content changes.

  • Able to take incremental backups

  • Compressed backups among others

  • Uses a single static binary with simple installation requirements and configuration


The full description of this tool can be found at https://restic.net/#introduction.



Metafile 1 The rest of the documentation uses the folder structure shown in the image below






The script accomplishes a backup at a regular interval of a set of folders and files at a specified interval of days hours or minutes


The script is provided the parent restic config directory location through the -c option a profile name through the -p option and the backup frequency in days (d) hours (h) or minutes (m) through the -i option.


Provided these inputs the script expects to find the folder structure as shown in the image. The profile name is a directory under the restic config directory having exactly the same name as the profile name.


The details on how to configure and setup the Amazon AWS S3 buckets for restic repository can be found at this short well written article on this subject Fast and Secure Backups to S3 with Restic.


Each profile folder must have two files:


  1. An environment file named .<profile name>.restic.env that has all the credentials required to connect to the remote Amazon S3 restic repository. An example content of an environment file is


unset HISTFILE

export AWS_DEFAULT_REGION='ap-outh-1'

export AWS_ACCESS_KEY_ID='MY AWS S3 ACCESS KEY id'

export AWS_SECRET_ACCESS_KEY='AWS S3 access key'

export RESTIC_PASSWORD='a key to encrypt the backups'

export RESTIC_REPOSITORY='s3:https://s3.ap-south-1.amazonaws.com/mys3bucket'



  1. A file named <profile name>-restic-backup.files that has one per line the folders that need to be backed up. An example is provided for a profile name allfiles below




The reason I chose this specific configuration model is that I found the concept of profiles as used by Vorta the GUI front end for BorgBackup very useful.

We can configure different profiles that backup a different set of files, for example a profile that backs up my videos and musics and another profile that backs up other important files.

I run the backup of my important files daily while the music and videos are backed up once a week

You can also setup multiple backup jobs for the same profile executing at different frequencies.

For example I can if I want to schedule via cron a daily, weekly and monthly backup of my important files profile via cron, if I so desire


If I want, I can backup all the profiles to the same repository or each profile to a different restic backup repository. If using the same restic backup repository for multiple profiles, it is advisable to create the restic environment file in the config directory and create soft links with the appropriate profile name in the respective profile directory(s). This way any change in this global environment in one place is immediately reflected for all the profiles sharing this environment file.


Under the hood the script does the following:


  1. Create a unique host name to be passed on to restic via the -–host command line parameter by combining the hostname and the unique machine id from /etc/machine-id as below:


    This ensures that if by any chance multiple machines having the same host name share the same repository, the host name recorded in the restic snapshots remain unique and separate for each host.

  2. Create a unique backup tag to be passed on to restic via the -–tag restic command line parameter by assembling the profile name and interval thus

    For example if for my profile myfiles I have scheduled three backups, one a daily, the other a weekly and a monthly. I will pass daily, weekly and monthly to the -t parameter to the s cript. The script will create three distinct tags myfiles-daily, myfiles-weekly, myfiles-monthly for each of these backup jobs and pass that on to restic via the -–tag parameter.



This mechanism ensures that each backup snapshot can easily be identified by combination of the host and backup tag when listing snapshots using the command restic snapshots


Thus it becomes relatively easy for me to list all the backup snapshots for myfiles profile taken weekly and if I want restore any of them as the those snapshots would have the backup tag myfiles-weekly attached to each snapshot


For the hourly, daily, weekly or monthly jobs, the jobs have to be scheduled that way via cron or using anacron.


There is also a command line parameter -d that can be used if one wants to delay the backup after being kicked of by cron by the provided seconds, minutes or hours.


The script creates a log file at the folder passed to the script via the -l command line parameter and adds timestamped log messages similar to syslog


Internally the script creates a per profile and backup frequency specific cache file under ~/.cache/restic-backup that records the snapshot id and last backup time for the specific backup for that profile and frequency.


Each time the backup is kicked off by the script, it checks if this cache file exists and gathers the last backup timestamp from this cache file, and if this file does not exist, then queries the restic repository to get the last backup time.


If the last backup was taken sooner than the frequency specified, the script does not do anything else it starts a backup and exits.


That is all!



restic-backup-documentation_html_c285d490ab7312fc