Problems with backup script for multiple hosts

Hey guys, I have a problem in my backup script which I would like to solve with you. Anyone is welcome to use it. As soon as it works I will upload it to github.

Maybe someone has a completely different idea.
Long story short:
I want to back up several systems. The config files for the systems are here:
/root/restic-backup/configs

Each host has its own config file:

# Host-Konfiguration für HOST1

# Allgemeine Einstellungen
REMOTE_USER="root"                # Benutzer für SSH
REMOTE_HOST="HOST1"                # Hostname oder IP-Adresse des zu sichernden S                                                                                                                                                             ystems
REMOTE_DIRS=(
    "/home/folder1"                   # Zu sicherndes Verzeichnis
    # "/pfad/zu/verzeichnis2"     # Weitere Verzeichnisse können hier hinzugefüg                                                                                                                                                             t werden
)

#EXCLUDES=("*.tmp" "*.log")        # Optional: Muster zum Ausschließen von Datei                                                                                                                                                             en

MAIL_TO="mail@domain.de"       # Benachrichtigungs-E-Mail bei Fehlern

# Backup-Zeitfenster
START_HOUR=7                      # Startzeit für Backups
END_HOUR=18                       # Endzeit für Backups
BACKUPS_PER_DAY=4                 # Anzahl der Backups pro Tag

# An welchen Tagen soll gesichert werden
BACKUP_DAYS=("Montag" "Dienstag" "Mittwoch" "Donnerstag" "Freitag")  # Wochentag                                                                                                                                                             e, an denen Backups durchgeführt werden sollen

# Aufbewahrungsdauer
RETENTION_DAYS=7                  # Wie lange Backups aufbewahrt werden sollen

# SSH-Verbindung
SSH_PORT=60005                    # Port für SSH-Verbindung
SSH_KEY="/root/.ssh/key-file-name.key" # Pfad zum privaten Schlüssel

The backup script looks like this:

#!/bin/bash
set -euo pipefail
# ==========================
# Konfiguration
# ==========================
CONFIG_DIR="./configs"
LOGDIR="./logs"
LOCKDIR="/var/run/restic-backup"
RESTIC_REPOSITORY="/sicherung/restic/backup"
RESTIC_PASSWORD_FILE="./restic-password.txt"
ADMIN_BACKUP=false  # Standardmäßig ist Admin-Backup deaktiviert

# Überprüfe auf den Parameter --admin-backup
if [[ "${1:-}" == "--admin-backup" ]]; then
    ADMIN_BACKUP=true
    echo "Admin-Backup-Modus aktiviert: Zeitbeschränkungen werden ignoriert."
fi

# Erstelle notwendige Verzeichnisse
mkdir -p "$LOGDIR" "$LOCKDIR"

# ==========================
# Funktionen
# ==========================

log() {
    local message="$1"
    echo "$(date +"%Y-%m-%d %H:%M:%S") - $message" | tee -a "${LOGDIR}/backup.log"
}

calculate_backup_times() {
    local start_hour=$1
    local end_hour=$2
    local backups_per_day=$3
    local time_slots=()

    local interval=$(( (end_hour - start_hour) * 3600 / backups_per_day )) # Sekundengenauer Abstand
    for (( i=0; i<backups_per_day; i++ )); do
        time_slots+=( "$(date -d "$((start_hour * 3600 + i * interval)) seconds" +%H:%M)" )
    done

    echo "${time_slots[@]}"
}

is_backup_day() {
    local today=$(date +%A)  # Aktueller Wochentag, z.B. "Montag"
    for day in "${BACKUP_DAYS[@]}"; do
        if [[ "$day" == "$today" ]]; then
            return 0  # Heute ist ein Backup-Tag
        fi
    done
    return 1  # Heute ist kein Backup-Tag
}

backup_host() {
    local config_file="$1"
    source "$config_file"

    # Prüfe, ob alle notwendigen Variablen gesetzt sind
    for var in REMOTE_USER REMOTE_HOST REMOTE_DIRS MAIL_TO START_HOUR END_HOUR BACKUPS_PER_DAY RETENTION_DAYS SSH_PORT SSH_KEY BACKUP_DAYS; do
        if [ -z "${!var+x}" ]; then
            log "Fehler in $config_file: $var nicht gesetzt."
            return 1
        fi
    done

    # Überprüfe, ob heute ein Backup-Tag ist
    if ! is_backup_day; then
        log "Heute ist kein definierter Backup-Tag für $REMOTE_HOST. Überspringe Backup."
        return 0
    fi

    local lockfile="${LOCKDIR}/backup_${REMOTE_HOST}.lock"
    exec 200>"$lockfile"
    flock -n 200 || {
        log "Backup für $REMOTE_HOST läuft bereits."
        return 1
    }

    trap 'flock -u 200; exit 1' INT TERM

    log "Starte Backup für $REMOTE_HOST."

    # Sicherung starten
    for dir in "${REMOTE_DIRS[@]}"; do
        log "Sichere Remote-Verzeichnis: $dir"

        # Erstelle Ausschlussoptionen für tar
        exclude_params=()
        for pattern in "${EXCLUDES[@]}"; do
            exclude_params+=( "--exclude=$pattern" )
        done

        # Führe das Backup durch
        if ! ssh -i "$SSH_KEY" -p "$SSH_PORT" "${REMOTE_USER}@${REMOTE_HOST}" \
            "tar -cf - ${exclude_params[@]} -C $(dirname "$dir") $(basename "$dir")" | \
            restic backup --repo "$RESTIC_REPOSITORY" \
                --password-file "$RESTIC_PASSWORD_FILE" \
                --stdin \
                --stdin-filename "$REMOTE_HOST$dir" \
                --tag "$REMOTE_HOST" \
                2>&1 | tee -a "${LOGDIR}/backup_${REMOTE_HOST}.log"; then
            log "Fehler beim Backup von $dir"
            echo "Fehler beim Backup von $REMOTE_HOST: $dir" | mail -s "Backup-Fehler" "$MAIL_TO"
        fi
    done

    # Alte Snapshots bereinigen
    restic forget --repo "$RESTIC_REPOSITORY" \
        --password-file "$RESTIC_PASSWORD_FILE" \
        --tag "$REMOTE_HOST" \
        --keep-daily "$RETENTION_DAYS" \
        --prune 2>&1 | tee -a "${LOGDIR}/backup_${REMOTE_HOST}.log"

    flock -u 200
}

# ==========================
# Hauptskript
# ==========================

log "Starte Restic-Backup-Service."

while true; do
    for config in "$CONFIG_DIR"/*.conf; do
        [ -e "$config" ] || continue
        backup_host "$config" &
    done

    wait
    log "Alle Backups abgeschlossen."

    # Im Admin-Modus wird die Schleife beendet, da nur ein Backup-Lauf gewünscht ist
    if [[ "$ADMIN_BACKUP" == true ]]; then
        log "Admin-Backup-Modus: Beende nach einem Durchlauf."
        exit 0
    fi

    log "Warte auf den nächsten Backup-Zyklus."
    sleep 60
done

Theoretically, it just works:
Read the configuration for each host, start a separate backup process in the background for each host.
There is a repository for the backups

However, the backups also start:

  1. do you find an error?
  2. because I use Tar as a pipe, restic only knows the top folders, I cannot display the files and subfolders of the snapshots
  3. I am testing it again, it takes a while: On the second run I have the problem that the same parent id of the snapshot is used

I hope that someone will find the time and inclination to fix the problem with me.

When the problem is solved I will implement this:
Automatically create one repository per host.
Then the script should be ready. It will then run as a service.

1 Like

I should perhaps also say that I suspect that the problem lies here:

# Führe das Backup durch
        if ! ssh -i "$SSH_KEY" -p "$SSH_PORT" "${REMOTE_USER}@${REMOTE_HOST}" \
            "tar -cf - ${exclude_params[@]} -C $(dirname "$dir") $(basename "$dir")" | \
            restic backup --repo "$RESTIC_REPOSITORY" \
                --password-file "$RESTIC_PASSWORD_FILE" \
                --stdin \
                --stdin-filename "$REMOTE_HOST$dir" \
                --tag "$REMOTE_HOST" \
                2>&1 | tee -a "${LOGDIR}/backup_${REMOTE_HOST}.log"; then
            log "Fehler beim Backup von $dir"
            echo "Fehler beim Backup von $REMOTE_HOST: $dir" | mail -s "Backup-Fehler" "$MAIL_TO"
        fi

I am routing the output from tar directly to restic. I don’t know if this is correct or how I could do it differently

Here is the problem:
I list all snapshots
# restic --repo /sicherung/restic/backup/ --password-file /root/restic-backup/restic-password.txt snapshots

I get the display of the snapshots

repository **3f717b06** opened (version 2, compression level auto)
ID        Time                 Host        Tags        Paths     Size
---------------------------------------------------------------------------
34513ef6  2024-11-22 10:03:08  DNS-NAME       SERVER-TAG   /home/user  59.736 GiB
---------------------------------------------------------------------------
1 snapshots

But as soon as I want to display the files, I only see the top ones:
restic --repo /sicherung/restic/backup/ --password-file /root/restic-backup/restic-password.txt ls 34513ef6

repository 3f717b06 opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
snapshot 34513ef6 of [/opt/HSH] at 2024-11-22 10:03:08.547873879 +0100 CET by root@v0128 filtered by []:
/home
/home/user

I don’t see any files within the snapshot.
This makes it difficult or even impossible for me to determine whether everything is working properly and whether it has backed up everything.

1 Like

You are only backing up a single file in each snapshot (the tar file which comes as stdin) - hence you only see a single file in each snapshot.

Note that restic is not able to handle tar files as input - except saving it as a single file.
Why are you not backing up from each host directly?

1 Like

I put it all together.
Based on an rsync script.

I can’t and don’t want to install anything on the remote hosts, if possible it should work without additional software.

How can I implement it differently?

1 Like

You can try to mount the remote host locally using sshfs and then backup the mounted dirs - but for this you might run into the as-path problematic…

1 Like

It seems to be a bigger problem, when I mount it via sshfs I have the problem that it can potentially change data on the host.

I had to mount as read only and that doesn’t work

1 Like

Hello @Judi ,

It is certainly possible to use sshfs read-only. It has such an option. You can enforce this on the remote hosts in case they do not trust the machine running restic. See https://www.reddit.com/r/archlinux/comments/1t02xe/how_do_i_allow_readonly_access_with_sshfs_mounts/?rdt=49062 .

1 Like