Restic not backing up docker container mounts, makes zip-bomb

Getting Help
1

Restic version: restic 0.17.0 compiled with go1.22.5 on linux/amd64
Host: 6.10.5-arch1-1 (64-bit)

Hello, I’m trying to back up /mnt/volumes from docker which is located at /media/server/docker. I seem to have a permission problem, though. Is there a way to add sudo to the compose? I try starting it with sudo docker-compose up -d but it throws the same errors.
Docker compose:

#based on: https://www.youtube.com/watch?v=WBBTC5WfGis
#github here: https://github.com/JamesTurland/JimsGarage/blob/main/restic/docker-compose.yml
services:
  backup:
    image: mazzolino/restic
    container_name: restic
    hostname: archlinux
    environment:
      RUN_ON_STARTUP: "true" #change as you wish
      BACKUP_CRON: "0 */12 * * *" #this is twice daily, i.e., every 12 hours
      RESTIC_REPOSITORY: /restic
      RESTIC_PASSWORD: snip
      RESTIC_BACKUP_SOURCES: /mnt/volumes
      RESTIC_COMPRESSION: auto
      RESTIC_BACKUP_ARGS: >-
        --tag restic-proxmox #add tags, whatever you need to mark backups
        --verbose
      RESTIC_FORGET_ARGS: >- #change as required
        --keep-last 10
        --keep-daily 7
        --keep-weekly 5
        --keep-monthly 12
      TZ: America/Chicago
    volumes:
      - /media/8tb/restic:/restic #change the left hand side to where you want to store the backups. 
      - /media/8tb/restic/restore:/tmp-for-restore #USE THIS FOLDER FOR RESTORE - CAN VIEW EACH CONTAINER
      - /media/server/docker:/mnt/volumes:ro
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

  prune:
    image: mazzolino/restic
    container_name: restic-prune
    hostname: archlinux
    environment:
      RUN_ON_STARTUP: "true"
      PRUNE_CRON: "0 0 4 * * *"
      RESTIC_REPOSITORY: /restic
      RESTIC_PASSWORD: snip
      TZ: America/Chicago
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

  check:
    image: mazzolino/restic
    container_name: restic-check
    hostname: archlinux
    environment:
      RUN_ON_STARTUP: "false"
      CHECK_CRON: "0 15 5 * * *"
      RESTIC_CHECK_ARGS: >-
        --read-data-subset=10%
      RESTIC_REPOSITORY: /restic
      RESTIC_PASSWORD: snip
      TZ: America/Chicago
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

#commands
#to see snapshots: sudo docker exec restic restic snapshots 
#to restore: sudo docker exec restric restic restore --target /temp-for-restore <snap name>

Much shorter example of repeating errors:

backups does not exist, skipping
using parent snapshot 46ffb58b
load index files
start scan on [/mnt/volumes]
start backup on [/mnt/volumes]
scan: lstat /mnt/volumes/overlay2/a8388be035e33083554b070f0bc68ba266d0d2c40e0c818fc42008ab0c6e3624/merged/proc/47/fdinfo/11: no such file or directory
error: failed to save /mnt/volumes/overlay2/a8388be035e33083554b070f0bc68ba266d0d2c40e0c818fc42008ab0c6e3624/merged/proc/1/task/1/attr/sockcreate: read /mnt/volumes/overlay2/a8388be035e33083554b070f0bc68ba266d0d2c40e0c818fc42008ab0c6e3624/merged/proc/1/task/1/attr/sockcreate: invalid argument

error: open /mnt/volumes/overlay2/a8388be035e33083554b070f0bc68ba266d0d2c40e0c818fc42008ab0c6e3624/merged/proc/1/task/1/mem: permission denied

For some reason the snapshots are HUGE too, 8+tb but there’s only about 100gb in there. If I try to restore it’s like setting off a zip bomb, and its in the /media/server/docker:/mnt/volumes location instead of the tmp-for-restore location here: /media/8tb/restic/restore:/tmp-for-restore

sudo docker exec restic restic snapshots 
ID        Time                 Host        Tags            Paths         Size
-----------------------------------------------------------------------------------
b52001fa  2024-08-13 12:25:29  archlinux   restic-proxmox  /mnt/volumes  8.317 TiB
cf8b29a2  2024-08-13 15:59:16  archlinux   restic-proxmox  /mnt/volumes  8.619 TiB
28b7c575  2024-08-14 00:00:00  archlinux   restic-proxmox  /mnt/volumes  10.433 TiB
d7dcdc74  2024-08-20 00:00:00  archlinux   restic-proxmox  /mnt/volumes  10.247 TiB
e786138c  2024-08-20 12:00:00  archlinux   restic-proxmox  /mnt/volumes  11.067 TiB
0890975b  2024-08-21 00:00:00  archlinux   restic-proxmox  /mnt/volumes  12.129 TiB
46ffb58b  2024-08-21 12:00:00  archlinux   restic-proxmox  /mnt/volumes  11.695 TiB
-----------------------------------------------------------------------------------

Thanks in advance for your help!

2

Although the error says “permission denied”, the solution is not to blindly use sudo. You’re apparently also backing up the merged overlay filesystems instead of just the volumes. Usually you just want to exclude the whole overlay2 folder. (“…/merged/proc/…” is the procfs filesystem from within one of the containers. It doesn’t make any sense to back that up).

3

Wow, handy information that you don’t need to back it all up, I added an exclude, and also noticed the comment in the args was doing something strange as well:

Starting Backup at 2024-08-22 15:42:51

#add does not exist, skipping

tags, does not exist, skipping

whatever does not exist, skipping

you does not exist, skipping

need does not exist, skipping

to does not exist, skipping

mark does not exist, skipping

backups does not exist, skipping

open repository

using parent snapshot 16c5a416

load index files

so now that portion looks like this:

RESTIC_BACKUP_ARGS: >- # for tag, add tags, whatever you need to mark backups (comment breaks things there)
        --tag restic-proxmox
        --verbose
        --exclude /mnt/volumes/overlay2/*/merged/proc

something still seems to be wrong, it was taking a while after scanning instantly, so I thought, maybe it’ll just need the weekend. Now the file is for some reason over 4tb, and there’s LOADS of snapshots. Way too much for so little data to back up.
image
So I removed all the old snapshots, maybe they’re still messed up?

$ sudo docker exec restic restic unlock
successfully removed 15 locks
$ sudo docker exec restic restic forget --tag restic-proxmox --unsafe-allow-remove-all
[0:00] 100.00%  49 / 49 files deleted

but now it just seems to hang, it has been sitting here

Checking configured repository '/restic' ...

Repository found.

Executing backup on startup ...

Starting Backup at 2024-08-24 08:16:27

open repository

no parent snapshot found, will read all files

load index files

start scan on [/mnt/volumes]

start backup on [/mnt/volumes]

also thanks to you, looked up that you can exclude with snapraid, so I figured with restic backing up the important volumes and providing more ‘static’ data to backup, snapraid could exclude all of the following locations that threw warnings about changing data in the snapraid.conf:

exclude /docker/volumes/
exclude /docker/overlay2/
exclude /docker/containers/

Edit: I think it’s working properly now, I misread your message and tried to be too specific with the exclude when I should have blocked out the entire overlay2 folder.

scan finished in 1.843s: 175369 files, 40.884 GiB

Files:           0 new,    15 changed, 175354 unmodified

Dirs:            0 new,    19 changed, 90420 unmodified

Data Blobs:      8 new

Tree Blobs:     20 new

Added to the repository: 8.572 MiB (304.723 KiB stored)

processed 175369 files, 40.884 GiB in 0:07

snapshot a3f1d63d saved

It was making snapshots in rapid succession between the restart unless stopped and the true for run on startup but now it just says

Checking configured repository '/restic' ...

Repository found.

Scheduling backup job according to cron expression.

new cron: 0 */12 * * *

which I think is the desired behavior, however, upon restore it was not in the indicated position:

[drm@archlinux backups]$ sudo docker exec restic restic snapshots -H archlinux
[sudo] password for drm: 
ID        Time                 Host        Tags            Paths         Size
-----------------------------------------------------------------------------------
da65ac68  2024-08-25 15:53:00  archlinux   restic-proxmox  /mnt/volumes  40.882 GiB
751cca1c  2024-08-25 15:54:52  archlinux   restic-proxmox  /mnt/volumes  40.882 GiB
fe064f67  2024-08-25 15:55:02  archlinux   restic-proxmox  /mnt/volumes  40.882 GiB
d8167558  2024-08-25 15:55:12  archlinux   restic-proxmox  /mnt/volumes  40.884 GiB
0e915f37  2024-08-25 15:55:22  archlinux   restic-proxmox  /mnt/volumes  40.884 GiB
c3f40a9d  2024-08-25 15:55:33  archlinux   restic-proxmox  /mnt/volumes  40.884 GiB
d7604292  2024-08-25 15:55:45  archlinux   restic-proxmox  /mnt/volumes  40.884 GiB
a3f1d63d  2024-08-25 15:56:01  archlinux   restic-proxmox  /mnt/volumes  40.884 GiB
-----------------------------------------------------------------------------------
8 snapshots
[drm@archlinux backups]$ sudo docker exec restic restic restore --target /temp-for-restore a3f1d63d
restoring snapshot a3f1d63d of [/mnt/volumes] at 2024-08-25 15:56:01.19139966 -0500 CDT by root@archlinux to /temp-for-restore
Summary: Restored 265817 files/dirs (40.884 GiB) in 4:49

it says it is in the /temp-for-restore which is /media/8tb/restic/restore:/tmp-for-restore,
but it was empty.
running ‘ls’ I can see two tmp-for-restore and temp-for-restore
yup, I’m just stupid and typed the wrong temp. Thanks!