Problem with fsfreeze

Getting Help
1

I have written a wrapper restic-xbackup which calls fsfreeze before restic backup to ensure that there is no writing to the filesystem as long as the backup is running.

I know restic is using TMPDIR, so I set:

# echo $TMPDIR 
/run

# df /run
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            1048576  1180   1047396   1% /run

But restic backup still runs in a deadlock, I assume it tries to write something to the / filesystem:

# fsfreeze -f /

# restic backup -v -v -x --exclude-file /root/.restic/exclude --tag _ / 
open repository

Here restic stops, deadlock.

What can it be?

2

Writing to cache? If restic runs on the machine that holds the root filesystem that you try to back up, how did you define the restic cache folder?
https://restic.readthedocs.io/en/latest/manual_rest.html#caching

2 Likes
3

Ahh… cache directory!
Option --no-cache solves my problem with fsfreeze!
THANKS!

4

Update: --no-cache makes restic very slow

Meanwhile I am using TMPDIR=/run which is a tmpfs filesystem not interacted with fsfreeze

2 Likes
5

I have still problems with fsfreeze: restic backup is in a deadlock.

I use:

export TMPDIR=/run
mount -o remount,size=1G /run
fsfreeze -f /
restic backup -v -v -x --exclude-file /root/.restic/exclude --tag _ --cache-dir /run/restic /

/run/restic is used, I see:

root@obertux:/run/restic# ll
drwx------ root root  - 2024-06-19 16:06:28 abe9456226872758e26577a8b4efd183b463f392ea5e7392b2d7eef4b05fb095
-rw-r--r-- root root 44 2024-06-19 16:06:28 CACHEDIR.TAG

but restic hangs after:

open repository
lock repository
using parent snapshot 1a31a786
load index files
start scan on [/]
start backup on [/]
scan finished in 2.998s: 136469 files, 7.342 GiB

I have to enter “fsfreeze -u /” (in a second window).

restic even hangs when I use

restic backup -v -v -x --exclude-file /root/.restic/exclude --tag _ --no-cache /

So, there must be a write of something other, too - and the freezed / filesystem suspends restic.
==> deadlock

For debugging I run:

strace restic backup -v -v -x --exclude-file /root/.restic/exclude --tag _ --no-cache / &>/run/restic.log

While restic was runnig, I run “tail -f /run/restic.log”:

write(1, "unchanged /boot/grub/i386-pc/pxe"..., 42) = 42
futex(0x1b6c948, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
llistxattr("/boot/grub/i386-pc/raid6rec.mod", 0x1b9c9c8, 0) = 0
newfstatat(AT_FDCWD, "/boot/grub/i386-pc/random.mod", {st_mode=S_IFREG|0644, st_size=3708, ...}, AT_SYMLINK_NOFOLLOW) = 0
futex(0x1b6c948, FUTEX_WAIT_PRIVATE, 0, NULLscan finished in 3.806s: 136477 files, 7.342 GiB

At this point there was the deadlock.
After “fsfreeze -u /” I saw:

Files:           0 new,     0 changed, 136477 unmodified
Dirs:            0 new,     1 changed, 18158 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Added to the repository: 11.969 KiB (2.403 KiB stored)

processed 136477 files, 7.342 GiB in 1:50
snapshot eecae221 saved
) = ?
+++ exited with 0 +++

I cannot see any problematic writing to the freezed / filesystem.

6

With try&error I have detected why restic backup runs in a deadlock with fsfreeze:

# export TMPDIR=/run
# fsfreeze -f /tmp
# restic backup -v --exclude-file /root/.restic/exclude --tag _tmp --no-cache /tmp
SG_TENANT=tik-sas-linux
RESTIC_REPOSITORY=s3:s3.tik.uni-stuttgart.de/tik-sas-linux.obertux.restic
open repository
repository abe94562 opened (version 2, compression level auto)
lock repository
using parent snapshot f8207fc1
load index files
[0:00] 100.00%  2 / 2 index files loaded
start scan on [/tmp]
start backup on [/tmp]

Here restic hangs again!

This happens only when I use /tmp as the backup argument, no problems with any other directory!

==> restic tries to write something to /tmp, ignoring the TMPDIR environment variable!

Because my /tmp is a subdirectory of / and not a tmpfs filesystem. it comes to the deadlock.

7

I filed in a bug report: github

8

Thanks for the solution ,This is very helpful for me.

9

Please see the update to my github bug report:
My diagnosis “restic ignores TMPDIR” was wrong, it was my fault - embarassing.

I have updated my solution restix