File Filtering Problems — Why Some Patterns Don’t Match Any Files on Linux

“Restic Version” Output

restic 0.18.1 compiled with go1.25.6 X:nodwarf5 on linux/amd64

Restic Command

RCLONE_LOG_FORMAT=date,time sudo restic backup -n --compression max --iexclude-file "/home/joseph/Documents/backup linux/Scripts/Restic backups (for Restic or backrest)/exclude.txt" --files-from "/home/joseph/Documents/backup linux/Scripts/Restic backups (for Restic or backrest)/include.txt" -o rclone.args="serve restic --stdio --b2-hard-delete --config /home/joseph/.config/rclone/rclone.conf --log-file '/home/joseph/.config/rclone/rclone.log' --log-level INFO --stats 2s --transfers 5 --checkers 5 --retries 3 --max-backlog 200000 --buffer-size 32M --track-renames --log-file-compress --log-file-max-size 200M --log-file-max-age 4h --log-file-max-backups 5" -r rclone:Google_Drive_Joseph_Backup:Linux/Linux\ Restic\ Backup/ /etc

Restic’s Problematic Command Output

pattern "/home/**/rtcwkey" does not match any files, skipping
pattern "/home/joseph/.local/share/Steam/**/*.cgp*" does not match any files, skipping
pattern "/home/joseph/.local/share/Steam/**/*.ini*" does not match any files, skipping
pattern "/home/joseph/.local/share/Steam/**/*.mhr*" does not match any files, skipping
pattern "/home/joseph/.local/share/Terraria/**/*.nbt*" does not match any files, skipping
repository 5c8460e6 opened (version 2, compression level max)
using parent snapshot 4b811cc2
[0:00] 100.00%  5 / 5 index files loaded

Files:           0 new,  2380 changed, 12692 unmodified
Dirs:            0 new,   972 changed,   285 unmodified
Would add to the repository: 32.707 MiB (2.806 MiB stored)

processed 15072 files, 2.590 GiB in 0:06

My Problematic Restic Files

exclude.txt
# Those that have "#" in front of them are not detected by the "--iexclude-file" flag

/home/joseph/.cache
/home/joseph/.config/Code - OSS
/home/joseph/.config/chromium
/home/joseph/.config/discord
/home/joseph/.config/Element
/home/joseph/.config/google-chrome*
/home/joseph/.config/heroic
/home/joseph/.config/rclone/*.gz*
/home/joseph/.config/rclone/*.log*
/home/joseph/.config/RSS Guard 4/
/home/joseph/.config/Signal/
/home/joseph/.config/zen
/home/joseph/.local/share/backrest/tasklogs/.inprogress
/home/joseph/.local/share/goverlay/
/home/joseph/.local/share/kactivitymanagerd/
/home/joseph/.local/share/lutris/
/home/joseph/.local/share/spotify-launcher/
/home/joseph/.local/share/TelegramDesktop/
/home/joseph/.local/share/Trash/
/home/joseph/.local/share/umu/
/home/joseph/.local/share/Steam/*
/home/joseph/.local/share/Steam/steamapps/common/Left 4 Dead 2/**.tga*
!/home/joseph/.local/share/Steam/steamapps/common/Left 4 Dead 2/left4dead2/cfg/*
!/home/joseph/.local/share/Steam/steamapps/common/Left 4 Dead 2/left4dead2/materials/**.vtf
!/home/joseph/.local/share/Steam/steamapps/common/Left 4 Dead 2/left4dead2/materials/**.vmt
!/home/joseph/.local/share/Steam/steamapps/common/tModLoader/CalamityandQoL/**.nbt*
!/home/joseph/.local/share/Steam/**/*.cfg*
!/home/joseph/.local/share/Steam/**/*.cgp*
!/home/joseph/.local/share/Steam/**/*.ini*
!/home/joseph/.local/share/Steam/**/*.json*
!/home/joseph/.local/share/Steam/**/*.mhr*
!/home/joseph/.local/share/Steam/**/*.txt*
!/home/joseph/.local/share/Steam/**/*.vdf*
!/home/joseph/.local/share/Steam/steamapps/common/Team Fortress 2/tf/cfg/*
!/home/joseph/.local/share/Steam/steamapps/common/Team Fortress 2/tf/custom/*
!/home/joseph/.local/share/Steam/steamapps/common/tModLoader/CalamityandQoL/**/*.nbt*
!/home/joseph/.local/share/Steam/steamapps/common/*Wolf*
#!/home/joseph/.local/share/Terraria/**.{dat,json,sha256,txt}*
/home/joseph/.mozilla/
!/home/joseph/.wolf/
include.txt
# Those that have "#" in front of them are not detected by the "--files-from" flag

/home/**/rtcwkey
/home/joseph/.local/share/Steam/**/*.cfg*
/home/joseph/.local/share/Steam/**/*.cgp*
/home/joseph/.local/share/Steam/**/*.ini*
/home/joseph/.local/share/Steam/**/*.json*
/home/joseph/.local/share/Steam/**/*.mhr*
/home/joseph/.local/share/Steam/**/*.txt*
/home/joseph/.local/share/Steam/**/*.vdf*
/home/joseph/.local/share/Steam/steamapps/common/RealRTCW/**.mhr*
/home/joseph/.local/share/Steam/steamapps/common/Team Fortress 2/tf/cfg/
/home/joseph/.local/share/Steam/steamapps/common/Team Fortress 2/tf/custom/
/home/joseph/.local/share/Terraria/**/*.dat*
/home/joseph/.local/share/Terraria/**/*.json*
/home/joseph/.local/share/Terraria/**/*.nbt*
/home/joseph/.local/share/Terraria/**/*.sha256*
/home/joseph/.local/share/Terraria/**/*.txt*
/home/joseph/.wolf/
/home/joseph/Documents/AutoHotkey scripts/
/home/joseph/Documents/backup linux/
/home/joseph/Documents/Backups uploaded to internet/
/home/joseph/Documents/Christian stuff/
/home/joseph/Documents/Cooking/
/home/joseph/Documents/ET/other stuff/
/home/joseph/Documents/Passwords & Crypto/
/home/joseph/Documents/Software stuff & more/
/home/joseph/Documents/Wolfenstein - Enemy Territory

Why don’t some patterns match any files? I’ve been studying and testing both Restic and Rclone for days (i.e. reading docs, trying commands and flags for both Restic and Rclone, etc.), and I can’t understand these errors. Somone with much greater experience, help me. Thank you. :slightly_smiling_face:

I’m actually not sure about this myself. The documentation says:

To match across an arbitrary number of subdirectories, use the special ** wildcard.

But it also says:

Patterns use the syntax of the Go function filepath.Match

There is no mention of ** doing anything special in the documentation for that function.

@fd0 @MichaelEischer Can one of you shed some light on this? Is this a doc bug?

No, it’s not a bug in the documentation. Restic supports ** by its own implementation, since as you say ** is not part of what filepath.Match supports. The code for it is in the match() function in internal/filter/filter.go. So in short it supports both.

1 Like

I opened this PR trying to clarify this misunderstanding.
Feedback is welcome :folded_hands:

1 Like

I wonder if it’s a bug in Restic. :thinking: I’m sure the files are where the inclusion file should find them. I might check their locations again with fd.

--files-from currently does not support **. Only --exclude does. The reason is that --files-from currently relies on go stdlib functionality to collect the relevant files. Switching to our own filesystem walking along with using the internal filter implementation would be possible. Just replacing the currently calls to filepath.Glob should be possible with reasonable effort. The optimal solution of directly integrating with the archiver would be far more complex though (and probably overkill for now).

I couldn’t find a Github Issue when taking a look, so please open a new issue with the feature request.

4 Likes

I posted this feature request on GitHub. Thanks for the info and the suggestion, @MichaelEischer.

2 Likes