PS C:\Users\X\Documents\utilities\restic> .${RESTICEXE} version
restic 0.17.1 compiled with go1.23.1 on windows/amd64
Function Set_Var_Restic_C {
$global:RESTIC_SOURCE_DRIVE = 'C' #Drive letter with no colon or slash.
$global:RESTIC_REPOSITORY = "${backupdrive}:\restic_backup_${RESTIC_BKUPYR}\${RESTIC_SOURCE_DRIVE}"
$env:RESTIC_REPOSITORY = $RESTIC_REPOSITORY
$global:CACHEDIR = "C:\Users\X\Documents\utilities\restic\cache_dir\${RESTIC_SOURCE_DRIVE}"
$env:CACHEDIR = $CACHEDIR
$global:RESTIC_IEXCLUDE_FILE = "C:\Users\X\Documents\utilities\restic\iexcludes_${RESTIC_SOURCE_DRIVE}.txt"
#$global:RESTIC_BKUPLOG_FILE = "${RESTICDIRLOG}restic_${ymdhm}_${RESTIC_SOURCE_DRIVE}.log"
$global:RESTIC_STDERR_FILE = "${RESTICDIRLOG}restic_${ymdhm}_${RESTIC_SOURCE_DRIVE}_stderr.log"
$global:RESTIC_STDOUT_FILE = "${RESTICDIRLOG}restic_${ymdhm}_${RESTIC_SOURCE_DRIVE}_stdout.log"
}
PS C:\Users\X\Documents\utilities\restic> .${RESTICEXE} check --read-data | Out-File -FilePath $RESTIC_CHECK_FILE
error for tree 11e5edf4:
tree 11e5edf42acdf3e7677530191aa8c1b6e6d6033448e775dde6ae659466896270: node "GetHelp.exe" with invalid type "irregular"
error for tree d5dfc17c:
tree d5dfc17c6fb9231a42b32355d2e21d26f1bf34bd5046d8212ec77d01a286b2e2: node "AffinityDesigner2.exe" with invalid type "irregular"
error for tree 04e25d28:
tree 04e25d28d4faaa2edf97d8572bd098cac930610d16f7a5652c1c3cb5d042dde4: node "MediaPlayer.exe" with invalid type "irregular"
error for tree 8d817fe1:
tree 8d817fe1d878f33c9723d96d0230e2b13427297ca9f7aa50e191cee26afbc49d: node "GetHelp.exe" with invalid type "irregular"
pack 81c8a7d832637eb21b49eb5cd318c8de587b4c127c6622499149aacb8a0c5267 contains 2 errors: [blob 4cd645150f37bc1266eb604e2fd54942d781cacde14672e93481c696878960d0: decrypting blob <data/4cd64515> from 81c8a7d8 failed: ciphertext verification failed unexpected pack id 284735e72620f6755f8215c421172cbd39bf41351ccefc837ecac7bb693cc594]
restic repair packs 81c8a7d832637eb21b49eb5cd318c8de587b4c127c6622499149aacb8a0c5267
restic repair snapshots --forget
Questions:
(1) What is “irregular” type and why is it invalid?
(2) As a guess is this likely a hardware problem with the external hard drive?
I am running Windows 10. Each calendar year a new repository is created then a backup of C:/ with a lot of excludes is done to an external hard drive. Once a week another backup is done. The restic executable is updated when a new version comes out. The errors were noticed today and have not been there before this.
I have no actual need to restore anything but am looking for a likely cause of these errors. If the likely cause is bad external hard drive then a good solution to me is buy another external hard drive and create a new repository on it.
I’m running restic 0.17.1 compiled with go1.23.1 on windows/amd64 on Windows 11
I’m backing up a folder to three different destinations: Google Drive, IDrive e2 and Backblaze.
Each backup is a separate task and in all three I’m getting a similar error when checking:
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.aep" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.dmg" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.doc" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.inf" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.jpeg" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.lnk" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.m4a" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.mpg" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.py" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.url" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.wav" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.xlsx" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.xml" with invalid type "irregular"
tree 8b7a78c7b13a6314a67ddc8961b291e909d23589d61d692925d0daee52df1647: node "placeholder.zip" with invalid type "irregular"
The repository is damaged and must be repaired. Please follow the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html .
Maybe is a bug?
The errors were noticed today and have not been there before this.
Same!
I have no actual need to restore anything but am looking for a likely cause of these errors.
Same!
But I am only backing up to the cloud and in all three cases I get the same error.
Yes I am willing to give the PR a try. As a retired DBA I do remember a bunch of technical things but never learned how to use GitHub so instructions would be needed. I have go1.17.8 installed for goofing around so does it need to be updated or deleted?
Once the software had been down loaded and installed do I just do a regular backup? For myself I don’t like the idea that a backup command doing a “fix” to the repository, a backup command should only add data. If this is just fixing the reporting of “irregular” that would be fine.
I am running windows 10.
no longer include irregular files in a snapshot. Note that with the PR the backup command will still complain about irregular files, but those will no longer be included in the snapshot. Existing snapshots are not modified in any way.
To remove irregular files from existing snapshots you can use the restic repair snapshots <ids> --forget command.
That could be the case. Since Go 1.23 (used for restic 0.17.1) most reparse points (probably also used by Cryptomator under the hood) are treated as irregular files. The big problem there is that reparse points can significantly alter filesystem semantics with no way for restic (or Go) to known what to expect. Similarly, restore would previously just restore those files as plain files.
Downloaded and installed go version go1.23.1 windows/amd64
Downloaded the fix-backup-irregular.zip
Did the build. Ran a backup with the PR version.
stderr.log: {“message_type”:“error”,“error”:{“message”:“C:\Users\X\AppData\Local\Microsoft\WindowsApps\Microsoft.GetHelp_8wekyb3d8bbwe\GetHelp.exe: unsupported file type "irregular"”},“during”:“archival”,“item”:“C:\Users\X\AppData\Local\Microsoft\WindowsApps\Microsoft.GetHelp_8wekyb3d8bbwe\GetHelp.exe”}
{“message_type”:“error”,“error”:{“message”:“C:\Users\X\AppData\Local\Microsoft\WindowsApps\Microsoft.ZuneMusic_8wekyb3d8bbwe\MediaPlayer.exe: unsupported file type "irregular"”},“during”:“archival”,“item”:“C:\Users\X\AppData\Local\Microsoft\WindowsApps\Microsoft.ZuneMusic_8wekyb3d8bbwe\MediaPlayer.exe”}
{“message_type”:“error”,“error”:{“message”:“C:\Users\X\AppData\Local\Microsoft\WindowsApps\SerifEuropeLtd.AffinityDesigner2_3cqzy0nppv2rt\AffinityDesigner2.exe: unsupported file type "irregular"”},“during”:“archival”,“item”:“C:\Users\X\AppData\Local\Microsoft\WindowsApps\SerifEuropeLtd.AffinityDesigner2_3cqzy0nppv2rt\AffinityDesigner2.exe”}
{“message_type”:“exit_error”,“code”:3,“message”:“Warning: at least one source file could not be read”}
The check command gives the same results as before.
No additional errors happened.
I would like to run the “restic repair snapshots --forget” but how do I determine the snapshot id number? Fro the error message "error for tree “8d817fe1” is “8d817fe1” the snapshot id?
@fede I have not created any encrypted files myself. But software vendors may have created them.
I looked further back into the older stderror.log and noticed that the same files have been having problems for quite a while. So this is not a new situation.
That error message means that the PR worked insofar as that it no longer includes the irregular file in the snapshot. I think I’ve seen that error for files in C:\Users\X\AppData\Local\Microsoft\WindowsApps also with restic versions before 0.17.0. Microsoft has done some magic to those applications that are downloaded from the store. It might be best to just exclude them from the backup, but check that there’s no user data stored in the WindowsApps folder.
No that id is a tree id. You could either run restic find --tree 8d817fe1 get a list of snapshots that contain that tree. But it is probably much easier to just run restic repair snapshots --forget without specifying snapshots, which will process all snapshots but only modifies problematic ones. To know beforehand which snapshots would be modified, you can run restic repair snapshots --dry-run.
restic repair snapshots --forget removed the problem snapshots.
Blockquote
PS C:\Users\X\Documents\utilities\restic> .${RESTICEXE} check
using temporary cache in C:\Users\X\AppData\Local\Temp\restic-check-cache-137808379
create exclusive lock for repository
repository cf0c7fe4 opened (version 2, compression level auto)
created new cache in C:\Users\X\AppData\Local\Temp\restic-check-cache-137808379
load indexes
[0:00] 100.00% 45 / 45 index files loaded
check all packs
check snapshots, trees and blobs
[0:03] 100.00% 40 / 40 snapshots
no errors were found
Thank you for helping fix this issue and identifying that Microsoft Store items may the cause of the problem. I shall need to adapt my scripts to find errors and shove them in my face. Thank you for all of Restic.
First of all, I apologize for using this thread for a similar problem.
Thanks to the commands you shared, I found the problem.
I was also backing up a Dropbox folder and the error was when I backed up the Cache folder and did not exclude it.
All the errors were in the same folder which has the files named “placeholder” as content.
I use restic 0.17.1 compiled with go1.23.1 on windows/amd6 on Windows 10.
I probably have the same problem with nodes with invalid type “irregular” backup from in C:\Users\X\AppData\Local\Microsoft\WindowsApps\
Unfortunately the problem persists after excluding the directory from backup and running “repair snapshots --forget”:
The repair command leaves all snapshots unmodified and the check command still shout out error in trees containing nodes with invalid type “irregular”.
Is there any other way to fix the existing repository or do I have to delete an recreate it? Or do I have to wait with the repair command until a fixed restic version is released?
You’ll need to recompile restic from source, as @MichaelEischer indicates above. If that is too complicated, I think the easiest approach is to delete the last few snapshots that you made with 0.17.1, and revert to 0.17.0 for the time being.
lol this is interesting. I had the same error. Searched for it, found this.
It’s the same “invalid type: irregular” errors. Repairing the index and then the snapshot appears to have fixed it. A diff between the snapshots just show 10 tree blobs added, and 10 removed.
What makes this interesting is… the only snapshot affected is my oldest snapshot - taken on 2021-09-16 with most certainly an older version of Restic lol
restic 0.17.1 on windows 11 10.0.22631.4169 (23H2) on NTFS
all of my snapshots have a ~/.kube/config.yaml that links to ~/.kube/config on windows 11 ntfs of a symlink I made with mklink command as admin.
when running restic check it shows tree b45d3f381fb1438fe768ebd161326ced574e7c27ba4740f68165bbf2b2950006: node "config.yaml" with invalid type "irregular"
restic repair index does not solve.
should i open an issue?
EDIT: I just compiled the mentioned above branch and it gives this error now:
Output
$ resticprofile -v storj.check
2024/10/04 16:11:15 resticprofile 0.28.0 compiled with go1.23.0
2024/10/04 16:11:15 using configuration file: C:\Users\Scott\AppData\Local\resticprofile\profiles.yaml
2024/10/04 16:11:15 loading: C:\Users\Scott\AppData\Local\resticprofile\profiles.yaml
2024/10/04 16:11:15 files in configuration are relative to "C:\\Users\\Scott\\AppData\\Local\\resticprofile"
2024/10/04 16:11:15 memory available: 20243MB
2024/10/04 16:11:15 setting priority class BELOW_NORMAL
2024/10/04 16:11:15 IONice is only supported on Linux
2024/10/04 16:11:15 using restic 0.17.1
2024/10/04 16:11:15 profile 'storj': starting 'check'
2024/10/04 16:11:15 command environment:
AWS_ACCESS_KEY_ID=jwybzjvxjugoygelk6zno6bdusgq
AWS_SECRET_ACCESS_KEY=Ă—Ă—Ă—
2024/10/04 16:11:15 starting command: c:\Users\Scott\restic.exe check --password-file=C:\Users\Scott\AppData\Local\resticprofile\restic_password.txt --repo=s3:https://gateway.storjshare.io/nunya --verbose=1
using temporary cache in C:\Users\Scott\AppData\Local\Temp\restic-check-cache-3799556503
create exclusive lock for repository
repository 06d05213 opened (version 2, compression level auto)
created new cache in C:\Users\Scott\AppData\Local\Temp\restic-check-cache-3799556503
load indexes
[0:01] 100.00% 3 / 3 index files loaded
check all packs
check snapshots, trees and blobs
[0:05] 0.00% 0 / 85 snapshotserror for tree b45d3f38:
tree b45d3f381fb1438fe768ebd161326ced574e7c27ba4740f68165bbf2b2950006: node "config.yaml" with invalid type "irregular"
error for tree fd7cfe12:
tree fd7cfe1278fa8f06f9408e6eb63ec6bf69b3fa7df207597869a3aff2bd819fb6: node "config.yaml" with invalid type "irregular"
error for tree aa3c489e:
tree aa3c489e2f0191b73a20389a2cead7d690593cf91355bbfc43c1ba36a51b38b5: node "config.yaml" with invalid type "irregular"
[0:14] 100.00% 85 / 85 snapshots
The repository is damaged and must be repaired. Please follow the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html .
Fatal: repository contains errors
2024/10/04 16:11:32 check on profile 'storj': exit status 1
I think you should attempt to repair the repository with the branch version. Make sure to use --dry-run first. As far as I understand the branch contains some logic to delete the bad/unrecognized items.
That said, I would rather have Restic understand symlinks under Windows, as they are used quite often these days: Windows itself uses it for windows apps, Cygwin makes them, and a bunch of other programs do so too.
Under Unix-like systems, I would assume that Restic stores a symlink as a symlink, and when restoring, it should re-create the symlink as it was.
