Hi,
How would one get rid of this persisting error? Pruning did not do away with it.
error: node "file1", last"file1": nodes are not ordered or duplicate
Thanks much
warnod
I would say if you want help you post more details?
It appeared one day during backup and stayed ever since. I am unsure what caused it.
open repository
lock repository
using parent snapshot <>
load index files
start scan on <>
start backup on <>
scan finished in 16.758s: <>
error: node "file1", last"file1": nodes are not ordered or duplicate
Files: 0 new, 5 changed, <> unmodified
Dirs: 0 new, 5 changed, <> unmodified
Data Blobs: 5 new
Tree Blobs: 6 new
Added to the repository: 232.127 KiB (60.269 KiB stored)
processed <> files, <> GiB in 0:59
snapshot <> saved
What version of restic?
What repo destination?
What command?
You think people can see your screen?
1 Like
This is a filesystem problem not a restic problem; the folder in question contains two files with the same name. Please run fsck / chkdsk to fix. In recent restic versions, this error has been downgraded to a warning so that the backup still completes. But either way please repair the filesystem.
1 Like
@MichaelEischer , this is not a file system error only. This errors mostly appears on UnionFS, and restic somehow see underlying readonly layer as well writable layer that seats on top and reporting it as two similar files instead of backing up only top (most recent, - writable) layer.
Restic just uses the standard posix api calls to list the content of a folder. If these return duplicate files when using UnionFS, then UnionFS is broken.
1 Like
While I having access to that system that produced this error, I made a simple file walker to test GoLang on that UnionFS
package main
import (
"fmt"
"os"
"path/filepath"
)
func main() {
filepath.Walk(".", func(path string, obj os.FileInfo, err error) error {
if err != nil {
return err
}
if obj.IsDir() {
fmt.Printf("-- Dir: [%s] mTime: %v\n", path, obj.ModTime())
} else {
fmt.Printf("File: [%s]\n\tSize: %d,\tmTime: %v\n", path, obj.Size, obj.ModTime())
}
return nil
})
}
and it shows as expected - the only one copy of filenames, even so it is on the same UnionFS where restic reports duplicated filenames
I just took a look at the implementation of filepath.Walk in the latest go version and the listing of files in a directory there is essentially identical to what restic uses internally.
So the question is now why that code snippet behaves differently than restic. Please provide more details on the used restic version, the exact error message, more context information etc.
I’ll be more than happy to do that !
restic version: restic 0.17.3 compiled with go1.23.3 on freebsd/amd64- I used GoLang 1.23.3 amd64 on Linux with cross-compilation to freebsd amd64 to test that file walker snippet.
I got this error on an industrial machine that running FreeBSD amd64. The core OS is starting and running from read-only flash drive (formatted as UFS2) and runs in read only mode with exclusion to /tmp, /proc, ... which are RAM disks. The actual data storage is ZFS based mirrored pool where is one partition is set as zvolume, which means it basically exposed as a plain block device and formatted as UFS2 with soft updates, the same as boot drive, so no conflicts on filesystem level. That ZFS volume then mounted with UnionFS (on FreeBSD it isn’t fuse) on top of read-only directories, such as /root, /usr/local, /var/db, /usr/home and other that supposed to be “changeable”, by running:
/sbin/mount_unionfs -o copymode=transparent -o whiteout=whenneeded -o noatime /upper-writable/layer /lower-read-only/layer
Such configuration helps to separate OS from actual data as well allows to map missing components (as well it vandal resistant). All the programs running without problem on top of UnionFS including snippet I showed above.
When I run restic on those UnionFS mapped location, the mentioned error is popping up only on files that has a second (writable, that seats on top of read only ones) incarnation of original file, the errors shows like that:
start scan on [/usr/local/etc]
start backup on [/usr/local/etc]
...
error: node "driver.list", last"driver.list": nodes are not ordered or duplicate
error: node "driver.list", last"driver.list": nodes are not ordered or duplicate
error: node "cert.pem", last"cert.pem": nodes are not ordered or duplicate
error: node "cert.pem", last"cert.pem": nodes are not ordered or duplicate
error: node "devd", last"devd": nodes are not ordered or duplicate
...
and it finished with error #3 : some source data could not be read (incomplete snapshot created).
I guessing that the same problem would be on Linux if one would use OverlayFS or unionfs-fuse. I will try to play with that (restic with overlayfs on Linux) when get back from trip
EDIT:
I switched to a curious modeand decided not wait on testing and result is:
OverlayFS on Linux working as expected,resticbackup directories and files in overlay directory without problem
That looks like the unionfs on FreeBSD just concatenates directory entries, which sounds rather odd.
But I still don’t have the slightest clue why your filepath.Walk snippet doesn’t produce duplicate entries.
I also don’t think it’s a good idea to let restic silently ignore duplicate files as the previous occurrences of the “nodes are not ordered or duplicate” error were often related to damaged filesystems.
No, it is a job for nullfs. UnionFS doing exactly two layers, combining one (lower) which is original directory (usually read-only) and another one (upper, usually writable layer) that seats on top of lower, so in united directory, where will be listed files from lower layer only in case there no the same filenames in upper layer, but as soon as one created the filename in united directory that match filename in lower layer, the all program seeing the only that file in upper layer that masks file with the same name in lower layer. Basically it is works in the same way as OverlayFS on Linux, but FreeBSD unionfs just allows more low level controls, how layers behaving. The mount option I showed above is replicated OverlayFS behavior - upper layer take precedence in case of name clashing. The simple example:
- vendor supplied original bootable operation system that works in read only mode with SSL certificates that was actual on the moment of distribution (living on lower layer), but over time, on customer premises, when certificate got expired it will be renewed and since lower layer is readonly, then new certificate file will be created on unionfs in upper layer that will mask outdated read-only certificate from lower layer.
I must say sorry for confusion, I ran that snippet on a directory where no files in upper layer and as result snippet showed one copy from lower layer (the only one copy).
I did test this time more carefully, run my snippets on the directory where restic shows duplicated filenames and it behaves the same as restic, - show two files with the same name in case there presented files in lower and upper layers on unionfs.
Since it looks really wrong, I made a plain C program:
#include <stdio.h>
#include <stdlib.h>
#include <dirent.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <time.h>
#include <string.h>
void print_file_type(struct stat *file_stat) {
if (S_ISDIR(file_stat->st_mode)) {
printf("Directory");
} else if (S_ISREG(file_stat->st_mode)) {
printf("Regular File");
} else if (S_ISLNK(file_stat->st_mode)) {
printf("Symbolic Link");
} else {
printf("Other");
}
}
int main(int argc, char *argv[]) {
struct dirent **namelist;
struct stat file_stat;
int n;
if (argc < 2) {
fprintf(stderr, "Usage: %s <directory_path>\n", argv[0]);
return 1;
}
n = scandir(argv[1], &namelist, NULL, alphasort); // alphasort - sorts entries alphabetically
if (n == -1) {
perror("scandir");
return 1;
}
printf("Contents of directory %s:\n", argv[1]);
for (int i = 0; i < n; i++) {
char path[1024];
snprintf(path, sizeof(path), "%s/%s", argv[1], namelist[i]->d_name); // file path
if (stat(path, &file_stat) == -1) {
perror("stat");
continue;
}
printf("%s:\n", namelist[i]->d_name);
printf(" Type: ");
print_file_type(&file_stat);
printf("\tSize: %ld bytes", file_stat.st_size);
char time_buffer[80];
struct tm *tm_info = localtime(&file_stat.st_mtime);
strftime(time_buffer, sizeof(time_buffer), "%Y-%m-%d %H:%M:%S", tm_info);
printf("\tLast modified: %s\n\n", time_buffer);
free(namelist[i]);
}
free(namelist);
return 0;
}
that uses only POSIX’s scandir call and ran it over that folder in question where Golang programs (both, my snippet and restic) producing double names. As expected, scandir returns the only one single filename from upper layer, even so the filename presented on both layers. Also, ls, mc working as expected on top of unionfs directory and “see” the only one filename.
That’s for sure. I started this conversion when found error pattern that isn’t related to broken filesystem.
As of now I’m close to conclusion that it is GoLang in charge of this error. It pretty weird, if it sees two filenames, but it shows both with the same timestamp and size that belong only to the copy that living on the top of upper layer. I think I might go to investigate it further, on different OS and GoLang versions since unionfs as well overlayfs are both in heavy usage on “live/read-only” bootable CD/flash drives
I’d expect that to call _getdirentries with slightly different parameters than Go. In particular the buffersize is likely different. Can you check that using strace (or what the FreeBSD equivalent is called)?