Good day.
Firstly, thank you for this fine restic utility. We’ve been using it for a few months and are we impressed with its capabilities.
This is probably our first major problem with the tool and we figured that it was important enough to report it.
We are currently using restic 0.9.5 for a large data recovery project to assist one of our customers.
The source system with the damaged data set is a FreeBSD system. Restic was installed on this and the backup target was a Linux system running MinIO. There was a overheating problem on the FreeBSD system which prevented the restic snapshot from being fully formed. So we used “restic recover” to build the snapshot.
We moved the HDDs from the failing FreeBSD box to another FreeBSD box. Ran another “restic backup” to the same restic repository from the previous attempt and this seemed to create a viable snapshot.
This new snapshot should contain everything from the previous snapshot plus some additional data that the last snapshot did not receive. “restic ls” seems to show us all the files within the new snapshot.
There are over 9 million files which equate to at least 50 TB.
Now we want to restore that new snapshot onto another Linux system with restic installed on it. We used “restic restore” and within a few minutes I could see the restoration of all the stored directories (or so it seems) within the snapshot. Hours later, no files get restored. 24 hours later, still no files but the “restic restore” process is still running. Stopped the restore, tried “restic check” and removed an old lock. Ran the restore again…still no files.
We broke out of the “restic restore” process and instead switched to “restic mount”.
Since then, I’ve been using one rsync process at first to copy data from the mount to the intended restored directories. I’ve increased the count to three different rsync instances…each pointing to different restore target locations. It’s been going solidly for almost a day now but just slower than we would expect probably due to the FUSE underpinnings perhaps?
Anyway, why would “restic restore” fail to restore files when directories are restored? Is there something unusual about the size of the snapshot and the total of files which is stressing restic’s design characteristics?
I’m not sure how much experimentation we can do at this time since we are rushing to get this data out of the snapshot via mount and onto the new Linux box. But perhaps when this project is over we can take a closer look to see where the bottleneck or failures exist.
Thanks.