I am restoring some large image files (50GB) using restic restore.
It appears that restic pre-allocates restored files (by writing zeroes?) first before restoring?
Never noticed it before restoring small files but when I restore a 50GB file, it takes a few minutes to pre-allocate the files before it even starts restoring.
Not sure of the reasoning for that, but it slows the restore down, especially if restoring to spinning rust since you essentially are writing twice…
Wondering if anyone can shed some light on why restic does that, and if it is necessary.
No:) Filling a file with zeros would be waste of time. What restic does is preallocating space for a file.
If you are on *nix OS you can try it yourself:
fallocate -l 100G test.file
and voila 100G size file will be created in no time.
So no - there is no writing anything twice.
This is quite typical approach when you have to create large file and fill it with content which might often come out of order. All torrent clients do it for example. It is actually much faster than trying to do it without preallocation.
On windows, it takes several minutes or longer for restic to pre-allocate the file (on NTFS)
However, I tried using the --sparse option with restore and that seems to fix it.
This is different thing. It means that your files contain long runs of zero bytes and by using --sparse restore process can skip writing them.
--sparse is actually somewhat related to the problem here. On Unix the preallocation in the filesystem works by adding a new range to the file but marking it as uninitialized. The only difference to a sparse file is whether the OS already allocates disk space for those ranges or not.
I’m not entirely sure what is supported by NTFS. So we might have a case here that Windows does not have such uninitialized sections or they work differently. Either way, it’s probably still a good idea to open a Github issue. (Although that will depend on someone with more Windows knowledge taking a look)