I have been testing Restic, and I’ve come across an issue where the 2nd backup is as intensive and slow as the first backup. My test directory is 413GB, with 8448 directories, 150537 files.
The initial backup took something like 3 hours. During backup, CPU is close to 100% (it is a Celeron G1610T, so it’s not very powerful). After a successful initial backup (I have RESTIC_REPOSITORY and RESTIC_PASSWORD set):
D:\Tools\Restic>restic snapshots
ID Date Host Tags Directory
bdfbd30e 2017-09-08 16:54:17 piccolo D:\Data\Users
The repository is a local USB 3 drive (U:\Restic), and the source is a local SATA hard drive. This is on Windows 2012 R2, running the latest version of Restic as of a few days ago.
As you can see, the 2nd backup is going to take almost the same amount of time as the original backup. The CPU is very high (99%), and while the backup is running, Restic is reading through all the files (I checked on the Windows Resource Monitor):
Your CPU doesn’t have AES-NI, so that will definitely slow things down as far as encryption goes.
But that said I’m thinking the second backup should be quicker than the first, restic shouldn’t have to encrypt all the stuff again, right? I don’t know.
That’s what I also suspect, this matches the observed behavior: Restic re-reads and re-hashes all files, but does not upload anything new to the repo (because almost all data is already there). Hm, interesting case. I think this is a bug. Would you mind creating an issue in the GitHub issue tracker?
I encountered similar behavior on Linux and created a new issue. I believe it’s caused by the way restic compares time when the system timezone is UTC. I wonder if the timezone on this Windows box is also UTC?
Thanks. I’ve added a comment on the issue. I don’t think it is applicable to me, as a) Windows doesn’t give you the option of setting the clock to UTC (independently of timezone), and b) although I am in the UK, it is British Summer Time (daylight savings) right now, so not at UTC anyway.
I’ve spent an hour or so trying to get the build environment to work, but not having a lot of luck. Managed to build the docker container; it’s not in the official docker repository (why not?) so I had to figure out that I needed to download the Dockerfile first.
git clone https://github.com/restic/builder.git
cd builder
docker build -t restic/build .
Ok, now I have no idea how to download tar.gz of the source code.
cd ..
git clone https://github.com/restic/restic.git
tar -czvf restic-source.tar.gz restic/
Now when I try to run the docker build command I get this:
Ah, interesting. I thought it was pretty clear in the manual: Install Go, clone the repo, then run go run build.go, that’s it. No need for Docker. I’m sorry for the poor user experience!
Docker is only interesting for reproducing exactly the released binaries. The Docker image used for that (repo) is online: https://hub.docker.com/r/restic/builder/ and you should be able to get it with docker [run|pull] restic/builder. It is not meant to compile anything else than a .tar.gz from an official release. So no wonder you got nowhere with giving it a tar of the restic repo checkout…
We probably need to make sure that the build.sh script in the builder container is executable. I’ll add that. Thanks for the pointer!
I’m very interested if you have any idea on how to improve the documentation!
go run build.go
go run build.go --goos windows --goarch amd64
…both yield identical files, with the difference that one has .exe on the end. Given that I am running this on a Photon OS VM, that can’t be right! In other words I’m trying to run the Linux binary on a Windows machine, no wonder it doesn’t work!
$ go run build.go
$ ./restic version
restic 0.7.2 (v0.7.2-16-ge91749bb)
compiled with go1.9 on linux/amd64
$ go run build.go --goos windows --goarch amd64
$ file restic.exe
restic.exe: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
$ md5sum restic*
9613b514b94484fc039a1fd772baac1e restic
a40fa0098bdda62092205c2f2763eaf7 restic.exe