I don’t think so. Setting the environment variables GOARCH and GOOS for cross-compilation worked flawlessly since Go 1.5 or so (that’s what build.go does internally).
Please try the following: Create a new file:
package main
import "fmt"
func main() {
fmt.Println("test")
}
Then run:
$ GOOS=windows GOARCH=amd64 go build -o main.exe main.go
$ file main.exe
main.exe: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
I suspect that the command go does something unexpected on your system and does not pass on the GOARCH and GOOS environment variables.
Where did you get the version of Go you’re using? Go 1.8.1 is rather old (1.8.3 or 1.9 are the current versions).
Ok, that looks good. Honestly, I have no idea what’s going on here. That’s basically what build.go does: Copy all the files to a temporary directory, set the environment variables and call go build with them in the temporary directory.
It’s VMware Photon (a purpose built extremely tiny Linux image optimised for VMware), and Go comes from TDNF (a yum clone?) which is how Photon is updated, probably from this repo (no idea who bintray is):
You could try with the official Go version (1.9 is best): Download it from https://golang.org/dl/, extract it somewhere, then call $DIR/bin/go with build.go as usual.
By the way: You can also do that on Windows, no Linux required…
Now I immediately started it again, but instead of just running the initial scan + possible uploading a few KiB/MiB of recent changes since then, it seems to re-calculate the hashes for the whole filesystem (no upload activity during this time):
Unfortunately, the debug.log grew too large and filled up my remaining free space, so I had to truncate -s10M debug.log it. Will try to reduce verbosity for the next run or free up some more space to be able to provide valuable debug information.
What I’m currently seeing it when tail-ing it, is stuff like this:
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2574 id ef6633bf not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2566 id 9913ca9d not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2574 id ef6633bf not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2566 id 9913ca9d not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2574 id ef6633bf not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2566 id 9913ca9d not found
2018/01/05 14:59:15 archiver/archiver.go:545 archiver.(*archivePipe).compare 2542 old job: home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/metadata.xml
2018/01/05 14:59:15 archiver/archiver.go:546 archiver.(*archivePipe).compare 2542 new job: home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/metadata.xml
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2574 id ef6633bf not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2566 id 9913ca9d not found
2018/01/05 14:59:15 archiver/archiver.go:556 archiver.(*archivePipe).compare 2542 same filename "home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/metadata.xml"
2018/01/05 14:59:15 archiver/archiver.go:597 archiver.archiveJob.Copy 2542 job home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/metadata.xml is file
2018/01/05 14:59:15 restic/exclude.go:226 main.rejectByDevice.func1 2541 item /home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/texlive-documentation-mongolian-2012.ebuild, test dir /home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian
2018/01/05 14:59:15 restic/node.go:555 restic.(*Node).IsNewer 2542 node /home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/metadata.xml is not newer
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2570 id 970a7ce9 not found
2018/01/05 14:59:15 archiver/archiver.go:610 archiver.archiveJob.Copy 2542 job home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/metadata.xml add old data
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2570 id 970a7ce9 not found
2018/01/05 14:59:15 restic/exclude.go:226 main.rejectByDevice.func1 2541 item /home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive/texlive-documentation-mongolian/texlive-documentation-mongolian-2012.ebuild, test dir /home/elias/code-repositories/public/gentoo/repositories/gentoo/dev-texlive
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2570 id 970a7ce9 not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2574 id ef6633bf not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2576 id 1e17e09a not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2570 id 970a7ce9 not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2574 id ef6633bf not found
2018/01/05 14:59:15 repository/index.go:142 repository.(*Index).Lookup 2570 id 970a7ce9 not found
Ah, that’s a bummer. I still think it’s a variant of #1290. Would you like to run more tests in order to find out what’s going on? If so, the next step would be to play around with the backup target paths a bit more:
Backup only a (small) single dir /home, then run a subsequent backup. The second backup should not read the files and be very fast
Backup two dirs, e.g. /home and /etc, that should be slow on the first, and fast on the second backup.
This way, slowly work your way up to what /* expands to, make a note when the second backup in each step is slow too.
By the way, the effect that you’re observing is independent of the backend, you could also use a local (or sftp) backend if you have the space locally.
Hm, maybe we can still tune that a bit more. I’d need to dig into the details to find out what’s going on here. The debug message is about that the file is to be excluded because it’s not on one of the allowed file systems (so the exclude is by --one-file-system, but it shouldn’t need to traverse the dir, because /proc/21186 should already have been excluded. Hm.
Didn’t get around to this yet - will hopefully find some time during the next days to look into this.
The problem here seems to be, that the expansion of /* includes excluded paths again (the ones excluded by --one-file-system as well those excluded by --exclude= or --exclude-file), as also indicated by this output line:
e.g.: /dev or /proc - should be excluded by --one-file-system /nix - should be excluded by the line /nix in the file referenced by --exclude-file=/etc/restic/moria/backup.exclude
For /tmp it’s reasonable to appear and for the directory itself to be scanned (but not its content):
it was included by the expansion of /*
only its sub-items are excluded by --exclude=/tmp/*
it’s a separate filesystem (tmpfs), but this also doesn’t apply to /tmp itself, but only sub-items
Ah, don’t worry about the scan line, it’ll just show what you pass restic backup in verbatim, the exclusion logic happens later. I’m sure that e.g. /dev and /proc are still excluded.