No output when backing up to Backblaze B2

gened · July 30, 2021, 10:07pm

The host does change – but there don’t appear to be any successful transactions with this host in previous instances of this log (which is 532Mb currently).

interesting…

I’ll relay this info to B2

Thanks for the tip.

G

gened · July 30, 2021, 10:11pm

I will rerun the script with the B2_LOG_LEVEL envar and report back.

This process runs over several hours – hopefully I can trip this error again.

G

gened · July 30, 2021, 10:18pm

Attempting to connect to that machine via curl returns nothing (just sits there).
Attempting to connect to other machines listed in the logs exits with Connect reset by Peer…

[gene@tws12 tmp]$ curl pod-000-1156-02.backblaze.com
curl: (56) Recv failure: Connection reset by peer
[gene@tws12 tmp]$ curl pod-000-1162-18.backblaze.com

I’m starting to wonder whether that node is borked.

G

MichaelEischer · July 30, 2021, 10:23pm

Well that host probably has some temporary network problems. These are usually fixed within a few hours.

gened · July 30, 2021, 10:24pm

I hope that you are right – but I’ve not managed to get this script to complete in a few weeks!
(it’s currently backing up 240-some active projects…)

MichaelEischer · July 30, 2021, 10:34pm

Hmm, there have been some hickups with B2 from time to time, but I don’t think that the lasted longer than a few days. A few weeks definitely sounds like too long, so there might be something wrong noone has noticed so far.

If I understand the B2 library used in restic correctly, then it should request a new upload url/server after each failed upload. That should only cause a few failed upload attempts before settling on a working server. So, I can’t really exclude that there’s a bug somewhere in the error handling…

gened · July 31, 2021, 8:00pm

Thanks for the insight Michael.

Is there anything else I can do on my end to help drill down to the cause of this problem?

MichaelEischer · August 3, 2021, 6:36pm

The log output generated via B2_LOG_LEVEL (at least I hope that that option works) could be interesting, if it captures the problem.

gened · August 3, 2021, 6:58pm

Heya Michael,

Thanks for following up. I’ve re-run the process with the B2_LOG_LEVEL envar set to 2, but I can’t see anything different about the output in the logs. Are we expecting anything different in the restic log? or is the B2 data meant to be logged elsewhere?

MichaelEischer · August 4, 2021, 6:49pm

I’d have expect the log entries to be printed on stdout/stderr or when restic exits. But apparently these are only shown if restic exits with an non-zero exit code, sorry for that. The fix for that should be to comment out the following line:

github.com

restic/restic/blob/7049f1cbfca022fc267a987c1b13136c1759b7ba/cmd/restic/main.go#L91

    
      
          	default:
          		return true
          	}
          }
          
          
var logBuffer = bytes.NewBuffer(nil)
          
          
func init() {
          	// install custom global logger into a buffer, if an error occurs
          	// we can show the logs
          	log.SetOutput(logBuffer)
          }
          
          
func main() {
          	debug.Log("main %#v", os.Args)
          	debug.Log("restic %s compiled with %v on %v/%v",
          		version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
          	err := cmdRoot.Execute()
          
          
	switch {
          	case restic.IsAlreadyLocked(errors.Cause(err)):

gened · August 4, 2021, 11:57pm

Thanks for the follow up Michael,

In the meatime, the B2 team ended up putting that pod in a state where it is not prepared to receive more data – so my data will get redirected to another location, and I can complete the backups. I’m not certain whether they will get back to me with an epilogue – but if they do, i’ll be sure to share that here for posterity.

Thank you again for the nudges. It was very helpful.

G