Is this error normal and will it recover automatically?

I am using Restic to backup to Backblaze using their S3 compatible API. I ran a backup for about four hours last night and it logged about 33 errors. Most are in the format “Save(…) returned error, retrying after…”.

  1. Is this normal?
  2. Does it retry the backups and would I get another error if it was unable to do it after a retry?
  3. Or does it just never get that file backed up?

Sample of output:


Save(<data/db73f3ee54>) returned error, retrying after 552.330144ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/55b672c688>) returned error, retrying after 720.254544ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/e96702961e>) returned error, retrying after 582.280027ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/29967688bd>) returned error, retrying after 468.857094ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/ad449d2b88>) returned error, retrying after 462.318748ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/0abe4110d5>) returned error, retrying after 593.411537ms: client.PutObject: Put "https://s3.us-west-002.backblazeb2.com/...": write tcp 192.168.1.xxx:54272->149.137.xxx.xxx:443: use of closed network connection
Save(<data/811f323c2a>) returned error, retrying after 282.818509ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/4b8c01b900>) returned error, retrying after 328.259627ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/643dc7ab8d>) returned error, retrying after 298.484759ms: client.PutObject: Put "https://s3.us-west-002.backblazeb2.com/...": write tcp 192.168.1.xxx:53908->149.137.xxx.xxx:443: use of closed network connection
Save(<data/89da17c0f3>) returned error, retrying after 400.45593ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/aad2501a2b>) returned error, retrying after 507.606314ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/03171a4908>) returned error, retrying after 656.819981ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/a6dcd2b5f8>) returned error, retrying after 357.131936ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/7115c988ae>) returned error, retrying after 440.328595ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/f9dcd5afae>) returned error, retrying after 409.029087ms: client.PutObject: An internal error occurred.  Please retry your upload.
Save(<data/94f60f052c>) returned error, retrying after 484.444922ms: client.PutObject: An internal error occurred.  Please retry your upload.

I never have these problems, but then again I don’t use B2. It is what it is, and it’s not uncommon to see retries. It can be due to the service you communicate with, or your Internet connection, or other things.

If restic in the end isn’t able to back up your data successfully it will tell you. If it did tell you that it saved a snapshot and told you the ID of that snapshot, then it has completed successfully.

It might still be a good idea to use the check command to verify the integrity of your repository, considering you are getting errors from the backend. It does seem a bit fishy.

This does not give us the full picture, it’d be better if you provide the complete output.

There is not much more to the output. It never completed because it timed out. I do that so it does not cost so much and tie up my internet during the day. It should complete and make a snapshot this evening or maybe the next. Time stamps would be nice.

open repository 
lock repository 
load index files 
no parent snapshot found, will read all files 
start scan on [/data/path/nextcloud/ /home/some/config/] 
start backup on [/data/path/nextcloud/ /home/some/config/] 
scan finished in 3.254s: 39065 files, 84.352 GiB 
Save(<data/db73f3ee54>) returned error, retrying after 552.330144ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/55b672c688>) returned error, retrying after 720.254544ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/e96702961e>) returned error, retrying after 582.280027ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/29967688bd>) returned error, retrying after 468.857094ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/ad449d2b88>) returned error, retrying after 462.318748ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/0abe4110d5>) returned error, retrying after 593.411537ms: client.PutObject: Put "https://s3.us-west-002.backblazeb2.com/...": write tcp 192.168.
1.xxx:54272->149.137.xxx.xxx:443: use of closed network connection 
Save(<data/811f323c2a>) returned error, retrying after 282.818509ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/4b8c01b900>) returned error, retrying after 328.259627ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/643dc7ab8d>) returned error, retrying after 298.484759ms: client.PutObject: Put "https://s3.us-west-002.backblazeb2.com/...": write tcp 192.168.
1.xxx:53908->149.137.xxx.xxx:443: use of closed network connection 
Save(<data/89da17c0f3>) returned error, retrying after 400.45593ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/aad2501a2b>) returned error, retrying after 507.606314ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/03171a4908>) returned error, retrying after 656.819981ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/a6dcd2b5f8>) returned error, retrying after 357.131936ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/7115c988ae>) returned error, retrying after 440.328595ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/f9dcd5afae>) returned error, retrying after 409.029087ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/94f60f052c>) returned error, retrying after 484.444922ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/7ee0eb01e0>) returned error, retrying after 391.517075ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/0a4ace4c86>) returned error, retrying after 396.550928ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/d7da8f35f3>) returned error, retrying after 589.542338ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/170d11b661>) returned error, retrying after 359.276526ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/36178e2e15>) returned error, retrying after 351.593438ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/eacc62485b>) returned error, retrying after 430.435708ms: client.PutObject: Put "https://s3.us-west-002.backblazeb2.com/...": write tcp 192.168.
1.xxx:37874->149.137.xxx.xxx:443: use of closed network connection 
Save(<data/eb2e8427fd>) returned error, retrying after 535.336638ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/e06bb01354>) returned error, retrying after 681.245719ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/26a3ff679e>) returned error, retrying after 396.557122ms: client.PutObject: Put "https://s3.us-west-002.backblazeb2.com/...": write tcp 192.168.
1.xxx:54102->149.137.xxx.xxx:443: use of closed network connection 
Save(<data/5e4aa4de12>) returned error, retrying after 398.541282ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/1b07f3f905>) returned error, retrying after 626.286518ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/a19db97a54>) returned error, retrying after 353.291331ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/06a517a755>) returned error, retrying after 682.667507ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/9830d5d36d>) returned error, retrying after 598.359583ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/d7a9098d79>) returned error, retrying after 511.910153ms: client.PutObject: An internal error occurred.  Please retry your upload. 
Save(<data/5c54d84780>) returned error, retrying after 264.151541ms: client.PutObject: An internal error occurred.  Please retry your upload.

Restic will not have to upload again what was already uploaded, so it’s fine to just re-run it to continue the backup process.

Maybe check if there are any logs on the B2 side of things, shedding light on all those errors.

As that error is only reported once, this means that the retry (not reported by restic) was successful. Since restic 0.15.0 there’s an indication, that these retries were successful.

To get timestamps you could pipe the output of restic through ts.

Apart from that, the errors shows that the underlying problem happens on the B2 side. To be on the safe side, it might be a good idea to run restic check --with-cache (–with-cache should avoid most downloads from B2) to ensure that all expected files exists.

2 Likes