Looking at my logs again, this problem has only happened twice (on two separate clients). The first client randomly tried to delete a blob during
backup, got a
append-only), and eventually tried posting the blob again and succeeded. Snapshot complete, no infinite loop.
Further investigation on my second client shows something else. It tried posting two data blobs. The first one, let’s call it
x, succeeded with
200. The second one,
y, failed with
502 (possibly restic-server crashed so proxy returns a 502?).
502, the client ran in an infinite loop as follows and crashed: (1) Attempt to delete blob
x. (2) Attempt to post blob
Perhaps the restic client, after receiving
502 on blob
y, thought that blob
x wasn’t posted and tried to post it again (perhaps safety feature? restic failed blob
y so wants to make sure
x is there? which is weird because client should know
x is there as it got a 200).
Comments on (2): Looking at the restic-server code, the client may have ran into this: https://github.com/restic/rest-server/blob/a87d968870b4dec64abfe1270e538fdc01524a81/handlers.go#L519-L521. The code here says that if the blob/data already exists on the server, then return
Anyways I’m not sure if this is a problem with the restic server or client. One fix I think should go in is changing the code on the server to not return a
403 if the data blob already exists on the server. Instead return a
200. Perhaps then if the client ever runs into this again it won’t go in this endless loop of deleting and posting as it will see the data finally got posted successfully.