Repository goes to "NOTREADY" after 24 Hours

Getting Help
1

Hi Everyone!

I’ve been using Restic in conjunction with Velero (used to be Heptio Ark). I have been experiencing an issue where Velero reports that the backups have partially failed after 24 hours of the repository being online. If I delete the repository and rebuild it from scratch, it returns to normal operation before crashing again (at the 24 hour mark). Here are the details of the issue that I’ve been able to identify thus far:

Velero Version:

Client:
Version: v1.0.0-rc.1
Git commit: d05f8e53d8ecbdb939d5d3a3d24da7868619ec3d
Server:
Version: v1.0.0-rc.1

Output of “velero restic repo get -o yaml”:

apiVersion: velero.io/v1
kind: ResticRepository
metadata:
  creationTimestamp: 2019-07-07T16:36:00Z
  generateName: media-default-
  generation: 662
  labels:
    velero.io/storage-location: default
    velero.io/volume-namespace: media
  name: media-default-d24zg
  namespace: velero
  resourceVersion: "3812308"
  selfLink: /apis/velero.io/v1/namespaces/velero/resticrepositories/media-default-d24zg
  uid: 4da6e59e-a0d5-11e9-9200-00505687fd92
spec:
  backupStorageLocation: default
  maintenanceFrequency: 24h0m0s
  resticIdentifier: s3:http://172.25.11.35:9000/velero/restic/media
  volumeNamespace: media
status:
  lastMaintenanceTime: 2019-07-09T20:29:50Z
  message: |-
    error running command=restic init --repo=s3:http://172.25.11.35:9000/velero/restic/media --password-file=/tmp/velero-restic-credentials-media960455502 --cache-dir=/scratch/.cache/restic, stdout=, stderr=Fatal: create key in repository at s3:http://172.25.11.35:9000/velero/restic/media failed: repository master key and config already initialized

    : exit status 1
  phase: NotReady

As near as I can tell, it looks like the repository is locked? (repository master key and config already initialized)

I cannot find anything on how to solve this.

Any help that can be provided would be great!

Thanks!

2

You’re running restic init on a repository that has already been initialized. Don’t do that. :slight_smile:

3

LOL

So new question, how do I stop velero from doing that? I have a scheduled backup that runs every 3 hours and completes within 10 minutes from the start. Is it leaving it initialized and trying to re-initialize again after 3 hours?

4

No idea, I don’t use velero.

That’s what it looks like. FWIW, “leaving it initialized” is basically what you want. An uninitialized repository is empty and has no keys. If you delete the keys and config, all the existing data is lost.

You should only run restic init one time per repository. This creates the config and initial key. Running it a second time makes no sense, as the config and key already exist.

5

Ok, I’ll reach out to Velero and see if they can help me identify why it’s intializing when it’s already initialized.

Thanks!

6

Just to circle back, I submitted the following with Velero:

The following fix is in the works:

It seems to definitely be an issue with Velero.