Restic with wasabi repository - Issues after init and first backup


#6

Thank you for the reformatting and the ideas.

It is:

I will check what you mention using mc client. I was able to mount the s3 instance with s3fs and my credentials, and to list a file that I uploaded separately from the backup.


#7

Thank you, This may probably be the case. I have versioning enabled. I do not know if it is related to this but I noticed that when I tried to download the file in question, it is not possible and I get a message about missing file or so.
I will check later with “showing versions” and see how it goes.


#8

@cdhowie

Regarding this:

I see that there is only one version of the lock and it was created at the time of initializing my last repo.

But there are several versions of the “config” file, one for every repo I tried to use and later deleted. I will check later if deleting these config files helps.


#9

Deleting the config files should not matter. That’s not what restic is complaining about.


#10

Thank you.

Additionally, to this point

I was now able to download that lock file.


#11

@cdhowie

Given that

I wonder if you could expand on your advice given that there is only one version of the lock:

Should I try to delete it and see if that helps or should I be only do that if there were old versions?
And if I delete it, should I delete the file or file + directory?

I just found something interesting about the locks. It is the only file in the repository that when I toggle between enable showing versions and not enabling showing versions, there is a change in the way the file is presented.

When showing versions is not enabled:

When showing versions is enabled:

It is the only file where there is one version of a file that shows this behavior. All other files/directories where there is only one version only show a picture like the first picture even when toggling between enabling and not enabling showing versions. I do not know if this is completely relevant but wanted to add the information.

Thank you.


#12

I am using wasabi without issue, both backing up and restoring. My policy is slightly different to yours:

“Action”: [
“s3:PutObject”,
“s3:GetObject”,
“s3:DeleteObject”
],

You don’t have the DeleteObject.


#13

I stand a bit corrected. I see you have an additional policy for the lock. Does this mean you will never do a prune?


#14

Thank you.

I started with a policy like yours and then following:

And also following:

I made the changes.

I could return to:

And see what happens.

Do you think that I should keep the specific?

Regarding

I am still not familiar with that operation but it suggest some cleaning and that I should have a broader DeleteObject policy.


#15

I am not a restic expert so I’ll let them give you specific advice. In my case I back up just under 2TB of personal data and don’t worry about other users. Therefore I don’t worry about append only operation and I don’t use versioning. Once a week I do a forget keeping 90 days and then prune.


#16

Thank you, it is good to know restic + wasabi is working for you.
Also good to know about forget and prune.


#17

Hello,

I was able to read a file in the S3 bucket.

 #s3fs xxxxx /s3mnt -o passwd_file=~/.pwd-s3fs -o url=https://s3.wasabisys.com
 #cd /s3mnt
 #ls
 config  data  index  keys  locks  snapshots  test2.txt 
#more test2.txt
Test 2

I am now trying to see if deleting the only version of locks or maybe some little change in the IAM policy will work.


#18

@cdhowie @fd0
Well, I made some progress and it seems to be a very peculiar situation.
Lock file still there, usual situation:

~$ restic backup test2.txt
open repository
repository bc0455dc opened successfully, password is correct
Load(<lock/b3e52e2f6d>, 0, 0) returned error, retrying after 647.799643ms: The specified key does not exist.
 signal interrupt received, cleaning up

I deleted the lock file (only way was to enable showing versions and delete it, if I unable showing versions it is the same file and it will not be deleted, the same file changes status between current and past version as I showed in Restic with wasabi repository - Issues after init and first backup (this intends to point to post #11 in this topic)

After deleting the lock file, success again

~$ restic backup test2.txt
open repository
repository bc0455dc opened successfully, password is correct

Files:           0 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 0 B  

processed 1 files, 7 B in 0:01
snapshot da1688bd saved

Now trying again with the lock file there, I go back to the previous situation about the lock file.

Now I delete the lock file again and new success

~$ restic snapshots
repository bc0455dc opened successfully, password is correct
ID        Time                 Host        Tags        Paths
------------------------------------------------------------------------------
7dd44441  2019-01-20 23:22:15  yyyy-yy              /home/xxxx/test2.txt
da1688bd  2019-01-21 19:02:56  yyyy-yy              /home/xxxx/test2.txt
------------------------------------------------------------------------------
2 snapshots

Now that the lock file is there again, any other operation brings me back to the problem.

Not sure what else I can try. Any other suggestion will be most welcomed.


#19

@cdhowie @fd0

Ok now it seems to work.

  1. Deleted locks
  2. Unabled bucket versioning in my bucket

Now repeated restic commands work without the need to manually delete the lock

~$ restic snapshots
repository bc0455dc opened successfully, password is correct
ID        Time                 Host        Tags        Paths
------------------------------------------------------------------------------
7dd44441  2019-01-20 23:22:15  yyyy-yy              /home/xxx/test2.txt
da1688bd  2019-01-21 19:02:56  yyyy-yy              /home/xxx/test2.txt
------------------------------------------------------------------------------
2 snapshots
~$ restic snapshots
repository bc0455dc opened successfully, password is correct
ID        Time                 Host        Tags        Paths
------------------------------------------------------------------------------
7dd44441  2019-01-20 23:22:15  yyyy-yy              /home/xxxx/test2.txt
da1688bd  2019-01-21 19:02:56  yyyy-yy              /home/xxxx/test2.txt
------------------------------------------------------------------------------
2 snapshots
~$ restic backup test2.txt
open repository
repository bc0455dc opened successfully, password is correct

Files:           0 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 0 B  

processed 1 files, 7 B in 0:01
snapshot 1925f42d saved
~$ restic snapshots
repository bc0455dc opened successfully, password is correct
ID        Time                 Host        Tags        Paths
------------------------------------------------------------------------------
7dd44441  2019-01-20 23:22:15  yyyy-yy              /home/xxxx/test2.txt
da1688bd  2019-01-21 19:02:56  yyyy-yy              /home/xxxx/test2.txt
1925f42d  2019-01-21 19:49:10  yyyy-yy              /home/xxxx/test2.txt
------------------------------------------------------------------------------
3 snapshots

Is this the way the bucket has to be set, i.e. not enabling bucket versioning? Does that have any consequence for the performance of restic?

Thank you for all the guidance :smile:.


#20

Versioning should not break restic. It sounds like Wasabi is including deleted objects when retrieving a listing of objects. This would either be a bug on Wasabi’s side, or restic is not using the Wasabi API correctly (e.g. when listing objects, it’s asking it to include versioned objects whose most recent version is a delete marker).

Versioning should have no impact on performance.

The point of versioning is to enforce that an API key cannot irreversibly delete/corrupt data in the repository. If some ransomware gets on the machine performing backups and the API key can overwrite or permanently delete objects, you might as well not have backups in the first place. If you turn on versioning, and use an API key that is not allowed to delete prior versions of objects, then you can always roll the bucket back to a point in time when the repository contents were valid.

tl;dr: Versioning is about security, not performance.


#21

Wasabi say that restic is certified for use, so you could contact their support


With a link to this topic and they might be able to identify the issue.


#22

@cdhowie
Thank you for the explanation. It helps to understand the advantage of having versioning enabled.

Given your statement

This could be an issue on either side, right?

@D.O.Scott

Thank you. Good idea, I will do that and report back if anything useful comes back.


#23

@cdhowie @D.O.Scott

While I wait for Wasabi support to respond, can I make a viable case if I enable versioning and delete the locks file after every operation so I can perform further backups etc.?

While not optimal, will this give the security features that @cdhowie described or would the locks deletion each time impair to roll back previous versions of the objects should I need to?

Thank you.


#24

Personally I would suspend versioning until you get an answer rather than go through that hassle. You are using Linux and as far as I know there are no proven ransomware attacks, so the risk is minimal.


#25

@D.O.Scott

Thank you for your advice! Appreciated.