Repository is already locked

cdhowie · January 8, 2019, 4:45am

Try putting --no-lock after /usr/bin/restic. Generally, options/switches should always be before non-option arguments.

It’s possible that restic takes everything after cat lock to be an ID ("show me the locks eeb53... and --no-lock) which obviously doesn’t make sense, and prevents --no-lock from actually working.

sudo -E -u restic -- /usr/bin/restic --no-lock cat lock eeb53d35a1530ddbec1705d9c0b75eb12ab329f9cb11c46fd9f34008a1245ae3

If this command doesn’t work, then you’ve found a bug.

Bharathkumarraju · January 8, 2019, 4:49am

Yes, Tried with the same as well but no luck unfortunately.

[admin@prod-db-barman001 ~]$ sudo -E -u restic -- /usr/bin/restic --no-lock list locks
repository 3c742af6 opened successfully, password is correct
eeb53d35a1530ddbec1705d9c0b75eb12ab329f9cb11c46fd9f34008a1245ae3

[admin@prod-db-barman001 ~]$ sudo -E -u restic -- /usr/bin/restic --no-lock cat lock eeb53d35a1530ddbec1705d9c0b75eb12ab329f9cb11c46fd9f34008a1245ae3
repository 3c742af6 opened successfully, password is correct
Fatal: unable to create lock in backend: repository is already locked exclusively by PID 19309 on ip-172-31-254-13.eu-central-1.compute.internal by centos (UID 1000, GID 1000)
lock was created at 2019-01-07 12:03:44 (16h45m8.130721014s ago)
storage ID eeb53d35
[admin@prod-db-barman001 ~]$

fd0 · January 8, 2019, 7:31am

Ok, hm. Cat you please try with the following patch applied:

diff --git a/cmd/restic/cmd_cat.go b/cmd/restic/cmd_cat.go
index e735daf8..57877701 100644
--- a/cmd/restic/cmd_cat.go
+++ b/cmd/restic/cmd_cat.go
@@ -39,11 +39,11 @@ func runCat(gopts GlobalOptions, args []string) error {
                return err
        }
 
-       lock, err := lockRepo(repo)
-       defer unlockRepo(lock)
-       if err != nil {
-               return err
-       }
 
        tpe := args[0]

this disables locking for the cat command.

Bharathkumarraju · January 8, 2019, 7:55am

oh my god, dynamic deployment of code. Have to generate new rpm?

fd0 · January 8, 2019, 10:13am

Uhm, I’m not sure what you’re talking about, but you can easily build restic yourself (even from a different OS, closs-compilation in Go is easy): Install Go, clone the repo, apply the patch, then use go run build.go --goarch amd64 --goarch linux.

The resulting restic binary can be used on any 64 bit Linux, you can copy it to one of the servers and run it. Or give it the necessary credentials to acces the repo and run it locally.

Bharathkumarraju · January 14, 2019, 12:54pm

looks i find out the issue, SInce we are running restic forget --prune from another ec2 instance it is taking more time to run and complete the job. So our restic backup getting failed saying that the restic lock already has been created

any work around for this?

cdhowie · January 14, 2019, 4:10pm

The simplest workaround is to schedule operations requiring an exclusive lock for when backups won’t be happening for several hours.

Bharathkumarraju · January 21, 2019, 12:46pm

ah thank you.
is restic backup always checks for the token on snapshot? I mean once we trigger backup it should work right eventhough backup could take 5 hours of time? our AWS session token lasts for one hour only, is that a problem for the restic?

fatal: [172.20.32.103]: FAILED! => {"changed": true, "cmd": "/opt/bin/perform-backup.sh", "delta": "2:34:39.923047", "end": "2019-01-21 12:37:10.403454", 
"msg": "non-zero return code", "rc": 1, "start": "2019-01-21 10:02:30.480407", 
"stderr": "Save(<data/c5d2fb6918>) returned error, 
retrying after 709.461956ms: client.PutObject: We encountered an internal error. Please try again.\n
Save(<data/cd324479a3>) returned error, retrying after 384.108656ms: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 746.156735ms: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 339.859476ms: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 1.060933735s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 944.39985ms: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 626.969894ms: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 1.530815463s: client.PutObject: The provided token has expired.\n
ave(<data/cd324479a3>) returned error, retrying after 621.447617ms: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 1.2520675s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 2.362924457s: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 2.320850891s: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 1.704713704s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 2.738744502s: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 1.504866011s: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 2.664273974s: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 2.109485187s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 3.307287802s: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 5.564221834s: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 6.989004659s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 6.664668763s: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 7.53732321s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 5.341114202s: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 7.872190029s: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 7.754236646s: client.PutObject: The provided token has expired.\n
Save(<lock/a51e105692>) returned error, retrying after 16.242143546s: client.PutObject: The provided token has expired.\n
Save(<data/7fdc767012>) returned error, retrying after 10.939491392s: client.PutObject: The provided token has expired.\n
Save(<data/cd324479a3>) returned error, retrying after 8.151585845s: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 389.807142ms: client.PutObject: The provided token has expired.\n
unable to refresh lock: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 708.474529ms: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 462.64464ms: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 1.628069526s: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 579.68295ms: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 925.724046ms: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 1.800814626s: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 852.550639ms: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 1.677193271s: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 4.043006998s: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 5.436099694s: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 3.562825097s: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 6.552883951s: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 4.447282508s: client.PutObject: The provided token has expired.\n
Save(<index/9fcc1fb746>) returned error, retrying after 16.266748172s: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 6.81074462s: client.PutObject: The provided token has expired.\n
Save(<lock/c9aa65e2e8>) returned error, retrying after 5.24968132s: client.PutObject: The provided token has expired.\n
unable to refresh lock: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 585.689542ms: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 376.664553ms: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 1.429332259s: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 1.792412489s: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 1.675276871s: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 2.447211796s: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 4.546210801s: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 4.983849002s: client.PutObject: The provided token has expired.\n
ave(<lock/be91e52601>) returned error, retrying after 11.629684518s: client.PutObject: The provided token has expired.\n
Save(<lock/be91e52601>) returned error, retrying after 18.994097647s: client.PutObject: The provided token has expired.\n
unable to refresh lock: client.PutObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 299.234242ms: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 1.051233419s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 1.396273685s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 2.130289026s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 2.381363954s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 4.467074266s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 4.258409627s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 7.867174041s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 17.234045382s: client.RemoveObject: The provided token has expired.\n
Fatal: unable to save snapshot: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 651.678682ms: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 406.929871ms: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 1.28079994s: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 1.352432833s: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 2.954136239s: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 3.720041086s: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 7.170308772s: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 7.105733341s: client.PutObject: The provided token has expired.\n
Save(<lock/4b248c8b36>) returned error, retrying after 7.981330938s: client.PutObject: The provided token has expired.\n
unable to refresh lock: client.PutObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 687.946071ms: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 1.015145615s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 1.361482387s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 1.948773138s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 2.227475513s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 4.303965969s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 4.570641642s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 4.506597195s: client.RemoveObject: The provided token has expired.\n
Remove(<lock/2f905ea58b>) returned error, retrying after 9.436103224s: client.RemoveObject: The provided token has expired.\n
error in cleanup handler: client.RemoveObject: The provided token has expired.", 
"stderr_lines": ["Save(<data/c5d2fb6918>) returned error, retrying after 709.461956ms: client.PutObject: We encountered an internal error. Please try again.", 
"Save(<data/cd324479a3>) returned error, retrying after 384.108656ms: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 746.156735ms: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 339.859476ms: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 1.060933735s: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 944.39985ms: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 626.969894ms: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 1.530815463s: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 621.447617ms: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 1.2520675s: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 2.362924457s: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 2.320850891s: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 1.704713704s: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 2.738744502s: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 1.504866011s: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 2.664273974s: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 2.109485187s: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 3.307287802s: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 5.564221834s: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 6.989004659s: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 6.664668763s: client.PutObject: The provided token has expired.",
"Save(<data/cd324479a3>) returned error, retrying after 7.53732321s: client.PutObject: The provided token has expired.",
"Save(<lock/a51e105692>) returned error, retrying after 5.341114202s: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 7.872190029s: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 7.754236646s: client.PutObject: The provided token has expired.", 
"Save(<lock/a51e105692>) returned error, retrying after 16.242143546s: client.PutObject: The provided token has expired.", 
"Save(<data/7fdc767012>) returned error, retrying after 10.939491392s: client.PutObject: The provided token has expired.", 
"Save(<data/cd324479a3>) returned error, retrying after 8.151585845s: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 389.807142ms: client.PutObject: The provided token has expired.", 
"unable to refresh lock: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 708.474529ms: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 462.64464ms: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 1.628069526s: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 579.68295ms: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 925.724046ms: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 1.800814626s: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 852.550639ms: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 1.677193271s: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 4.043006998s: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 5.436099694s: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 3.562825097s: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 6.552883951s: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 4.447282508s: client.PutObject: The provided token has expired.", 
"Save(<index/9fcc1fb746>) returned error, retrying after 16.266748172s: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 6.81074462s: client.PutObject: The provided token has expired.", 
"Save(<lock/c9aa65e2e8>) returned error, retrying after 5.24968132s: client.PutObject: The provided token has expired.", 
"unable to refresh lock: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 585.689542ms: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 376.664553ms: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 1.429332259s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 1.792412489s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 1.675276871s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 2.447211796s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 4.546210801s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 4.983849002s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 11.629684518s: client.PutObject: The provided token has expired.", 
"Save(<lock/be91e52601>) returned error, retrying after 18.994097647s: client.PutObject: The provided token has expired.", 
"unable to refresh lock: client.PutObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 299.234242ms: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 1.051233419s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 1.396273685s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 2.130289026s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 2.381363954s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 4.467074266s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 4.258409627s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 7.867174041s: client.RemoveObject: The provided token has expired.", 
"Remove(<lock/2f905ea58b>) returned error, retrying after 17.234045382s: client.RemoveObject: The provided token has expired.", 
"Fatal: unable to save snapshot: client.PutObject: The provided token has expired.",
 "Save(<lock/4b248c8b36>) returned error, retrying after 651.678682ms: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 406.929871ms: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 1.28079994s: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 1.352432833s: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 2.954136239s: client.PutObject: The provided token has expired.",
 "Save(<lock/4b248c8b36>) returned error, retrying after 3.720041086s: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 7.170308772s: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 7.105733341s: client.PutObject: The provided token has expired.", 
 "Save(<lock/4b248c8b36>) returned error, retrying after 7.981330938s: client.PutObject: The provided token has expired.", 
 "unable to refresh lock: client.PutObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 687.946071ms: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 1.015145615s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 1.361482387s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 1.948773138s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 2.227475513s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 4.303965969s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 4.570641642s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 4.506597195s: client.RemoveObject: The provided token has expired.", 
 "Remove(<lock/2f905ea58b>) returned error, retrying after 9.436103224s: client.RemoveObject: The provided token has expired.", 
 "error in cleanup handler: client.RemoveObject: The provided token has expired."], 
 stdout": "\\nBackup has failed", 
 "stdout_lines": ["\\nBackup has failed"]}

fd0 · January 21, 2019, 12:56pm

I don’t have any experience with time-limited AWS tokens, but guessing from the error messages it looks like the expired token prevents restic from uploading more data.

Bharathkumarraju · January 21, 2019, 1:14pm

yeah, Thanks for the reply. I mean once restic backup triggers it should sustain the thread until the backup completes right or am i wrong here?

[admin@prod-db-barman001 ~]$ date
Mon Jan 21 14:10:03 CET 2019
[admin@prod-db-barman001 ~]$ sudo -u restic aws sts assume-role --role-arn arn:aws:iam::123456789:role/restic-backup --role-session-name run-backup | jq
{
  "AssumedRoleUser": {
    "AssumedRoleId": "XXXXXXX:run-backup",
    "Arn": "arn:aws:sts::123456789:assumed-role/restic-backup/run-backup"
  },
  "Credentials": {
    "SecretAccessKey": "XXXXXXXXXXXXXXXXX",
    "SessionToken": "FQoGZXIvYXdzEFcaDLHbP5CH6e7TM6blVyLuATfjg9CmLB3DCB7zXAFQ1/vviTiTg8kl/CQzNc+zgp4DAnbjll+RXsju/H0wbU4q2quYDq02X/pZ4UcoQ3dCKq9TxCAL",
    "Expiration": "2019-01-21T14:10:08Z",
    "AccessKeyId": "XXXXXXXXX"
  }
}
[admin@prod-db-barman001 ~]$ date

cdhowie · January 21, 2019, 4:08pm

Restic uploads multiple objects to S3 to perform the backup. AWS doesn’t maintain any kind of a “transaction” or anything between these uploads; it sees each as a discrete request. There’s nothing “sustaining any thread” (not sure what you even mean by that).

Presumably, once the access key expires, S3 no longer accepts the put-object requests and the backup is effectively interrupted.

fd0 · January 22, 2019, 10:32am

I’m sorry, but I don’t understand what you’re describing. I admit that I don’t have much experience with s3 using anything besides regular, unlimited credentials, so I may be wrong

Each upload to s3 is its own HTTP request which includes the token. As soon as the token is not valid any more, the service rejects the request. There’s no “renewal” of any kind that I’m aware of. This would be kind of counter-productive, as it would give attackers a way to extend a token’s lifetime indefinitely, which I think is not what the designers have intended…

Bharathkumarraju · January 30, 2019, 9:58am

Thank you for your detailed explanation.

Bharathkumarraju · January 30, 2019, 10:11am

So my issue is like the restic forget --prune taking a lot of time more than 16 hours were my restic backup trying to trigger on same repo, that is the issue is there any way we can speedup the restic forget --prune?
below is my command
RESTIC_REPOSITORY=s3:s3.amazonaws.com/${bucket} /usr/local/bin/restic forget --prune --keep-last 2 --keep-daily 14 2>&1

and the issue i am getting is as below.

FAILED! => {"changed": true, "cmd": "/opt/bin/perform-backup.sh", "delta": "0:00:05.417970", "end": "2019-01-29 23:00:23.127268",
"msg": "non-zero return code", "rc": 1, "start": "2019-01-29 23:00:17.709298",
"stderr": "Fatal: unable to create lock in backend: repository is already locked exclusively by PID 11183 on ip-172-31-254-11.eu-central-1.compute.internal by centos (UID 1000, GID 1000)

lock was created at 2019-01-29 05:03:57 (16h56m22.497694257s ago)
storage ID 8e44efff
Fatal: create key in repository at s3:s3.amazonaws.com/rbmh-mit-backup-ffm-prod-ss-db-barman failed: repository master key and config already initialized
Fatal: unable to create lock in backend: repository is already locked exclusively by PID 11183 on ip-172-31-254-11.eu-central-1.compute.internal by centos (UID 1000, GID 1000)

lock was created at 2019-01-29 05:03:57 (16h56m25.131742477s ago)
storage ID 8e44efff", 
"stderr_lines":
["Fatal: unable to create lock in backend: repository is already locked exclusively by PID 11183 on ip-172-31-254-11.eu-central-1.compute.internal by centos (UID 1000, GID 1000)",

"lock was created at 2019-01-29 05:03:57 (16h56m22.497694257s ago)",
"storage ID 8e44efff",
"Fatal: create key in repository at s3:s3.amazonaws.com/rbmh-mit-backup-ffm-prod-ss-db-barman failed: repository master key and config already initialized", "",
"Fatal: unable to create lock in backend: repository is already locked exclusively by PID 11183 on ip-172-31-254-11.eu-central-1.compute.internal by centos (UID 1000, GID 1000)",

"lock was created at 2019-01-29 05:03:57 (16h56m25.131742477s ago)", 
"storage ID 8e44efff"],
"stdout": "Backup has failed",
"stdout_lines": ["Backup has failed"]}

@fd0 @cdhowie any clue would be greatly appreciated thank you!!!