Anomalous restic runtime cmd errors

jikenjoe · November 18, 2022, 7:21pm

🍒 ~/Scripts/Backups🔒 took 7s 
🐡ish Terminal ➜  ./list-snapshots.sh 
repository 7e639bb7 opened (repository version 2) successfully, password is correct
ID        Time                 Host         Tags        Paths
-------------------------------------------------------------------------------
9a74be62  2022-11-17 10:16:10  workstation              /etc
                                                        /home/jikenjoe
                                                        /root
                                                        /var/lib/libvirt/images
                                                        /var/log

d5c5ce6d  2022-11-17 23:42:05  workstation              /etc
                                                        /home/jikenjoe
                                                        /root
                                                        /var/lib/libvirt/images
                                                        /var/log

bc677e4b  2022-11-18 13:17:22  workstation              /etc
                                                        /home/jikenjoe
                                                        /root
                                                        /var/lib/libvirt/images
                                                        /var/log
-------------------------------------------------------------------------------
3 snapshots

🍒 ~/Scripts/Backups🔒 took 18s 
🐡ish Terminal ➜  ./prune-s3-repo.sh 
unknown command "./pw" for "restic"

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ✗  cat list-snapshots.sh 
#!/bin/bash
source ./params
PASS="./pw"

restic -r $RESTIC_REPOSITORY -p $PASS snapshots

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ➜  cat prune-s3-repo.sh 
#!/bin/bash
source ./params
PASS="./pw"

restic -r $RESTIC_RESPOSITORY -p $PASS snapshots

### "unknown flag --dry-run" and "--repack-uncompressed"
### also tried with "--repack-uncompressed=true"
#restic -r $RESTIC_RESPOSITORY prune --dry-run --repack-uncompressed
#restic -r $RESTIC_RESPOSITORY prune --repack-uncompressed=true --repack-smaller=true
#restic -r $RESTIC_RESPOSITORY -p $PASS snapshots

### "unknown flag --repack-uncompressed"
#restic -r $RESTIC_RESPOSITORY prune --repack-uncompressed 

### works with forget not prune ### --repack-smaller unknown
#restic -r $RESTIC_REPOSITORY -p $PASS forget -l 4 --dry-run --prune --repack-small=true --repack-uncompressed

### Final CMD ###
#restic -r $RESTIC_REPOSITORY -p $PASS forget -l 4 --prune --repack-small=true --repack-uncompressed

Hey Restic Forum,

It seems that for some reason when I run the same command in a different file, I get and error message of "unknown command “./pw” for “restic”

I also have the error of “unknown flag --dry-run” and “–repack-uncompressed” also tried with “–repack-uncompressed=true” when running prune ← I read up on this a bit and there seemed to be a consensus that the “–dry-run” flag wasn’t implemented in the 0.11.0 release but instead is functional in the forget command as of version 0.13.0.

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ➜  restic version
restic 0.14.0 compiled with go1.19 on linux/amd64

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ➜  la list-snapshots.sh prune-s3-repo.sh
.rwxr-xr-x root root 105 B Fri Nov 18 13:35:20 2022  list-snapshots.sh
.rwxr-xr-x root root 737 B Fri Nov 18 14:04:35 2022  prune-s3-repo.sh

As for the other flags I am not certain.

Any ideas why this is occuring?
Thanks!

MichaelEischer · November 19, 2022, 4:30pm

Without knowing more about the script, my guess would be that there’s something wrong related to quoting the command parameters. But without the script that’s pure guesswork.

Make sure to put those options after prune. --repack-uncompressed was added in restic 0.14.0.

jikenjoe · November 20, 2022, 4:50am

Hi Michael,

I did post the script, I used cat on the files and posted the output to my OP.

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ✗  cat list-snapshots.sh 
#!/bin/bash
source ./params
PASS="./pw"

restic -r $RESTIC_REPOSITORY -p $PASS snapshots

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ➜  cat prune-s3-repo.sh 
#!/bin/bash
source ./params
PASS="./pw"

restic -r $RESTIC_RESPOSITORY -p $PASS snapshots

### "unknown flag --dry-run" and "--repack-uncompressed"
### also tried with "--repack-uncompressed=true"
#restic -r $RESTIC_RESPOSITORY prune --dry-run --repack-uncompressed
#restic -r $RESTIC_RESPOSITORY prune --repack-uncompressed=true --repack-smaller=true
#restic -r $RESTIC_RESPOSITORY -p $PASS snapshots

### "unknown flag --repack-uncompressed"
#restic -r $RESTIC_RESPOSITORY prune --repack-uncompressed 

### works with forget not prune ### --repack-smaller unknown
#restic -r $RESTIC_REPOSITORY -p $PASS forget -l 4 --dry-run --prune --repack-small=true --repack-uncompressed

### Final CMD ###
#restic -r $RESTIC_REPOSITORY -p $PASS forget -l 4 --prune --repack-small=true --repack-uncompressed

Without knowing more about the script, my guess would be that there’s something wrong related to quoting the command parameters. But without the script that’s pure guesswork.

If you mean what’s inside the “./pw” file. It’s just a plaintext of my password. I have all my scripts setup as such and so far I’ve been able to backup and also list-snapshost with that syntax.

Make sure to put those options after prune . --repack-uncompressed was added in restic 0.14.0.

Also as you can see from the above, I tested the script with different flags and syntax, each on it’s own run, while commenting out the command lines I wasn’t testing at that given execution.

jikenjoe · November 20, 2022, 6:22am

EDIT: It seems that for whatever reason this issue is specific to the script file i’ve created because I recreated the script with list-snapshots.sh by:

1) cp list-snapshots.sh test-prune.sh
2) vim into it and delete unnecessary commands
3) add line "restic -r $RESTIC_REPOSITORY $PASS prune --dry-run --repack-uncompressed --repack-smaller"

When I run the script I get the output:

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ➜  ./test-prune.sh 
repository 7e639bb7 opened (repository version 2) successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 4 snapshots
[0:04] 100.00%  4 / 4 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:01] 100.00%  10862 / 10862 packs processed

to repack:        447997 blobs / 167.655 GiB
this removes:      10158 blobs / 1.665 GiB
to delete:         11772 blobs / 9.770 GiB
total prune:       21930 blobs / 11.435 GiB
remaining:        506382 blobs / 214.972 GiB
unused size after prune: 2.248 GiB (1.05% of remaining size)

The script that doesn’t work is:

🍒 ~/Scripts/Backups🔒 took 4s 
🐡ish Terminal ✗  cat prune-s3-repo.sh 
#!/bin/bash
source ./params
PASS="./pw"

restic -r $RESTIC_RESPOSITORY $PASS prune --dry-run --repack-uncompressed --repack-small

Error output:

🍒 ~/Scripts/Backups🔒 took 3s 
🐡ish Terminal ➜  ./prune-s3-repo.sh
Fatal: unable to open repository at ./pw: ReadDir: open pw/keys: not a directory

The test script that works:

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ✗  cat test-prune.sh
#!/bin/bash
source ./params
PASS="./pw"

restic -r $RESTIC_REPOSITORY -p $PASS prune --dry-run --repack-uncompressed --repack-small

The file chowns and chmods are:

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ✗  la prune-s3-repo.sh test-prune.sh 
.rwxr-xr-x root root 146 B Sun Nov 20 01:22:44 2022  prune-s3-repo.sh
.rwxr-xr-x root root 148 B Sun Nov 20 01:20:30 2022  test-prune.sh

alexweiss · November 20, 2022, 6:33am

In your not-working script the “./pw” is taken as the repository (the -p is missing).
But as you pointed out, this is not a restic issue but a script issue you are dealing with. Maybe try one of the various wrapper scripts available for restic?

(BTW: This is a good example why a config file support does help to improve the application)

jikenjoe · November 20, 2022, 10:09am

Oh nice catch. Thanks!
I did correct that error and it does resort back to the original error of:

🍒 ~/Scripts/Backups🔒 took 11s 
🐡ish Terminal ➜  ./prune-s3-repo.sh 
unknown command "./pw" for "restic"

I also edit my params file and add:
export RESTIC_PASSWORD='<password>'
but that seems to out the same error but with a slight difference:

🍒 ~/Scripts/Backups🔒 took 11s 
🐡ish Terminal ➜  ./prune-s3-repo.sh 
unknown command "<password>" for "restic"

Also interesting, I didn’t realize restic had wrapper scripts for a streamlined process. I do understand your reasoning but honestly I would like to create the scripts myself just for the experience. Also I like it!!

And there’s another error:

🍒 ~/Scripts/Backups🔒 
🐡ish Terminal ➜  ./prune-s3.sh 
Resolving password failed: Fatal: <password> does not exist

Could this be because I used ‘single quotes’ to encapsulate the password?

What can the possible reasons be for these sort of errors?

MichaelEischer · November 20, 2022, 10:30am

I’ve missed the scripts inbetween all the other output and all these emojis .

There’s another typo here. It should be repository, not reSpository.

When using variables in a command line please add quotes unless you know that these are not necessary. And I’d recommend to add a set -eu line at the top of the script, the e will cause the script to abort if there’s a non-zero exit code (which is currently silently swallowed by your wrapper script). And the u will cause bash to err out if a variable is undefined. For debugging set -x is also quite helpful.

jikenjoe · November 20, 2022, 10:50am

Ohh damn, silly mistakes and yea haha the emojis can be much when I add it all as output

Also thanks I didn’t know about the “set -eux”.