Damanged index, while dumping from stdin with fresh repository

Getting Help
1

Hi there,

i am trying to get this to run for several weeks now for now avail.

My setup is:

restic 0.11.0 compiled with go1.15.3 on linux/amd64

running through docker-compose in docker. I have only the cache directory mounted as an anonymous volume, set with RESTIC_CACHE_DIR.

I can connect to my own s3 (with minio) or tried to connect to wasabi: same results. My backup script looks like this:

#!/usr/bin/env bash

set -eo pipefail

database=$1

docker-compose exec -T db bash -c "mysqldump --user=root --password="\${MYSQL_ROOT_PASSWORD}" ${database} " \
    | gzip \
    | docker-compose exec -T restic sh -c "restic backup --tag mysql --tag cepheus --stdin --stdin-filename ${database}.sql.gz"

when I backup I have a lot of the small databases in progression, I tried sequential and parralel, but I always end with these errors:

error: tree f7195bf96d965a81445bde917404f4905aafebc8a3070ea6f3863abb1753a42f is not known; the repository could be damaged, run rebuild-index to try to repair it

This is a part of the log of the very first run. With a fresh bucket and a new init of a repo.

processed 1 files, 117.045 KiB in 0:00
snapshot 85e5600b saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.

mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 22.982 KiB

processed 1 files, 22.695 KiB in 0:00
snapshot 13e779b0 saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.
error: tree 02afdfbc185f730aa246cfeeead43f20b23ca5c7c2fdb6e4f121c614e010ab13 is not known; the repository could be damaged, run `rebuild-index` to try to repair it

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 20.908 KiB

processed 1 files, 20.613 KiB in 0:00
snapshot 7e22cbfe saved
Warning: failed to read all source data during backup


mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 163.999 KiB

processed 1 files, 163.710 KiB in 0:00
snapshot 83579124 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.814 KiB

processed 1 files, 18.519 KiB in 0:00
snapshot 0f6594d5 saved


Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 21.447 KiB

processed 1 files, 21.161 KiB in 0:00
snapshot c140610d saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.761 KiB

processed 1 files, 18.472 KiB in 0:00
snapshot 4b9b3abc saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.


mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 22.274 KiB

processed 1 files, 21.979 KiB in 0:00
snapshot 906cc895 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.781 KiB

processed 1 files, 18.480 KiB in 0:00
snapshot 0f2b53b1 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 92.819 KiB

processed 1 files, 92.526 KiB in 0:00
snapshot a7863dd5 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.151 MiB

processed 1 files, 1.151 MiB in 0:01
snapshot 65b26d6e saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 4.084 MiB

processed 1 files, 4.084 MiB in 0:01
snapshot 7b70144d saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 20.536 KiB

processed 1 files, 20.240 KiB in 0:00
snapshot 9c9c9e84 saved




Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.926 KiB

processed 1 files, 18.635 KiB in 0:00
snapshot 9bad8608 saved
dumping mysql..
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 20.785 KiB

processed 1 files, 20.490 KiB in 0:00
snapshot 0762e8b9 saved
dumping sprotten_stories..

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 21.562 KiB

processed 1 files, 21.268 KiB in 0:00
snapshot bcc6e3e1 saved
error: tree f7195bf96d965a81445bde917404f4905aafebc8a3070ea6f3863abb1753a42f is not known; the repository could be damaged, run `rebuild-index` to try to repair it

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 55.927 KiB

processed 1 files, 55.631 KiB in 0:00
snapshot 15e9b143 saved
Warning: failed to read all source data during backup
dumping sprotten_stories_dev..
dumping sprotten_stories_staging..
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.059 MiB

processed 1 files, 1.058 MiB in 0:00
snapshot 104657cc saved
dumping teamcity..
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 185.939 KiB

processed 1 files, 185.643 KiB in 0:00
snapshot 12556ec0 saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.
dumping webforge_cms..
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.030 KiB

processed 1 files, 750 B in 0:00
snapshot 4dbdf142 saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.036 KiB

processed 1 files, 752 B in 0:00
snapshot 8f99edf2 saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.905 KiB

processed 1 files, 1.614 KiB in 0:00
snapshot 86238a34 saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 3.112 MiB

processed 1 files, 3.112 MiB in 0:01
snapshot 20ce359b saved

```
processed 1 files, 117.045 KiB in 0:00
snapshot 85e5600b saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.

mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 22.982 KiB

processed 1 files, 22.695 KiB in 0:00
snapshot 13e779b0 saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.
error: tree 02afdfbc185f730aa246cfeeead43f20b23ca5c7c2fdb6e4f121c614e010ab13 is not known; the repository could be damaged, run `rebuild-index` to try to repair it

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 20.908 KiB

processed 1 files, 20.613 KiB in 0:00
snapshot 7e22cbfe saved
Warning: failed to read all source data during backup


mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 163.999 KiB

processed 1 files, 163.710 KiB in 0:00
snapshot 83579124 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.814 KiB

processed 1 files, 18.519 KiB in 0:00
snapshot 0f6594d5 saved


Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 21.447 KiB

processed 1 files, 21.161 KiB in 0:00
snapshot c140610d saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.761 KiB

processed 1 files, 18.472 KiB in 0:00
snapshot 4b9b3abc saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.


mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 22.274 KiB

processed 1 files, 21.979 KiB in 0:00
snapshot 906cc895 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.781 KiB

processed 1 files, 18.480 KiB in 0:00
snapshot 0f2b53b1 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 92.819 KiB

processed 1 files, 92.526 KiB in 0:00
snapshot a7863dd5 saved

mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.151 MiB

processed 1 files, 1.151 MiB in 0:01
snapshot 65b26d6e saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 4.084 MiB

processed 1 files, 4.084 MiB in 0:01
snapshot 7b70144d saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 20.536 KiB

processed 1 files, 20.240 KiB in 0:00
snapshot 9c9c9e84 saved




Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 18.926 KiB

processed 1 files, 18.635 KiB in 0:00
snapshot 9bad8608 saved
dumping mysql..
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 20.785 KiB

processed 1 files, 20.490 KiB in 0:00
snapshot 0762e8b9 saved
dumping sprotten_stories..

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 21.562 KiB

processed 1 files, 21.268 KiB in 0:00
snapshot bcc6e3e1 saved
error: tree f7195bf96d965a81445bde917404f4905aafebc8a3070ea6f3863abb1753a42f is not known; the repository could be damaged, run `rebuild-index` to try to repair it

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 55.927 KiB

processed 1 files, 55.631 KiB in 0:00
snapshot 15e9b143 saved
Warning: failed to read all source data during backup
dumping sprotten_stories_dev..
dumping sprotten_stories_staging..
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.059 MiB

processed 1 files, 1.058 MiB in 0:00
snapshot 104657cc saved
dumping teamcity..
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 185.939 KiB

processed 1 files, 185.643 KiB in 0:00
snapshot 12556ec0 saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.
dumping webforge_cms..
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.030 KiB

processed 1 files, 750 B in 0:00
snapshot 4dbdf142 saved
mysqldump: [Warning] Using a password on the command line interface can be insecure.

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.036 KiB

processed 1 files, 752 B in 0:00
snapshot 8f99edf2 saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 1.905 KiB

processed 1 files, 1.614 KiB in 0:00
snapshot 86238a34 saved

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 3.112 MiB

processed 1 files, 3.112 MiB in 0:01
snapshot 20ce359b saved

Any idea? Read the other topics for that, none that applied to my case. I can rule out the s3 cause it happens on wasabi and on my own minio

2

Side note, you want to pass the --rsyncable option to gzip. Without this, there will be very little deduplication of the gzipped files, possibly none at all.

1 Like
3

Does the backup also fail if you store the database dump in a temporary file and pass that to restic? Is the cache directory shared between multiple restic instance? Is the anonymous volume the same for all restic instances?
Does restic check report any errors?
Does the error disappear if you wait a few seconds between restic invocations?

As your backup approach seems to involve creation lots of snapshots a word of warning: restic currently handles several hundreds of snapshots well, but will slow down noticeably for thousands of snapshots.

4

I changed, to dump al dbs into one directory and snapshot that directory: That is stable.

But now, when I restore one of the dumps, it’ll will restore all the dumps… That is … inconvenient?

Cache directory is only used on that node, one instance.

One run would be about 100 Snapshots.

Couldn’t check the others now, since it was a production system and is now … productiv :smiley:

5

The restore command has an --include <file name/pattern> option which allows selecting which files to restore.