When to use restic check, restic prune and restic rebuild-index?

Getting Help
1

Hi Team,

We are using restic extensively to backup all our on-prem data to s3(~400GB daily).

We are using below commands to do that

 restic_status=$(sudo -u restic bash -c 'source /home/restic/.restic-env ; /usr/bin/restic snapshots')
 if [[ $(echo $?) == 0 ]]
 then
 sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic backup /data/backup.tgz 
 else
 sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic init"
 sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic backup /data/backup.tgz  
 fi

But sometimes i am getting some errors like below, when i see some of your forums you guys recommending to use restic check and restic rebuild-index and restic prune , So in my above script where can i use these three commands restic check and restic rebuild-index and restic prune

Do I need to run daily while backing up?

The two types of errors which we see as below.

  1. “msg”: “non-zero return code”, “rc”: 1, “start”: “2018-08-17 23:00:17.172642”,
    “stderr”: “load <index/116e39cae2>: invalid data returned, ignoring”,
    “stderr_lines”: [“load <index/116e39cae2>: invalid data returned, ignoring”],

    https://github.com/restic/restic/issues/1229

Create: open /home/restic/.cache/restic/b92fb5e7b434b2c583af49d55b07b043326d2de0ee1f5a710e400af539a0cae9/snapshots/79/794e708a8b180ead8e107c1aecd86547d3e466f67af7b7c11e318a7f07e38c60: file exists\nLoad(<snapshot/794e708a8b>, 0, 0) returned error, retrying after 13.961376776s

2

is there anyway we can run restic check all the time when we run backup or restic rebuild-index while doing backup to avoid above errors…is there any best practise to use while doing restic back up please let me know?

3

Hi Team,

Any best practices to do restic backup?

Thanks in advance for your reply.

4

Those 3 commands are save to use at any time. How often they should be run depends on your needs and your environment. For example, on most of my backups I’m running restic check and restic prune (with restic forget) weekly. restic rebuild-index is run only when needed (which is rarely the case).

Usually there should be no need to run restic prune and restic rebuild-index daily. Do you see these errors on a regular basis? Is your internet connection kinda unreliable or are the issues caused by S3?

1 Like
5

@764287
Thank you so much for your reply. I think I have to include restic check command before restic backup in oder to make sure all indexes are good.

I am not seeing these index ignore errors regularly. When it runs again it is working fine somehow I don’t know.

May be ideal case is let’s do restic check before restic backup daily. Is this good?

6

While restic check is never a bad idea it also needs a lot of time to complete. If you make daily backups only, you can probably afford to run restic check before every backup run. But judging from the issue you linked to, it should be sufficient to run restic rebuild-index which is a lot faster.

As this seems to be a cosmetic issue and no data is lost (maybe @fd0 can confirm this), I personally wouldn’t run restic rebuild-index that often.

1 Like
7

In the backup process, I’d personally only run restic backup. When restic backup fails, the error needs to come to your attention for a manual intervention.

I don’t think adding other commands like rebuild-index to the backup process is a good idea. After all, you would like to be aware of any problems, especially if they are repetitive - and fix the root cause, as opposed to hide it with an automatic rebuild-index.

1 Like
8

@764287 Thank you for your reply.
IS there anyway i can script the restic backup output, like if it gives error i can do rebuild-index in the script itself or it definitely needs the manual attention?

@lukastribus, yes i am completely with you. At least i can use restic check right with restic backup?

Is it a good idea to put restic check each time before running restic backup… i am bit curious that you guys ever implemented this in your automation scripts or something like that to do restic check all the time?

9

I don’t think that using rebuild-index is needed every time you backup. I run check in every backup but after the backup. I don’t know, for me it makes more sense to check after you did a backup because it’ll check everything, including your latest backup. I don’t know if I’m wrong.

I use this script and I haven’t had any troubles.

1 Like
10

I run restic via a wrapper script which counts how many times the backup has been run. It does a quick verify every backup, a check --read-data every 20 backups (by default) and a forget/prune every 10 and 20. I don’t run rebuilds automatically at all.

If restic errors or the check/prunes fail, I get an email.

1 Like
11

How often you want to run restic check or restic check --read-data is different for each setup. If the destination storage is local and download is therefor free, why not.

If it goes through a slow WAN connection, you may have to evaluate, same as if your storage service charges API calls and download costs or if it simply takes to long.

My point is this: restic backup should be robust enough to either run successful, or fail hard requiring manual intervention.

I agree that restic check is useful, however it is not a replacement for periodic restore tests.

1 Like
12

Does restic check download all the data (which is expensive with S3), or does that only happen if you use the --read-data parameter?

1 Like
13

No, restic check does not download all the data. I assume it needs to download some index data and checks the presence of referenced files, afaik.

1 Like
14

So guys, Thanks a lot for your inputs. After reading restic docs your inputs, I have decided to use like this. if the script fails i will maually login and check what i should do, do i need to rebuild index or not?

restic_status=$(sudo -u restic bash -c 'source /home/restic/.restic-env ; /usr/bin/restic snapshots')
if [[ $(echo $?) == 0 ]]
then
  sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic backup ${backup_dir}  --tag 
${today}"
if [[ $(echo $?) == 0 ]]
 then
     echo ""
    echo "Backup is successfull"
 else
    echo "Backup has failed"
    exit 0
fi
 sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic check"
if [[ $(echo $?) == 0 ]]
then
    echo ""
    echo "Repository check is successfull"
else
    echo "Repository check has failed"
    exit 0
fi

else
sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic init"
sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic backup ${backup_dir}  --tag 
 ${today}"
 if [[ $(echo $?) == 0 ]]
 then
    echo ""
    echo "Backup is successfull"
else
    echo "Backup has failed"
    exit 0
fi
sudo -u restic bash -c "source /home/restic/.restic-env ; /usr/bin/restic check"
if [[ $(echo $?) == 0 ]]
then
    echo ""
    echo "Repository check is successfull"
 else
    echo "Repository check has failed"
    exit 0
fi
 fi
1 Like
15

So for it is working fine. Let us know if anything wrong in my script respective to the restic?