Risks/consequences of prune when backup in progress or vice versa

[rawtaz]

Sorry for the ignorant question, but why would you let a script that prunes start before you finished your initial backup? That makes no sense to me.

[Torslanda]

Simply because there’s no way to know when the backup script will finish. How long does it take to back up 3TB? The time taken may vary depending on whether it’s the initial full (3 TB) backup, a subsequent snapshot where only a couple of files have changed, or a major subsequent backup where a multi-GB 4K video file has been added to the backup selection. If I need to be absolutely sure that the backup has finished before running the prune then I would basically have to do it all manually and babysit it until it finishes, and then run the prune manually as well sometime later. It all defeats the object of automation and scripting. That’s why I want the prune script to run on its schedule, but exit gracefully when it realises there’s an active lock caused by the running backup.

[764287]

@Torslanda

However, it looks like it has removed my active lock whilst the backup was running, based on the output of my prune script (see below) and my script is now complaining that there are packs not referenced in any index and that my repository contains errors. I therefore fear that it has removed the lock, begun to prune whilst my backup was in progress, and corrupted my backup, meaning that I’ll need to start it over again. Can anyone tell me what’s happening here and whether my interpretation is correct?

I’d assume that your internet connection got interrupted at some point or something else went wrong which interrupted the backup after restic uploaded some packs but before it could update the index. Running restic rebuild-index will probably clear the warnings.

@Torslanda

Simply because there’s no way to know when the backup script will finish. How long does it take to back up 3TB? The time taken may vary depending on whether it’s the initial full (3 TB) backup, a subsequent snapshot where only a couple of files have changed, or a major subsequent backup where a multi-GB 4K video file has been added to the backup selection. If I need to be absolutely sure that the backup has finished before running the prune then I would basically have to do it all manually and babysit it until it finishes, and then run the prune manually as well sometime later. It all defeats the object of automation and scripting. That’s why I want the prune script to run on its schedule, but exit gracefully when it realises there’s an active lock caused by the running backup.

You are using systemd ro run your backups which provides some neat ways to define dependencies. Using Conflicts should disable restic prune as long as restic backup is running.

PS: Looks like ericw already wrote some scripts to deal with this.

[Torslanda]

@764287

I’d assume that your internet connection got interrupted at some point

That’s entirely possible. It has been known for my ISP to perform maintenance overnight and for it to be back up again by morning so I don’t usually know whether it’s been off or not.

@764287

Running restic rebuild-index will probably clear the warnings.

Do I need to stop the (still running) backup before doing that? And then afterwards if I restart the backup will it pick up where it left off?

If an interrupted connection can cause these issues then I’m wondering how I can script around this to make my automated system reliable and failsafe under most circumstances.

@764287

You are using systemd ro run your backups which provides some neat ways to define dependencies. Using Conflicts should disable restic prune as long as restic backup is running.

Thanks - I hadn’t considered conflicts; I’ll definitely look at that. I have a basic understanding of systemd but I can’t take credit for the systemd scripts - I followed Rolf’s tutorial here: Backup auf Cloud-Speicher mit »restic« · GitHub which seems to be exactly what I wanted. I’ll check out Eric’s as well, thanks.

[764287]

@Torslanda

Do I need to stop the (still running) backup before doing that? And then afterwards if I restart the backup will it pick up where it left off?

If a backup gets interrupted and is restarted, restic will just continue to upload data. restic might need some minutes to recognize which parts have already been uploaded but it will roughly continue were it stopped.

That said, I’d personally let the backup finish and restic rebuild-index; restic prune; restic check afterwards.

@Torslanda

If an interrupted connection can cause these issues then I’m wondering how I can script around this to make my automated system reliable and failsafe under most circumstances.

You could simply schedule more backups. Having more snapshots usually doen’t hurt as the data is deduplicated. systemd should prevent a 2nd backup run if the 1st hasn’t finished yet.