During 'restic prune', check to see if an index-rebuild has already been done

When you execute a restic prune, judging by the output (e.g. building new index for repo), it appears to first do the same thing that restic rebuild-index does.

If so, it would be helpful to (many users?) if it first checked if the index has already been recently successfully rebuilt, with no significant operations done since, with the exception of related things like the following:

  • Any number of additional restic rebuild-index for whatever dumb reason
  • restic recover
  • Another restic prune
  • restic forget

Yes this would probably fall under one or more of the following categories:

  • “Protecting The User From Themselves”
  • “Stupid User Tricks”
  • “Users Who Think They Are Smarter Than Restic Devs”

But since rebuilding the index can be among the most incredibly expensive things done (in terms of time) aside from an actual long backup job, this could relieve a considerable amount of pain involved in any of those three categories - which, lets face it, are all pretty common and is only going to get worse as our species devolves into what the documentary “Idiocracy” predicts.

And yes, I often find myself in all three of those categories at once. Since my backups often get interrupted (regardless of fancy gymnastics to select smaller backup sets), and since I have a natural distrust of humans no matter how well-designed and written software seems to be, my backup script was periodically first doing a rebuild index, then restore, then forget, then prune. (Yes I now understand that was unnecessary and not even accomplishing what I thought it was.) The point being, the first and last step in that chain were apparently doing, in subset, the same thing - rebuilding the local index from scratch.

Sure, we can’t always invest stretched resources just to help users avoid pain when they do stupid things. But with pain that severe, and IF it (hypothetically) involved only checking a few flags and with no interface change, then why not?

After all, one of the biggest hurdles to adopting backup software - or any CLI-based FLOSS software exceeding some level of complexity or heavy-lifting, especially when storage and/or transfer costs could be quite high - is learning what the software lets you do without complaint but is either harmful, expensive, and/or completely unnecessary. (Trust me on that. I’ve evaluated at least a dozen FLOSS and commercial backup products. Often involving considerable pain and/or expense. :pensive:)

You don’t need to run rebuild-index after an interrupted backup. Also, I can advise you to try this branch, which is a bit more strict in terms of checking integrity while pruning.

\\* unnecessary opinion start *

And no, I don’t think the application should cover that use case IMHO. If you see a bug, please report. But trying that hard and asking for more sounds… weird, no?

\\* unnecessary opinion end *

Once https://github.com/restic/restic/pull/2718 is completed, prune will no longer have to rebuild the index, it will just update the existing one (along with further major prune speed-ups).

Actually it should never be necessary to rebuild the repository index, unless there has been some data corruption/loss in the repository itself. A failing backup client / network connection should not cause any problems.

2 Likes

Possibly also in terms of money! Many object storage services charge for each download request, independent of how much data is actually transferred.

1 Like

Actually, after splitting that PR in smaller parts, #2718 removes the first rebuilding of the index and #2842 removes the second rebuilding of the index.
Both PRs can IMO merged independently.

1 Like

That’s fair to have and express an opinion. As I did.

Sound advice.

I don’t understand this meaning, but that’s OK.

Also not understanding this meaning. I’m not asking for anything let alone “more” - only trying to “give back” improvement ideas/feedback that I already noted I don’t need - so I’m not sure if you’re addressing some other subtlety, or some meaning got lost in dialect or language. Either way it doesn’t feel constructive on the surface, so maybe there’s no need to clarify? :man_shrugging:

1 Like