First backup of huge server storage

I’ve just finished a backup that took several hours to complete, due to the amount of data to store, into AWS S3.

I got several warnings like the bellow:

Save(<data/53a6f93ae8>) returned error, retrying after 673.750131ms: client.PutObject: Put https://mybucket.s3.dualstack.us-west-2.amazonaws.com/data/53/<big_string_here>: write tcp 111.111.111.111:18038->222.222.222.222:443: use of closed network connection

Can these be ignored and can I trust that restic tries to repeat any failed operations?

Also, the output looks like this:

created new cache in /root/.cache/restic

Files: 7656223 new, 0 changed, 0 unmodified
Dirs: 1 new, 0 changed, 0 unmodified
Added to the repo: 238.573 GiB

processed 7656223 files, 432.862 GiB in 10:59:51
snapshot 94bce23d saved

Did it add 238.573 GiB, or did it add 432.862 GiB, to the S3 bucket?

I mounted and looked through some directories, and everything seemed to be fine, but of course I can’t manually check 7.6M files :slight_smile:
But I would like to understand the output above.

Thank you very much!

I almost thought it took six months for you to complete that initial backup, since it’s been about six months since you last posted (before this time) :wink:

Yes, restic will tell you if it was unable to fullfill its duty, and will retry failed operations. Sometimes too much IMO, but in your case it has simply retried the requests until it succeeded (otherwise it wouldn’t have told you that it created a snapshot in the end).

It added 238.573 GiB like it says - the other number 432.862 GiB is what it processed. Due to e.g. deduplication the end result is smaller than the size of the data set you asked it to backup/process. I’m not sure if excluded files are counted in that last number, but it doesn’t matter in this case.

I too think it looks good, you should have a proper first snapshot there.

2 Likes

Thanks very much @rawtaz

That makes total sense, and I noticed I had added an entry to the “restic.exclude” file that I had forgotten it there:

**/somedir/somehugedir

That was the directory that was missing to make up for the rest of the ~200GB missing.

However, even though I removed that entry from that exclude file and re-ran Restic Backup, it appears that the latest snapshot still doesn’t have that folder, and the output says it only added ~100Mb (from other changed files).

If I am removing an exclusion from the “restic.exclude” file, shouldn’t the next snapshot backup those missing files?

I double-checked the rest of the exclusions, and nothing should prevent that folder from backing up.
Also, it’s definitely included under “/opt” in the “restic.include” file.

Thank you very much!

1 Like

It should indeed. If it doesn’t I think you need to show the complete command and the include and exclude files, otherwise we can only guess. Another thing you can do is simply try to restore one or more of that directory’s files, to verify that they were indeed not backed up (which is what you expect).

I actually expect the opposite. I’m removing an entry from the exclusions, so that directory SHOULD be included in the next snapshot. But, that directory isn’t in the latest snapshot.

Command:

. /restic/restic.env; restic backup --tag manual --files-from=/restic/restic.include --exclude-file=/restic/restic.exclude > /restic/restic.log 2>&1

restic.include:
/etc
/opt
/root
/usr/local
/var

restic.exclude:
*~
**/node_modules/
**/vendor/
**/lost+found
(then, a number of fixed absolute paths, without any star *)

Thanks for correcting me, I’m an idiot, of course that’s what you expect. So you did use restic restore or restic ls or restic mount to verify that the directory/files are really not backed up? Which path is it you removed from the exclude file and that you are missing when you run subsequent backups? Can you show the complete output of the run?

You’re definitely not an idiot - you’re trying to help me here, and I appreciate that a lot.

Do you want to know who’s an idiot? Me…
No idea what happened, or I was probably just looking at an older snapshot,
but I’ve just double-checked now (while I was building a reply to you), and the un-excluded folder is there… [facepalm]

Thanks very much! Everything is working well.
Restic absolutely rocks… it just works well!

Haha, so there’s two of us :stuck_out_tongue:

Does this mean that the original amount of data that was uploaded, around 200 GB, contained all the data (including the directory you removed from the excludes later on)? Or did restic add some hundred GBs of data after you removed the exclude?

The first snapshot definitely doesn’t have that directory. It must have been the 2nd snapshot (which I don’t have the output anymore) that added the rest. The last snapshot is what added only 100Mb.

I was trying to run “restic stats --mode raw-data”, but this was taking forever, and consuming a lot of CPU.

Is there a way to see how much Restic calculated for each snapshot, without having to re-calculate each time? Does Restic save that info anywhere?

Would be nice to see that info in the output of “restic snapshots”.

Snapshot 1 – 200GB
Snapshot 2 – 400GB
Snapshot 3 – 405GB

Something like that, in the output table, in a new column “Size”.

Thanks very much!

1 Like