Creating effective append only backup

[cdhowie]

@quinn

I don’t like the “foot-guns” that are provided by “forget” and “prune”, so my first question is: does running the rest server in “append only” mode disable these features of the client?

Yes, if the client is run against that server and not directly against the data directory.

@quinn

duplicity has the concept of doing a full backup after some period of time, to protect against potential bugs in the “backup chain”, the idea being if there is some flaw in the incremental backups you would lose everything after that point in the incremental snapshots. Does restic support this, or is this even necessary

Restic does not currently support this as a proper feature. If you delete all of the index files on the server, the client will no longer know about any existing blobs and will not deduplicate anything. However, the missing index files will prevent older backups from being restored, and any future prune will remove the duplicate blobs anyway. Note that the indexes can always be regenerated with restic rebuild-index.

@quinn

should I use rclone to sync the backup files to s3Z or, somehow copy snapshots of the zfs file system to s3. or, use restic to back itself up to s3 somehow. not sure what the best approach here is. my goal here is to have local and remote backup, and also be reasonably protected against ransomware.

If you are never going to forget any snapshots, you can rclone with copy --immutable which will refuse to delete files on the destination as well as refuse to update any existing files on the destination.

Note that any potential attacker could compromise your S3 credentials, so you would also need bucket versioning along with credentials that do not allow deleting past versions of a file.

@quinn

for moving files to glacier, not sure if it is possible to identify “old snapshots” in the way restic stores its incremental backups on disk, so not sure if this would be possible.

There are many pitfalls with using the cold storage S3 tiers:

Restic and S3 Glacier Deep Archive

We had the same conversation when the Glacier storage tier (compare to Glacier Deep Archive) was announced and it honestly isn’t a good fit for restic for several reasons: The 180-day commitment makes pruning very difficult to do without causing early deletion fees. Simply pruning every 180 days doesn’t work because packs created one day ago could still need to be rewritten. Unless parent snapshots are simply not used, restic needs to be able to fetch tree objects on-demand, which necessitate…