I’m using Restic with Backrest for periodic backups of my media, primarily photos and videos from my own cameras. The setup is working great so far!
However, I don’t need to backup my videos in full resolution. If one of my ‘master drives’ fails, I’d still be happy recovering my footage in a decent resolution, as long as I can significantly reduce storage space for my backups. For reference, I find Google Photos’ video compression quite effective—it reduces 4K videos to 5-10% of their original size while still looking good on a phone. I’d love to achieve something similar for my backup strategy.
Are there any guides or best practices for incorporating lossy video compression into a Restic backup workflow? I imagine this might also be useful for people archiving data to AWS Glacier or tape storage. Since my backup system is write-once, read-many times, I could also develop an ad-hoc solution for compressing videos before archiving them. However, if there are existing approaches or tools that already accomplish this efficiently, I’d love to explore them instead.
You could pipe videos through ffmpeg and then on to restic to save them in a snapshot every time you run restic. But this is slow and uses a lot of CPU cycles so it doesn’t make much sense in my view. And doing it only once for all newly added video files kind of defies the purpose of using a snapshot based program like restic as every snapshot will contain only new videos.
So, alternatively, you could consider compressing all videos into a new “compressed” folder once at the time when you save them on your local drive. Then you can backup the “compressed” folder away with restic. You’d have to make sure you delete the compressed video when you delete the original, of course. The disadvantage of this is that you will use more drive space locally (if your calculation is correct, 5-10% more).
For photos you might consider using ImageMagick convert in a similar fashion.
I’m going to echo @nicnab on this one - at creation time of your video, you need to take a second lower res copy to a different directory and back that up, not your full res if you want to save space.
If you don’t want to do that in real time, you could script out background compression (Tdarr? Bash? PowerShell?).
Thanks, @nicnab and @g33kphr33k! I’ll generate lower-resolution copies of my videos in a separate directory for backup, while excluding the original files.
In the long run, I’d like to implement a FUSE layer that acts as a filter between Restic and the filesystem. Restic already respects the special CACHEDIR.TAG file, which tells it to exclude the current directory from backup. In a similar vein, I am thinking of adding another special file that provides mapping for these files. This way, I can conditionally include original files, exclude others, or redirect file access. For 4K videos not yet in a Restic snapshot, this would redirect to a pre-generated, low-resolution proxy. Restic then backs up the snapshot as it “sees”.
Would love to hear any thoughts on this approach!
I must admit that I didn’t fully understand your proposed solution but I guess it would work as restic doesn’t care, where the files come from. Just be aware that it creates hashes of chunks of files and backups everything it didn’t already backup. So these chunks should stay the same from one snapshot to the next.