Hi,
I’ve been using Restic for a few months now and I’m quite happy with it. But on a recent deployment of an email server, I’ve found that some of my inbound emails are corrupted at random times, sometimes before they arrive and sometimes afterward. Possibly during the times Restic runs.
When I contacted support for help, they told me it’s possible Restic is responsible:
I did a bit of research into Restic, and I found that it does lock files while performing backups to ensure data integrity and prevent conflicts during concurrent operations. So the fact that it runs every 6 hours means it’s continually locking files during the backup process and causing corruption when the mail server goes to access the same locked file. The fact that you are seeing these issues somewhat sporadically also checks out with that logic as the mail server may not have been accessing some of those directories or files during the backup process.
I have Restic set to run on a cron job every 6 hours. I wanted to know if Restic does actually lock files when it’s backing up, because I don’t think the documentation says this but I could be misinterpreting it.
I see other people are also using Restic for backing up email servers without trouble: Backing up busy email server
Thanks for any help!
Backing up files which are in use always is risky IMO. It is not restic specific issue but reality and nature of creating copy of live system.
If this is serious system used for production make sure you use modern filesystem supporting snapshots and always backup snapshot, never live system. Otherwise you are in the world of chances:) things might work if you are lucky but not by design.
Here good link I always share on this subject:
1 Like
That’s a good point! Files are constantly moving around, being written to, etc. by a mail server, so there’s no way to guarantee you’ve got 100% the right copy of a file. That’s reason enough not to use Restic for this case.
But would running Restic cause my emails to become corrupted on the mail server? Does Restic lock the files to ensure data integrity, and therefore cause corruption when the mail server goes to access the same locked file?
I do not have definitive answer for your specific question. You also do not provide any details. What OS you are using? What filesystem? Etc.
If you have access to your email software vendor support maybe you should ask them for advice what is the best approach for its data backup?
Ubuntu 24.04, ext4.
rsync is the recommended backup solution as it doesn’t use a locking mechanism like Restic.
I’m asking here because I want to figure out whether the corruption problem is being caused by the email software, by Restic locking the files it backs up, or something else completely. Restic developers are experts in Restic, and my email software vendor is an expert with their own software, so I figured Restic developers were the ones to ask about its behavior.
You could have snapshots with ext4 using LVM but if not possible do this in two steps.
rsync to other local location (can be other directory or external disk etc.)restic from that location
As far as I know the only locks that restic uses are with the repository. It would not make much sense to lock a file that is being read. Writing to a file that is being read by restic should not result with corruption. Even if restic did lock the file when reading, that would prevent other programs from writing to the file, and there would be no corruption.
Are you sure that the extra stress that restic places on the cpu/hardware is not causing the issues.
2 Likes
Okay, thanks for letting me know.
That could be possible. Checking my logs, the Restic backup runs every 6 hours and takes between 10 and 30 seconds to complete. Then it prunes old snapshots and checks packs for corruption. This whole process took 1 minute and 41 seconds the last time it ran, which is about average. I don’t have the stats for CPU usage over time, but just the email service wasn’t using more than 60%. Right now, it’s using less than 30%.
Though, since stopping the Restic cron job, I haven’t gotten any more corrupted emails in the past 24 hours.