Repeated damaged packs with Mega S4 backend

Getting Help
1

I am using Restic on 3 machines to backup large datasets of a few terabytes each to the Mega S4 backend, using the S3 API compatibility in Restic. Recently I ran restic check --read-data on them (which took a few days) and found that they all had a few damaged packs. I consider this worrying, as it seems that this has occurred on each machine/repo independently of each other. I wonder if I am using Restic wrong, or if there is a compatibility issue with the backend.

Here are some specifics: I have a common script that runs on each machine daily to trigger restic and notify me of backup failures. Here is the version that runs on one of my Linux hosts:

#!/bin/bash

repo="s3:s3.ca-west-1.s4.mega.io/redacted"

export AWS_ACCESS_KEY_ID=redacted
export AWS_SECRET_ACCESS_KEY=redacted
export RESTIC_PASSWORD=redacted

err() {
    curl -d "$1" ntfy.sh/redacted
    exit 1
}

if ! restic -r $repo --pack-size=100 backup /mnt --exclude /mnt/m15*; then err "Vakt backup failed. Check server config"; fi

if ! restic -r $repo forget --keep-daily 3 --keep-weekly 3; then err "Prune of vakt backups failed. Maybe there's a stale repo lock?"; fi

if ! restic -r $repo prune; then err "Unable to prune vakt backups. Investigate plz"; fi

if ! restic -r $repo check; then err "Vakt backup repo corrupt. Investigate"; fi

restic cache --cleanup

exit 0

The order of operations is the same on all machines: backup, forget, prune, check, cache --cleanup. Two of them have --pack-size 100 set; the other does not.

Here is the full output of restic check --read-data on one of the machines (this one running Windows 11). Note the frequent transfer retries; this appears to be consistent whenever the Mega S4 backend is used, but I would expect restic to handle these cases gracefully (and in fact it does appear to).

Terminal output 
PS C:\Users\ben> restic -r $repo check --read-data
using temporary cache in C:\Users\ben\AppData\Local\Temp\restic-check-cache-369260775
create exclusive lock for repository
repository d2c6334a opened (version 2, compression level auto)
created new cache in C:\Users\ben\AppData\Local\Temp\restic-check-cache-369260775
load indexes
[0:06] 100.00%  74 / 74 index files loaded
check all packs
check snapshots, trees and blobs
[0:03] 100.00%  13 / 13 snapshots
read all data
Load(<data/05a81629d5>, 104878864, 0) returned error, retrying after 1.310847009s: readFull: unexpected EOF
Load(<data/05a81629d5>, 104878864, 0) operation successful after 1 retries
Load(<data/95edfbba3e>, 19055449, 0) returned error, retrying after 1.216894284s: readFull: request timeout
Load(<data/95edfbba3e>, 19055449, 0) returned error, retrying after 1.082109306s: readFull: request timeout
Load(<data/95edfbba3e>, 19055449, 0) operation successful after 2 retries
Load(<data/75155c850e>, 105540424, 0) returned error, retrying after 1.364258527s: readFull: request timeout
Load(<data/75155c850e>, 105540424, 0) returned error, retrying after 2.163745958s: readFull: request timeout
Load(<data/75155c850e>, 105540424, 0) operation successful after 2 retries
Load(<data/b513659745>, 107369457, 0) returned error, retrying after 1.415071937s: readFull: read tcp 10.141.120.178:2276->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/155bae2437>, 105261852, 0) returned error, retrying after 804.387595ms: readFull: read tcp 10.141.120.178:4350->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/b513659745>, 107369457, 0) operation successful after 1 retries
Load(<data/155bae2437>, 105261852, 0) operation successful after 1 retries
Load(<data/d564f1a7a1>, 105349879, 0) returned error, retrying after 1.366726721s: readFull: unexpected EOF
Load(<data/d564f1a7a1>, 105349879, 0) operation successful after 1 retries
Load(<data/b5919ab459>, 17753561, 0) returned error, retrying after 1.447152731s: readFull: read tcp 10.141.120.178:6582->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/b5919ab459>, 17753561, 0) operation successful after 1 retries
Load(<data/e5bc9b2149>, 106226954, 0) returned error, retrying after 1.331888432s: readFull: read tcp 10.141.120.178:4319->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/e5bc9b2149>, 106226954, 0) operation successful after 1 retries
pack 770e388b80e4e482c948b4e180f1bddf7f75946afebeebb3cc6233e15f44dcfc contains 1 errors: [unexpected pack id 500bad309e2c0fcbe6c1c2e1d6fee46107dbf9b26125682e3cb5a0557c803ef6]
Load(<data/c9744eff7d>, 105345661, 0) returned error, retrying after 538.673722ms: readFull: request timeout
Load(<data/c9744eff7d>, 105345661, 0) returned error, retrying after 1.741474976s: readFull: request timeout
Load(<data/c9744eff7d>, 105345661, 0) operation successful after 2 retries
Load(<data/3985ca63c5>, 106099759, 0) returned error, retrying after 628.801669ms: readFull: read tcp 10.141.120.178:10541->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/b9c2ed3d7f>, 105154162, 0) returned error, retrying after 1.384095274s: readFull: read tcp 10.141.120.178:1073->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/396d94a483>, 17345909, 0) returned error, retrying after 820.51788ms: readFull: read tcp 10.141.120.178:4318->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/791d373a58>, 105616248, 0) returned error, retrying after 521.012119ms: readFull: read tcp 10.141.120.178:4327->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/396d94a483>, 17345909, 0) operation successful after 1 retries
Load(<data/69b029c5ca>, 17403185, 0) returned error, retrying after 1.25887504s: readFull: read tcp 10.141.120.178:4586->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/69b029c5ca>, 17403185, 0) operation successful after 1 retries
Load(<data/791d373a58>, 105616248, 0) operation successful after 1 retries
Load(<data/b9c2ed3d7f>, 105154162, 0) operation successful after 1 retries
Load(<data/3985ca63c5>, 106099759, 0) operation successful after 1 retries
pack dc6f3d09baf055491897ae0d67f508c542235210f51b7a35d947883cffdc3df6 contains 1 errors: [unexpected pack id 83443810a367ab9771dfde5571d2ebac05040d291883762cd0ce99b44b6d52ac]
Load(<data/ccee6d5ff0>, 105391371, 0) returned error, retrying after 755.939923ms: readFull: read tcp 10.141.120.178:9524->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/1c77b4eb70>, 105379020, 0) returned error, retrying after 822.974026ms: readFull: read tcp 10.141.120.178:5485->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/ccb9646e87>, 17316312, 0) returned error, retrying after 567.238054ms: readFull: read tcp 10.141.120.178:1656->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/ccee6d5ff0>, 105391371, 0) operation successful after 1 retries
Load(<data/bc8e20b7d7>, 106064147, 0) returned error, retrying after 1.273085811s: readFull: read tcp 10.141.120.178:1659->185.206.25.75:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/ccb9646e87>, 17316312, 0) operation successful after 1 retries
Load(<data/1c77b4eb70>, 105379020, 0) operation successful after 1 retries
Load(<data/bc8e20b7d7>, 106064147, 0) operation successful after 1 retries
Load(<data/fe231727ac>, 104988629, 0) returned error, retrying after 1.159497416s: readFull: request timeout
Load(<data/5efe976d0d>, 105253236, 0) returned error, retrying after 1.087554018s: readFull: request timeout
Load(<data/2e007d73af>, 105061720, 0) returned error, retrying after 1.441305604s: readFull: request timeout
Load(<data/8ec80e02e2>, 104882943, 0) returned error, retrying after 1.19227147s: readFull: request timeout
Load(<data/5efe976d0d>, 105253236, 0) returned error, retrying after 1.263889929s: readFull: request timeout
Load(<data/fe231727ac>, 104988629, 0) returned error, retrying after 2.704718244s: readFull: request timeout
Load(<data/8ec80e02e2>, 104882943, 0) returned error, retrying after 2.336122559s: readFull: request timeout
Load(<data/2e007d73af>, 105061720, 0) returned error, retrying after 1.194401982s: readFull: request timeout
Load(<data/2e007d73af>, 105061720, 0) operation successful after 2 retries
Load(<data/8ec80e02e2>, 104882943, 0) operation successful after 2 retries
Load(<data/fe231727ac>, 104988629, 0) operation successful after 2 retries
Load(<data/5efe976d0d>, 105253236, 0) operation successful after 2 retries
[131:47:07] 100.00%  70168 / 70168 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs dc6f3d09baf055491897ae0d67f508c542235210f51b7a35d947883cffdc3df6 770e388b80e4e482c948b4e180f1bddf7f75946afebeebb3cc6233e15f44dcfc
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

I would also like to note that two of the machines are running Debian Linux, and the other is running Windows 11. They are each running either restic version 0.17.3 or 0.18.1.

In the short term, I have repaired the repos and have suffered no data loss. However I suspect that these repos may become damaged again. So in the long term, I would like to find out whether restic or my Mega S4 backend are causing the issues and how future damage can be avoided.

2

I’d very much recommend updating to the current release of restic which at the time of writing this is 0.19.1, and correcting the repositories. Then see if the problem still persists.

3

Will do and update if the issue persists.

4

damage on three separate machines/repos at once usually points at the backend, not restic. the frequent transfer retries you mentioned are the tell. a lot of the cheaper s3 compatible providers mangle multipart uploads under retry and you only find out months later with check --read-data.

couple things that helped me on a flaky s3 endpoint: drop -o s3.connections (default 5) to 2 or 3 so you are not hammering it, and turn off multipart for smaller parts if the provider supports it. also worth setting a bigger pack size (–pack-size 64 or 128) so there are fewer objects and fewer chances to get a bad one.

and run check --read-data-subset=5% weekly instead of a full read-data every few months. catches it while the source files are still around to re-backup.

if it keeps happening on mega after 0.19.1 i would just test the same data to a different provider for a month. that tells you fast whose fault it is.

5

Thank you for these suggestions. I was unable to find an option to disable multipart uploads on restic’s end or the backend, aside from switching restic’s S3 interface to rclone. But anyway, it seems to me that restic only attempts multipart uploads on packs exceeding 200MiB, which is larger than the pack size on any of my machines. So I think that can be ruled out, if I am not misinterpreting the code that I linked (likely).

I am confused as to how the data corruption could have happened. It is my understanding that a hash is sent with the PutObject request, and an S3-compatible API should return an error if the hash does not match what it received. That would mean that either Mega S4/restic are not doing/handling this check correctly (bad), or the data is being corrupted at-rest after the upload (very bad). Looking at the S3 documentation, it appears that the hash is computed differently for multipart uploads, which I assume is where the defect that you suggest could exist. But again, I don’t think that restic is doing multipart uploads for me. So if my understanding is correct, this would point to an exceedingly elementary error on Mega’s (or restic’s) part.

The pack size is already 100MiB on one of the machines. It will be interesting to see if it takes longer than the others for that repo to become corrupted again.

check --read-data-subset looks useful. I will add that and s3.connections to my backup script. Actually, they look especially useful together; I have been dreading further troubleshooting of this issue because Mega S4 can be exceptionally slow to serve packs, meaning check --read-data takes several days, during which the repo is locked from new snapshots. This appears to be a latency issue stemming from the data being stored far away from me. Fortunately, it turns out that bandwidth scales linearly with the number of connections, up until saturating my gigabit downlink at s3.connections=80! I am not brave enough to use more than 2 connections for actual backups (yet), but for reading back the data on a regular basis this will make my life easier. I have also replaced the flaky gateway that my LAN was using before.

Using 2 connections for backups with the new gateway should help reduce transport errors, which should in turn improve the situation if the upload hash verification is indeed broken. Anyway, we will see if the issue persists.