Mismatch in file sizes between local and remote repository

Getting Help
1

I’m using Restic 0.17.3 on a Windows 10 machine and a Hetzner storage box.

I’ve created a backup that is around 53GB in size. Since it took forever to upload using Restic, I used an FTP program to upload the entire repository to the server.

When I ran the check command, the local repository reported no errors. However, there were index errors on the repository at the remote location. I then ran the build-index command and noticed that all the errors were due to file size mismatches. I suspect this might be related to differences in how Windows and Unix store file sizes, but I’m not certain.

What is the best way to avoid these issues? I want the first backup to be complete, but subsequent daily backups will be much smaller, so I plan to use Restic to directly update the remote repository.

Thanks for any advice!

2

Weird. Never seen that on Hetzner.

How did you sync the local repo up to Hetzner in the first place?

3

Uploaded with SFTP using FileZilla.

4

I don’t get it :smiley: Why did you not back up straight to that repository on the server in the first place? It sounds like you backed up to a local repository which you then copied to the repository server using FTP.

Is it possible that you used the ASCII mode in FTP to transfer the files rather than BINARY mode?

Generally speaking, you should provide more elaborate information about the symptoms of your problem and also the other things requested when you opened the form to create your initial post.

5
  1. The backup took too long, and twice resulted in disconnection.
  2. I’ve thought about it later, and it seems that FileZilla default is Auto mode and transfer files with no extensions as Restic files in ASCII mode and not binary. If this is the problem, I will test later with Binary mode.

  • The output of restic version.
    restic 0.17.3 compiled with go1.23.3 on windows/amd64

  • The complete commands that you ran (leading up to the problem or to reproduce the problem).
    The commnads were pretty basic, so I didn’t thought they are required:
    restic -r sftp:(remote_name):(repo_folder) backup C:\users\(myusername) --verbose=2

    • Any environment variables relevant to those commands (including their values, of course).
    none
    • The complete output of those commands (except any repeated output when obvious it’s not needed for debugging).
    The backup command completed successfully on the local folder.

6

Restic will not upload data that was already uploaded, so even if your backup process is interrupted several times, it should complete in the end.

7

I was worried that it will create some corruption. But I will give it another try based on your recommendation.

Any other way that can upload to the server without using Restic?

8

Restic is designed such that it can be interrupted and restarted when backing up. It’s fine.

You can upload files however you want without using restic. What is it you want to upload, exactly? And where is that something stored?

9

I can recommend rsync. If for some reason you want to have your local repo as a mirror in the cloud, you could regularily rsync the differences. That is what I’ve been doing for a few years and it works very well.

10

> You can upload files however you want without using restic. What is it you want to upload, exactly? And where is that something stored?

I’ve meant that I want to sync the local repository to a remote one. Not looking to upload any other files.
I will try with uploading via Restic own interface. I was worried that I’ll have to start over on each disconnect.

11

I was only looking to create the initial repository. I assume that after that, since the backups will be very small, it won’t be such a problem.
When you use rsync, does Restic works on the remote folder directly with no problem?

12

Yes it does.

1 Like
13

To copy or sync the local repository to your remote server, you can use a lot of different software, depending on what protocols the remote server supports. For example plain FTP(S), SFTP, SCP, rsync or rclone. At the end of the day, the files in the repository are just plain files, so as long as you can transfer them one way or the other, it’s fine!

As mentioned before, you do not have to worry about restic being interrupted and having to start over.

1 Like
14

Thanks for all the help and suggestions, I’ll try again!

15

I just wanted to share an update for future reference, in case it helps someone:
I eventually created the backup repository locally and synced it to the remote storage box using rclone.
Although both methods used SFTP, rclone was about 20% faster because it worked in parallel and the files are synced exactly with no problem of text/binary like in sftp program.
Since this was the initial backup, I also had better visibility into the progress, even though I understand that restic is capable of handling disruptions and disconnections effectively.
For anyone with a large initial backup to a remote location, it might be helpful to consider using rclone.

2 Likes
16

You might try using Restic with Rclone+WebDAV since you are using Hetzner - I find that even faster.

17

Thanks for the reply. I’ve only needed rclone for the first upload, I wanted make sure that the repository will be uploaded correctly. For the subsequent daily backups it’s less important since the uploads are usually very small (Max<1GB) so it moves quickly.

But I’ll make some tests anyway, it will be interesting to see the differences.

1 Like