Temporary Files

[Egill]

Hi,

first of all thanks a lot for this great tool and all your effort!

Currently I am in the process of evaluating restic for my backup needs. I want to backup 2TB of data to a cloud storage provider using sftp. As I have a SSD as my system drive I am a bit worried about disk writes.

After reading the documentation and in this forum I assumed, that restic would write all data to temporary files, e.g. for encryption, before uploading it. Maybe I am wrong with this.

I did some testing with restic 0.9.6 (installed via macports) backing up files with sizes between 100MB and 1GB. Before and after the backup I checked the smart values of the SSD, during backup I monitored the restic process with the activity monitor (macOS).

I was quite surprised, that backing up the files resulted only in negligible disk writes (0,02 GB for a 1GB file).

The problem ist, that I do not understand why… (my TMPDIR var points to a directory on the SSD) Maybe the operating system kept the data in a cache in the RAM and it was deleted before it was written to the disk?

As I have 64GB of RAM I am thinking of putting the temporary files on a ramdisk. What would be a reasonable size for such a ramdisk, used for restic only?

Would it be worthwhile, regarding disk writes, to disable the cache as well or is the amount of data written to the cache not worth the disadvantages?

Thanks for your help!

Egill

[cdhowie]

@Egill

Maybe the operating system kept the data in a cache in the RAM and it was deleted before it was written to the disk?

This is definitely plausible. I don’t believe restic does an fsync() on the temporary pack files. If the uploads happen quickly enough, the filesystem driver could indeed eliminate the buffered writes.

@Egill

As I have 64GB of RAM I am thinking of putting the temporary files on a ramdisk. What would be a reasonable size for such a ramdisk, used for restic only?

My understanding is that the target size for a pack file is about 8MB and there’s two archiver threads. For two complete packs and two in-progress packs, you’d want a minimum of 8 * 4 = 32MB. If you want to be safe, a 512MB tmpfs would work just fine (tmpfs does not reserve space in advance, so you’re not dedicating 512MB, just setting a limit on how much storage can be used).

[Egill]

Thanks for your help!

This is definitely plausible. I don’t believe restic does an fsync() on the temporary pack files. If the uploads happen quickly enough, the filesystem driver could indeed eliminate the buffered writes.

To test this I will try to limit the bandwidth, so that the upload takes longer.

My understanding is that the target size for a pack file is about 8MB and there’s two archiver threads. For two complete packs and two in-progress packs, you’d want a minimum of 8 * 4 = 32MB. If you want to be safe, a 512MB tmpfs would work just fine (tmpfs does not reserve space in advance, so you’re not dedicating 512MB, just setting a limit on how much storage can be used).

So this should be no problem. I would have to use a ram disk, as tempfs is not supported by macOS, at least as far as I know, but I could spare the 512 MB. I will try this out as well.