Is there a method to merge all the files in multiple snapshots?

Getting Help
1

Is there a way to merge the files in multiple snapshots?

I have a special requirement that may not fit within restic’s use cases, but I still would like to ask for advice.

I want to use restic to back up files to remote storage, thereby freeing up my hard drive space. I don’t care about snapshots.

For example, I want to back up the videos in a folder /home/aa/videos.
First, I execute the command: restic backup /home/aa/videos,
Then I delete all the files in /home/aa/videos on the local hard drive to free up space,
After a few days, I added some new files, and then I executed restic backup /home/aa/videos again…
Repeat this process.

Then I will get multiple snapshots.

When I want to watch videos, I execute command restic mount /mnt/videos, and I can retrieve all my video files.
However, there is a problem here: the videos will be separated into different snapshot folders:
/mnt/videos/snapshots/xxxx1
/mnt/videos/snapshots/xxxx2
/mnt/videos/snapshots/xxxx3
…
Inconvenient to access.

So, is there a way to display all the files combined in the same folder in mounted path?

1 Like
2

restic does not support it yet AFAIK.

If urgent and important you can try rustic for merging snapshots. It is alternative client supporting restic repository format.

2 Likes
4

Thank you for sharing all details! This is the right attitude:)

PS. I use both restic and rustic against the same repos as they have mutually exclusive features. Just never ever use both at the same time (as rustic is lock free).

2 Likes
5

Thank you for telling me this important point: “rustic is lock free”

6

Keep an eye out for Option to create cumulative snapshots · Issue #4804 · restic/restic · GitHub .

3 Likes
7

What problem are the snapshots causing you? I would suggest you’re best to keep them.

You really should keep multiple copies of your data. If something happens and you lose your restic repository the data is gone. Storage is relatively cheap these days.

8

So to provide the functionality you are looking for just have restic keep 1 snapshot, then when you copy ‘new’ videos to the repository it will perform dedup using a temporary space the retain one ‘complete’ snapshot containing all of your most current videos and hence mountable with rclone.

try a command like
‘restic forget --keep-last 1 --prune’ on the appropriate repository

It should work

9

Re-read the original post. The plan is to delete the source files after each backup. Your suggestion will result with the only copies of the files remaining being removed during the prune.

1 Like
10

yes your are correct, I completely forgot that he wants to delete the source data

my bad

1 Like
11

Okay, so you don’t care about having multiple copies.

To me, this sounds like the “category” of tool that should do what you want is some kind of cloud storage that supports streaming in that it creates a folder mount (or drive letter on Windows). For example, Google Drive can do this.

Can you elaborate on why you want to use a backup tool at all?

If it’s “only” the encryption you’re after, you could look into https://cryptomator.org/ which adds end to end encryption on top of any cloud storage.

12

I apologize for my poor English, but I’ll do my best to explain clearly.

I want to save my files to regular cloud drives, such as OneDrive, Mega, and various types of Chinese cloud drives. This refers to regular cloud storage services, not professional object storage services like AWS S3.

The fact is to just save, not backup, without the need for a snapshot feature. Then delete the local files to free up local hard drive space.

These cloud storage services generally have the following problems:

  1. There is a limit on the number of files in each folder.
  2. There is a limit on the size of individual files.
  3. Uploaded files do not support keeping the original modification time of the files.
  4. File operations are slow. This is a bit difficult to explain. It probably means that uploading a large file is fast, but uploading many small files is very slow.
  5. The filename has length restrictions.
  6. The filename cannot include special characters.
  7. Chinese cloud storage services will review file contents. If the file is illegal (in terms of law and politics), it will be deleted

So I searched for a way to overcome these problems. Restic seems to be a good solution. It can split large original files into small packs and also combine multiple small files into one large pack.

2 Likes
13

Thanks for the explanation! That makes perfect sense.

Unfortunately I don’t know a really good solution for this. Maybe restic really does come closest?

Just as an FYI, here is what I can tell you about Cryptomator, for each of the points you listed:

  1. The files that Cryptomator places in the cloud storage (I’ll call those “crypt files”) use various nested folders, and these folders have nothing to do with the real folders which you create inside the encrypted vault. For example, in my cloud drive, Cryptomator created 109 different folders on one level named “2E”, “2J”, …, "CT, “D3”, …, “X4”, “XA”, …, “ZU”. Inside of each of those are one or a few more folders with longer names, and in those folders there are individual crypt files - sometimes a few, sometimes a dozen. So if you have a folder inside your vault with e.g. 100 files, I would guess it is very likely that the crypt files for those files would end up in several different folders, which may help in staying below the files-per-folder limit.
  2. Cryptomator will not help with that. As far as I know every file inside the vault is stored as exactly one encrypted file and not split into several.
  3. Cryptomator will probably not help with this, either. The reason is that the crypt files in my vault have vastly different modification times, often older than the point at which I started using Cryptomator. I think Cryptomator sets the modification time of an encrypted file to the same value as the file inside the vault so that the client program of the respective cloud storage knows what it needs to sync.
  4. Cryptomator will not help with that as it does not combine several files into larger ones.
  5. The crypt files have names that can be somewhat long, for example fwrno2MCYmOnPAZASFvXtOTR682zXM0lxVe5DAs1Yju8ReNDOEDHUiuyjiPgNw==.c9r (68 characters).
  6. With Cryptomator, the storage service never gets to see the real names of the files inside the vault. So if those include special characters, it’s not a problem.
  7. Cryptomator’s encryption prevents the storage service from reviewing file contents (or names).
1 Like