Hi,
I attempted to copy the complete s3 based repository to a new one. So both are s3 based.
I’ve tried many ways with --from-repo s3:https://… --option source.access-key-id=xxx --option source.secret-access-key=yyy
What is the best way to just copy every snapshot from an old to a new s3 repo?
Thanks
Exactly what you tried - restic copy
Look at the documentation. First example does exactly this - copy all snapshots from the source repository to the destination repository.
And if you simply want to move one S3 repo to another S3 storage maybe the easiest is simply copy all files using whatever S3 compatible tool you like - awc CLI, rclone, etc.
I think the question was how to specify different S3 options for the source and the destination.
AFAIK this is not possible this way, but instead you have to use a rclone configuration for the source or destination.
Exactly.
I have two different S3 services with different keys and passwords.
I want to copy all old data / snapshots to the new one.
I’ve tried all googled hints. Nothing worked as expected.
Are you asking how to copy data from one S3 bucket to another bucket? From an S3 compatible service to S3? Details matter. Please be precise about what you’re trying to achieve if you’d like help.
If it’s from an S3 bucket to another S3 bucket then you can use bucket replication or aws cli
I have two completely different S3-compatible storage addresses, in different cities, with different access and secret keys. I want to import all old snapshots etc. from the old S3 storage into the new second S3 storage. In other words, I want to consolidate two S3 storages into one on restic level.
The problem seems to be, that restic is only able to handle one pair of access and secret key.
I’m now trying to copy the old files to the new S3 storage in a second bucket, so that I hopefully can use the restic copy between two buckets on the same (new) S3 storage…
Since a restic repository is just files, you can use any tools (rclone for example) to transfer the entire folder structure in S3 bucket A to S3 bucket B – it could also be just your local hard drive, as it doesn’t matter where the repository lives.
You don’t need to use restic itself to migrate your data. Does that make sense?
You mean I can simply mash-up the directories and get the older snapshots valid in the new S3 repository?
No. You can not “mash-up” two repositories. It is only option if you want to move S3 repo to new “home”
Somebody already provided solution what to do. Configure one S3 repository as a rclone remote. Then you can use restic copy. So from restic perspective you will copy between S3 and rclone repos.
You can use rclone / rsync to synchronize the same repository to multiple/different locations.
Meaning that you can keep multiple copies of the same repository at different locations. This can be a local copy on a spare hard drive attached to your computer, which you then synchronize to the cloud somewhere. In case the hard drive fails, you have access to the same restic repository via the copy in the cloud, which you can download to a new hard drive. In your case the hard drive is just another S3 bucket.
The reason I’m saying all this is because of this sentence:
Additionally:
If you mean two different repositories (separate restic init commands for each repository (to keep it simple)), then no you cannot do that.
Does this and the other replies to your question make sense and help you towards your goal?
Are there still unknown things or issues which you need help with?
Currently I have the old S3 storage/bucket on the old host,
and the temporarily S3 storage/bucket on the target host mounted with s3fs.
After copying the old files to the new temp. S3-storage/bucket I hopefully need only one pair of AWS_ACCESS_KEY and SECRET_KEY. Then I hopefully can consolidate the temp. S3-storage to the target S3 storage with
restic copy --repo s3.target-host/target-bucket --from-repo s3.target-host/temp-bucket
After that I can report the results… 
I’ve missed to tell, that the new S3 storage/bucket already has new snapshots… Maybe that makes the problem more clear.
Restic repositories are just sets of files and folders. Copy them to wherever you like using whatever tool you like. You have to copy the whole repository, you can’t copy part of it.
You can have as many repositories as you like in an S3 bucket, you just need to use folders. Here’s an example I use to backup on my windows PC
restic backup --no-scan --pack-size 128 --compression max --repo d:\Backups\Photos\ C:\Photos
Here’s an example from a Linux server that backs up to S3
restic backup --repo s3:s3.amazonaws.com/s3-bucket-name/folder /opt/
Thank you all, I see there was no need to copy all files to that temporarily folder on the new S3 host.
It would have been better and quicker, to mount only the old storage with S3FS/rclone and do the “restic copy” then. Sorry. In Germany we say “ein Brett vorm Kopf haben…”
I’m using restic and kopia since 2019. I should have known.