Directly copying restic data between repositories

tl;dr:

  • Copy data then index then snapshot.
  • DO NOT copy from a source that has ANY concurrent activity
    • (unless you are copying from an LVM/btrfs/ZFS snapshot of that source)
  • DO NOT copy to a destination that has ANY EXCLUSIVE concurrent activity (non-exclusive is okay).

This actually is not the correct order. The order should be:

  1. data
  2. index
  3. snapshot

If index is copied before data and the copy is interrupted, future backups will deduplicate against data that isn’t actually there, generating corrupt backups.

If snapshot is copied before data/index and the backup is interrupted, restores will fail as dependent blobs/indexes won’t be found.

This also ensures that any concurrent non-exclusive activity against the destination repository is safe. Concurrent processes might see (and ignore) a partial file but they shouldn’t fail.

Exclusive activity on the destination (such as check or prune) will almost certainly react poorly to the presence of incomplete files. Prune in particular could delete data that is in use if it saw the data pack containing blobs but not the snapshot referencing those blobs (because it was not yet copied when prune looked at snapshots).


This order becomes problematic when a concurrent process is running on the source. Consider a backup:

The backup might generate an index file containing a data file that the copy missed because it copied the directory containing the referenced pack before the pack was created, and now future backups will deduplicate data that isn’t there.

Likewise, if the backup finishes during the copy, the snapshot file will be copied over even though the referenced data may not be there for the same reason.

This would hint to copy files in the opposite order, but then the destination repository is effectively broken until the copy completes.

There is no bulletproof way to copy from a repository that is being written to, except to take an atomic snapshot of the whole thing (LVM, btrfs, or ZFS snapshots would work) and copy from the snapshot.

3 Likes