/data/ layout question

Getting Help
1

I noticed that all data that is put into s3 is put only into the directory /data/00 and no other directories (/data/[01-ff]) are used. Is it supposed to be like that? Is it possible to change this behavior?

restic version

restic 0.17.3 compiled with go1.23.3 on linux/amd64

How repos created:
restic init --repository-version latest
RESTIC_REPOSITORY=S3

2

Have you taken any snapshot yet? Other directories should be created as hoc when needed.

3

Yes, I have some snapshots:

# restic stats --mode raw-data
repository XXXX opened (version 2, compression level auto)
[2:52] 100.00%  2775 / 2775 index files loaded
scanning...
Stats in raw-data mode:
     Snapshots processed:  124
        Total Blob Count:  132139077
 Total Uncompressed Size:  64.525 TiB
              Total Size:  15.922 TiB
    Compression Progress:  100.00%
       Compression Ratio:  4.05x
Compression Space Saving:  75.32%

When/under what conditions will “Other directories should be created as hoc when needed.” happen?

4

S3 remote content after restic init:

/
├── config
└── keys
    └── f14e183fe048c8fd2eccbffcbb1c683d86a86852674a8db5fdb0abaa2b1f9691

After running first snapshot:

/
├── config
├── data
│   ├── a4
│   │   └── a4716484524240184d1cf10b5376e69b7faa4397597101c3a9dba6ca7f3715b7
│   └── fd
│       └── fdeaf48374136bdf9396d088cd211595991a56750494fbd029529c20f4bf432d
├── index
│   └── f224911525e9948ef522325844e3cd6a0a5d21ec2abbc8657867e322b7e3f9d1
├── keys
│   └── f14e183fe048c8fd2eccbffcbb1c683d86a86852674a8db5fdb0abaa2b1f9691
└── snapshots
    └── 7f3f82cb77587f162a955b0369b6b0a1dee1f6262cc20c4801b00e9f507e8198

and after second snapshot:

/
├── config
├── data
│   ├── 0d
│   │   └── 0defa7875a071b7175128cb8bac120996a1a9200d8440204a268b89fa5369b1c
│   ├── a4
│   │   └── a4716484524240184d1cf10b5376e69b7faa4397597101c3a9dba6ca7f3715b7
│   ├── d3
│   │   └── d3ee7bd2dda7cf0d94b228ec2d1f7c183c010b5f2c0158de28c8bc6bb9ef3c0c
│   └── fd
│       └── fdeaf48374136bdf9396d088cd211595991a56750494fbd029529c20f4bf432d
├── index
│   ├── 06d44b47939df5b19e6fd465974d66d83a16b92f5c09e0ee944289386310bc21
│   └── f224911525e9948ef522325844e3cd6a0a5d21ec2abbc8657867e322b7e3f9d1
├── keys
│   └── f14e183fe048c8fd2eccbffcbb1c683d86a86852674a8db5fdb0abaa2b1f9691
└── snapshots
    ├── 7f3f82cb77587f162a955b0369b6b0a1dee1f6262cc20c4801b00e9f507e8198
    └── badd32f9b1eab71e6cdba98e96fcf8b3b75b8c60cb9ccf30b2b79ebe53

What is your S3 provider?

Now as you maybe already know S3 storage does not have concept of directories. For “human readability” it is often simulated only when displaying content. Every object (file) you store in S3 uses key prefixes to represent the hierarchy. So you can assign prefix like “A/B/name”. This is what restic does.

Are these non existing “directories” shown or not will depend on software you are using to look at it. How do you check your S3 content? Which program do you use?

Also maybe some ancient restic version used different layout in the past and it is what you are still using. Either way it does not affects performance at all.

5

And the culprit might be here… as:

 --repository-version string       repository format version to use, allowed values are a format version, 'latest' and 'stable' (default "stable")

I guess you are using non default value for some specific reason:)

6

latest is currently the same as default, which in both cases is repository version 2.

At the size of your repository that’s completely impossible. Are you sure that the file listing you’ve used isn’t truncated?