Restic rclone:pCloud minor bug?

Getting Help
1

So I noticed pCoud has lifetime storage available, and they’ve been around for a few years at this point. I was thinking about getting the 2TB lifetime option, but wanted to do some testing with Restic first (since that’s primarily what I’d be using it for). I noticed pCloud wasn’t supported natively, but it was one of Rclone’s backends.

Soooo, I did this:

akrabu-macbook-air:~ akrabu$ restic init
created restic repository ead1d31fd7 at rclone:pCloud:Backup/akrabu-restic

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

akrabu-macbook-air:~ akrabu$ restic backup ~/Pictures
repository ead1d31f opened successfully, password is correct
created new cache in /Users/akrabu/Library/Caches/restic
rclone: 2019/09/01 19:14:26 ERROR : locks: error listing: directory not found
rclone: 2019/09/01 19:14:27 ERROR : index: error listing: directory not found
rclone: 2019/09/01 19:14:27 ERROR : snapshots: error listing: directory not found

Files: 60 new, 0 changed, 0 unmodified
Dirs: 2 new, 0 changed, 0 unmodified
Added to the repo: 184.769 MiB

processed 60 files, 198.247 MiB in 5:22
snapshot 1d4aa544 saved

During the backup, I checked and those three paths indeed did not exist. By the end of the backup, they were all three created.

akrabu-macbook-air:~ akrabu$ restic check --no-cache
repository ead1d31f opened successfully, password is correct
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
no errors were found

Subsequent backups no longer display the error, as the paths exist. Also running rebuild-index and prune and one final check all completed successfully.

So, I’m not sure if this is an rclone thing in general (first time I’ve used it as a backend), or an rclone:pcloud thing, but I thought someone might want to be aware of it. Either way it doesn’t seem detrimental, but it might cause some concern to someone just starting out. :slight_smile:

2

It seems more likely that pcloud follows the “eventual consistency” model of storage: when the put/create request returns success, requests to retrieve that resource might still fail for some amount of time. The success code does not mean “the resource exists now,” it means “we have received and persisted your request to create the resource and it will exist at some point in the future.”

S3 also follows this same model, but object creation is usually completed in less than a second. Perhaps pcloud has higher creation latency (either by design, or temporarily as a result of some system issue).

3

Well I tested it more by JUST doing the init and waiting. Those folders never showed up, on the website or on ForkLift (using WebDAV) until I ran a backup. :man_shrugging:t2:

4

In this case, “create directory” might be a no-op. Object storage systems like S3 usually store the entire path as the name of each object, so directories don’t really exist since you can create an object with an arbitrary path at any time.

Listing the contents of a specific prefix at one level is just asking the system “which prefixes exist under this prefix” which may or may not even be a supported operation to begin with.

The S3 console allows you to create directories, but these are just zero-length objects whose name ends with a trailing /.

Putting it all together: it’s possible that pcloud simply doesn’t support these kind of “directory” objects and so when restic goes to enumerate the sub-prefixes that exist under the repository prefix, pcloud doesn’t return snapshots etc. until something actually exists under that prefix.

At any rate, restic appears to tolerate these errors just fine so I’d say it’s safe to ignore them. They’ll be created on-demand when restic stores a snapshot, data pack, or index file.

5

From what I can tell, it’s actually a limitation of Rclone. Rclone does not create empty directories on any target. Tried it on pCloud and locally. Also tried just creating an empty folder on pCloud, which it allowed.

So TL;DR, Rclone doesn’t sync empty folders. And the only downside to this is that the very first backup will complain about said missing folders, and then it fixes it itself. :slight_smile:

1 Like