The length of the passkey shouldn’t influence the overall duration of the commands by more than a few nanoseconds 
. restic always transforms passkeys into a 256 bit key (in a process that takes around half(?) a second) which is then used to decrypt the keyfile which contains the actual encryption key. From that point on the passkey is no longer actively used.
You could try to set the environment variable GODEBUG="http2debug=2" which (hopefully) shows what’s going on. Note that this log will probably contain lots of sensitive information like access tokens or parts of the (encrypted) repository data. The only lines which are relevant for debugging this would be those which contain on of the following words: “:status”, “:method” or “:path”.
If that doesn’t work, we’ll need a debug build of restic to see whats going on. (Or do you have another way to inspect the API calls issued by restic?)