What file system is it? FAT/exFAT can be pretty slow - especially with directories with a LOT of files in them, like Restic makes. Might try NTFS if it isn’t already. The underlying file system of the container may matter too.
Could mess with larger cluster sizes? Perhaps even mix-matched cluster sizes between the container and the underlying file system are slowing things down?
Might try resizing the drive’s main partition and creating an encrypted partition with TrueCrypt - that tends to be faster. Also AES is much faster on newer machines than most of the other algorithms available in TC.
Any reason you’re using TrueCrypt when Restic has (forced) built in encryption? Lots of overhead there.
Ps. VeraCrypt replaced TrueCrypt since it’s no longer being maintained, just on the off chance you hadn’t heard yet. 