You’re absolutely right. If the cache is on the same disk as your repo, it has no benefit. The whole idea of caching is to keep stuff close when you’re dealing with remote repos, to cope with latency etc…
Do use --no-cache flag and you should actually expect things to go little faster (with no unneeded copying between repo and cache on the same disk).