the whole point of varnish software keeping a public version of "vinyl cache" as "varnish cache" with TLS is to give people a way to access a FOSS version with native TLS.
I think TLS is table-stakes now, and has been for the last 10 years, at least.
Terminate tls and you have your cache.
just use the tool that does the job.
TLS in -> hitch or caddy Cache -> varnish/vinyl TLS out -> haproxy
Connect them up with Unix sockets, if you like.