Have you checked out the L402[0] protocol?

It's basically using the HTTP 402: Payment Required status code and serving up a Lightning Network payment invoice.

Edit to add: it basically solves all of the caveat issues you identified.

[0]: https://l402.org/