This is a relatively straightforward problem to solve. There is a “payment required” HTTP status code and lightning network micropayments are very easy to set up and use.
The problem is that payments that cannot be centrally surveilled and centrally vetoed (without burden of proof of probable cause, usually) are actually illegal to implement.
To do anything like this requires strong ID and KYC for payer and recipient and that isn’t what bare URLs and the web are about.
All of the cool applications of online payments that would spread quickly and efficiently are illegal.
It’s truly sad.
I think another problem with micropayments is competing with free is hard. Say you have the best lightning paywall with an amazing UX where it's super easy and automatic to pay a creator a few sats. Most people assume the problem is user's don't want to pay but it's actually publishers who won't put up micropayments on their content. Here's why: for a big publisher, adding a lightning paywall is a big risk which may lose them their audience. For small publisher's, they don't care if 10 people send 10 sats, all they care about is becoming a big publisher