I think XHTML failed because it didn't give web devs any new capabilities, so most didn't feel the need to learn it and do the extra work of getting their tags correct.
Then html5 came along, providing all kinds of shiny goodies and saying not to bother with the tags. In the end, a more rigid standard would have been nice.. (Though this is mostly about the skin deep part of the standards.)
> I think XHTML failed because it didn't give web devs any new capabilities, so most didn't feel the need to learn it and do the extra work of getting their tags correct.
xhtml was entirely opt-in, people opted into it, then served broken content. xhtml failed because that broke content (from people who, again, had specifically opted into serving xhtml) was an utterly terrible for everyone involved, as the user would get a big fuck off page devoid of any content, information, or means of redress, and there was no way for administrators or authors to get notified that their content was broken.
Meanwhile HTML would usually let you do the things you wanted to, and if you noticed something was broken you'd usually be able to hunt down a contact form and send a notice.
HTML5 is not what killed xhtml, xhtml is what did that, because it was a dreadful experience all around and had absolutely no redeeming quality.
Hell, the W3C was so into xml at the time there was an xhtml5 serialisation for html5. Technically it's still there (https://html.spec.whatwg.org/multipage/xhtml.html). That was of great use to the nobody whatsoever who was interested.
> I think XHTML failed because it didn't give web devs any new capabilities,
and what new capabilities does this new proposal provide?
That is not how I remember it (for a data point of one shop in New England during the time): we embraced it because of the binary validation under multiple theories. There was a strong suggestion valid html did better from an SEO perspective, so we could sell that, a suggestion browsers would be less buggy with properly formed xhtml and a number of theories about what the future held for bots and scrapers to be able to easily ingest and parse your content (seen as a good thing then).
It failed because the smallest error by a client after the fact was like a server crash. Plus it would have created a mild barrier to entry when learning html at all.