I have been thinking about this seriously myself. Not with a specific existing schema like DocBook, but with a custom schema (defined by me) that I then compile to standard schemas, like DocBook or HTML.

This seems extensible to the degree that I want (i.e. semantically rich enough that you can conceivably hang any application from it). But I just can't bring myself to write in XML syntax, especially for maths.