I haven't yet written a UI library so it's not something I can state confidently, but based on adjacent work I've done, I imagine that a significant deal of complexity can be avoided by somewhat paradoxically embracing complexity early on and designing for all the eventualities.

UI frameworks are one place where outside of toy projects, YAGNI doesn't really apply as much, and so things like focus systems and accessibility affordances should be factored in from day one. If you go in with the attitude of writing something "elegant", you're going to end up bolting these things on after the fact and writing contorted code to make it all fit together. It's like finding out that you need a skyscraper after building foundations and framing for a cottage.