If you don't keep it as a toy, it can get out of hand quickly. For example, here's the partial implementation of QWidget, almost 500k of code: https://github.com/qt/qtbase/blob/dev/src/widgets/kernel/qwi...
FWIW it doesn't need to become 500k lines of code... or 13k in QWidget's case :-P. The file is large because it contains the full documentation in addition to the code but also in Qt is a QWidget can also be a (toplevel) window so it needs to support everything a window may want to support, including stuff like setting/getting title, maximize/minimize/etc events, etc.
You can avoid this complexity in the base widget class by placing those elsewhere, e.g. FLTK's base widget class[0] is much smaller at around 500 lines of code (+ documentation comments).
[0] https://github.com/fltk/fltk/blob/master/src/Fl_Widget.cxx
I haven't yet written a UI library so it's not something I can state confidently, but based on adjacent work I've done, I imagine that a significant deal of complexity can be avoided by somewhat paradoxically embracing complexity early on and designing for all the eventualities.
UI frameworks are one place where outside of toy projects, YAGNI doesn't really apply as much, and so things like focus systems and accessibility affordances should be factored in from day one. If you go in with the attitude of writing something "elegant", you're going to end up bolting these things on after the fact and writing contorted code to make it all fit together. It's like finding out that you need a skyscraper after building foundations and framing for a cottage.