People tend to use LVGL for this kind of projects. Seems adequate. No X server, direct framebuffer write, good tooling.