alt Hacker NewsI worked with ARM boards, I know a bit about it. Booting into Linux is never hard, it's all about using uboot, sometimes with tiny patches on top. I think it's actually even easier with android phones, as you don't have access to the low level bootloader, you just use fastboot stuff.
Having basic framebuffer in BIOS/UEFI is neat for toy OSes, but not very relevant for something practical. You gotta need proper driver for GPU. And if you're just starting, UART console is actually more preferable way to interact with board, IMO.
Booting into a mainline Linux kernel on your average junk-level SBC with all the hardware working (without simply sticking to an Android-like downstream/proprietary BSP) is quite hard, and that's what you need in order to make a phone usable as a daily driver. That's really the root issue; mobile phones are built as embedded devices, with no consideration for running a generic OS kernel. This isn't even an Android issue, OpenMoko was the same deal. If anything, Android was the first mobile platform to even loosely approach any kind of PC-like openness.