> Pixel phones are the only ones with secure enough hardware
The biggest thing that excludes most phones from supporting GrapheneOS is the lack of unlockable bootloader. Pixel phones also allow the developers to target a large but homogeneous hardware base.
There is no single biggest thing. GrapheneOS has a rather strict demands for a device they're willing to support, see https://grapheneos.org/faq#future-devices