I think the disagreement mainly lies between whether one should be able to use their devices as they wish, like a desktop/laptop, or whether all devices should be restricted so as to have a good baseline level of security.
As far as I understand, the Linux phone enthusiast's POV is that we should start seeing the phone as another general purpose computer that can be set as per one's needs and can be suitably hardened as and when the need arises.
I wonder what environment the GrapheneOS devs use for development and other general computing needs, and if such an environment could be emulated on a phone (that is assuming, of course, that their development environment is way more hardened than current desktop Linux).
I suppose they could be using something like Qubes OS. I myself use a rooted phone, which does admittedly have security drawbacks but is useful in various ways.