Your general purpose computer is connected to the internet 24/7. It runs a complex operating system which constantly gets updated and patched. It visits dodgy websites, downloads npm packages, backs up files to the cloud, etc.
Thus the attack surface is far greater.
Or, get a mass produced general purpose computer and never connect it to the internet.
Is it though? There is nothing you can solder onto it that will gain any meaningful access and go unnoticed. Tampering with the supply chain is a lot more difficult, easily detectable, and wouldn't scale. I can use 100% open source software, in an OS where it's signed and completely safe from interference - except the NSA of course, but they are not interested in your money.
For example, in MacOS it would be extremely hard to "sniff the display" as in this attack, unless you take obvious steps to authorize screen access.