We kinda already have it with NPU/TPUs, tho they are usually attached to CPUs (and for some reason come with near zero proper documentation).

I can see separate cards for datacenter use but for consumers they will probably come on same SOC as CPU