The big difference you're not looking at is the memory bandwidth. The M5 Pro has nearly twice the one of the M6:

170GB/s memory bandwidth Configurable up to 32GB unified memory

307GB/s memory bandwidth Configurable up to 64GB unified memory

That is also the factor you want to take into account for AI inference, of course.