I have experienced some similar issues. I think some of it related to the "locked" state of the device. Siri needs context data to answer, particularly the mom or some destination questions. Specifically for contacts or recent places data. This context isn't remotely stored, but provided by the device to Siri each time. I think when the phone is locked it doesn't have access to the data (reading or writing). When I mean "Siri", I mean both the on device and remote parts of it.

I think this also interacts with countries and states that have (possibly misguided) strict laws forbidding the "touching" of phones "while driving". My experiences suggest that using Siri when driving and the device is locked, it just gives up - I sort of see the start of it working then, bam, it stops. If I retry, I suspect that I've somehow "looked" at the phone in frustration, it saw my attention and unlocked. I now wonder if where I have placed the device is making a difference.

It does seem to work much better (when driving) if the device is already unlocked.

I also see odd things when using Shortcuts for navigation. If I've previously asked for walking directions and then speak the shortcut while driving it won't give directions until I switch to the "car" icon in maps. I think it might be trying to calculate the 15Km walking directions, but it doesn't complete before I tell it, frustrated, to stop.

When Siri doesn't work it is usually the times when I need it to. This is definitely a multiplier in disastisfaction.