I managed to solve this with my similar open source tool, including remembering "voices" across multiple sessions.. though I wouldn't say this is amazingly effective. It's not as good as when your call provider clearly knows which person is speaking, but it's largely been good enough.

Source code is here if you want to have an agent take a look: https://github.com/robgough/dictator/