I'm very intrigued by AI note takers, but I'm absolutely unwilling to expose me or my clients to this exact problem.

The solution (theoretically) is a purely local note taker, but I haven't found one that's any good. Tried meetily and others in the same vein, including briefly rolling my own. The breakdown in the pipeline seems to be reliable local diarization and speaker identification; even if the transcription is good, when speakers aren't accurately identified and speech isn't well grouped, there's no rescuing it in the summary step.