I've used Google Voice since it was Grand Central and switched to Fi the month it came out, and I have never experienced what you're describing.