It's very, very hard for untrained people to be strict about verifying any secret phrase. The attacker can make all kinds of excuses, while creating urgency, and many people quickly abandon verifying the phrase. A scene in One Battle After Another comes to mind.
At some point, the scam evolves to a live video of a gagged loved one being tortured. "Stop wasting my time or they lose another finger."
People aren't prepared for this shit.
Scammers can also trick the victim into reversing the roles and telling password to scammer. Even banks ocassionaly get this wrong. I have had my bank call me and ask me to read numbers from number card. If a trained bank employee following a script designed by (hopefully) an expert cant get it right, the chance of elderly relative spotting mistakes in protocol is close to 0.