“Easy” in relative terms for people who hack on electronics. Even if you remove the speaker, modern phones will tell the victim that someone else’s AirTag is moving along with them, unless the owner of the AirTag is also present.