You don't realize that "talk to my hand" is an insult, which is exactly what you are doing.