If you have a policy of never giving these people the time of day, then it's no different

If you're the sort of person that is too polite to just hang up and you might get hooked by a scam caller because you let them talk to you, then it's a big difference