unless you're contracted (fixed bid) working on jobs then you're getting paid hourly, salary, whatever... boss tells you to stand around and shoot the shit, thats what you do. I dont know why people think 'not my job' is a relevant answer... the job is what they tell you to do...
I do agree with you, and most jobs have a "and other duties as assigned" for that reason.
But I'll equivocate by saying there are exceptions. If you work a union gig (technically a contract), you have to be careful to stay in your lane unless you want a grievance filed. If you are a licensed engineer and your boss tells you to design/stamp something outside your domain of competence, you have a duty to say no. But that kind of stuff is the exception.