"you can only work on one thing at a time. It should always be the highest priority item"

The thing you're working on in any given moment is the highest priority thing (in "your" mind) by definition though. If you thought something else was higher priority, you would be doing it instead.

The only "argument" against that requires a third party who deems a different thing higher priority than what you're currently working on, and that leading to a mismatch of what is "highest" priority is, and you're lack of doing it in the moment.