If you can be tied to a chair and beaten with a rubber hose until you produce the token, it's just a password, albeit one that rotates.

TOTP works because you have to possess the secure device at the time you're authenticating. If you don't have the device, then no amount of time with the rubber hose can make you cough up the required token.