An interesting idea but in theory just three correct pass codes and some brute force will reveal the secret key so you'd have to be very careful about only inputting the pass code to sites that you trust well.
It's definitely computable on a piece of paper and reasonably secure against replay attacks.
Yep, I am aware, 2 or 3 OTP's and timestamps plus some brute forcing using the source-code. Server-side brute force by input should or could be implausible. But that is why I am signaling here that I would love a genius or a playful expert/enthusiast contributing a bit or two to it - or becoming a co-author.
I was wondering about the overall security. How did you determine that 3 pass codes and brute force will reveal the secret key?