The one thing that will hold out forever or will at least be the last thing to fall is critical thinking. So I think you simply interview for that.

At my small company we interview all candidates, regardless of experience level, using the same set of questions during the primary round. They’re a combination of open ended scenario type questions and pure debate style questions. We try to design them such that when you’re answering them, it’s similar to when you’re on the job collaborating on a technical solution, or triaging an incident.

We provide limited information at first to force the candidate to immediately ask for more info, to see what kind of questions they ask and what kind of information they care about. When they start to provide the solution, we like to throw in alterations or wrenches to see how they pivot. It’s very much “turn based”.

If you’re taking it seriously, it’s very obvious when candidates are either solving it for the first time, are recalling true experience, or are simply reciting information like they studied it. That’s why many of our questions work for many levels of engineering.

Additionally, because there is so much back and forth, you get to also “vibe-check” the candidate engineering wise. In our experience most candidates have enjoyed the process, so we feel like we uncover a lot about personalities here as well.

Unfortunately, the heel to this approach is that you pretty much have to conduct these in person. Because an LLM hypothetically would do well in this style of interview. We’re hybrid one day a week so it works for us, but it doesn’t exactly scale.