The future belongs to those who can refute AI, not just generate with AI

My entire professional life, I've been dealing with coworkers who just want their code shipped. Overly complex solutions to simple problems, inefficient algorithms, nested spaghetti code. And it was hard to just plainly reject it. I eventually had to become a mentor, teach them how to do it better. I've called myself a programmer even though I mostly just helped other people to program.

Over the years, I've lost the ability to focus on writing code myself, but I've seen a rebirth with the age of coding agents. I find it easier to review their code and they always follow my guidance. In fact, I sometimes feel like around year 2000 when I had "google" skills and the people around me did not.

It's much easier to produce garbage with AI and it will inevitably happen, but engineers who are able to think will simply use it as a new tool.

Lately I’ve been trying to develop this discernment skill by filing issues on vibe-coded projects, which requires taking a deep look into them and questioning their premise.

For example, there’s a tensor serialization called Tenso (71 stars) which advertises better cache line alignment after array deserialization. I suspect it’s heavily vibe coded. I read its source code and discovered that unlike what the README claims, the arrays it produces aren’t actually 64-byte aligned after all. https://github.com/Khushiyant/tenso/issues/5 They also have some issues around bounds checking: https://github.com/Khushiyant/tenso/issues/4

Another example: there’s a library called protobuf-to-pydantic (145 stars, 21 forks, 4 dependent packages on PyPI). I wanted to use this for one of my projects. I’m not sure whether it’s vibe coded or not. I found that it creates syntax errors in the generated output code: https://github.com/so1n/protobuf_to_pydantic/issues/121 Seems like a pretty surprising issue to miss.

For Tenso, the code quality is less of an issue than the “should this even be built or is the premise of the work solving the wrong problem,” which I suspect will be a factor in a lot of AI-generated work going forward.

I’m torn. On the one hand, I’m glad to be practicing this skill of “deciding what to rely on and what to refute,” but on the other hand, I certainly don’t feel like I’m being collaborative. These libraries could be their creators’ magnum opus for all I know. My issues just generate work for the project maintainers. I offer failing unit tests, but no fixes.

I earnestly think the future belongs to people who are able to “yes, and” in the right way rather than just tearing others’ work down as I’m currently doing. It’s hard to “yes, and” in a way that’s backed by a discerning skepticism rather than uncritiqueful optimism. Feels like a condradiction somehow.

We're building AI testing tools at QA.tech and this matches my experience. Great post. The hard part was never generating code. It's figuring out if what came out is actually correct. Our team runs multiple AI agents in parallel writing code and honestly we spend way more time on verification than generation at this point. The ratio keeps getting worse as the models get better at producing plausible-looking stuff.

The codebase growth numbers feel right to me. Even conservative 2x productivity gains break most review processes. We ended up having to build our own internal review bot that checks the AI output because human review just doesn't keep up. But it has to be narrow and specific, not another general model doing vibes-based review.

Examples of refutations would help. I agree with the thesis of learning epistemology but where does one begin? I hazard to suggest Montagovian semantics.

Why is Karl Popper’s face phasing in and out?

These are such a good points. With each new creation of technology, I don't think our comprehension gets better, our ability to collect data gets better. You can now grab so much of it, and analyze it so much, but if you don't know what you're looking at or the value or the Threat within, it's useless.

[dead]