Boris Cherny on Trying to Get Claude Code to Rewrite the Claude App

Better technology cannot compensate for poor product design. Two weeks' worth of tokens sounds like tens of thousands of dollars by the time it is done. Wouldn't it be better to hire someone who knows what they are doing, get it right, and teach the other engineers why and how? Tech dept and product dept creeping in on every LLM loop, compounding.

Boris is right about:

> The verification is probably the single most important thing that people do not get right, largely.

and so, so, so wrong about giving this prompt (for verification) and expecting that it succeeds at building a "good" app:

> I want you to run the Electron app in the Mac virtual machine, screenshot it, and then look pixel by pixel. Compare it to the Swift version. Don’t stop until you’re done.

Given he has let it rip for two weeks, I am assuming they have been post-training Claude for some version of this to be more _likely_ successful than not. However, IMO, the verification that you get from a visual comparison is shallow.

To state the obvious: there is a lot more than what meets the eye. But, I think, one could prompt a Fable/Opus 5 to actually go verify that "lot more"...

The question is: should one be imperative in asking for a specific types of verification (like a rubric) vs hoping that the Google/Anthropic/Open AI/Moonshot's post-training will take care of it.

I think, as things stand today, even with the best-in-class models today, I would be leaning more imperative. And it is not because I am an expert in SwiftUI or such. It is because I want to be able to say that _I_ (i.e. the human) verified that this thing works.

Interesting how he works with it, ”do you have access to macOS runner?” , “do you have access to repo?” And finally they will see that the biggest and hardest problem with native development is verification. So the magic word it seems, is “verify pixel by pixel”. But that would limit what you can verify and implement. There should be vision models trained in UI.

I’ve been formalizing behaviors to requirements and building more end to end tests to verify them. In reality, every requirement needs a verification element. Secondly, building traceability in your unit/integration tests helps “keep it honest” about how much it has built and verified.

Mostly I’m borrowing heavily from model based systems engineering and adapting it to some form of constitutional spec driven development.

The agents are now writing 90% of the code, which humans then throw away.

But actually this phenomenon of writing one to throw away is going to be amazing for what we can explore.

Pixel accurate is easy to say, hard to do, over-specific, and counterproductive.

If Claude is conscious he just instantiated hell.

Should we give Claude another two weeks to see the results (cost/loss/gain)... or just take whatever face value, now?

Next time, on Dragon Ball Z!

better is in the eye of the beholder?

[dead]

[dead]