logoalt Hacker News

How the Claude Code team thinks about verification

105 points • by craigmart • today at 10:20 AM • 112 comments • view on HN

Comments

OtherShrezzing • today at 11:35 AM

This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.

There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.

But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.

➕ show 12 replies
girvo • today at 11:01 AM

> With them, you can run dozens or hundreds in parallel.

Im already spending my $2k usd per month in tokens with just one or two agents. How do they expect me to do that?

➕ show 5 replies
koolba • today at 11:53 AM

> Nobody can read every change, so we use failures as one way to decide what needs a person.

Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.

➕ show 3 replies
Narann • today at 11:53 AM

The more I read articles like this, the more I realize Claude is good for process prototyping, but you need to "bake" the process with flat code/script/organization to decrease costs.

The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.

FartyMcFarter • today at 11:56 AM

> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.

If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).

If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.

hliyan • today at 12:05 PM

In my opinion, the output of a stochastic process can only be truly verified through a deterministic process. Tests already do this for functional accuracy. Linters can do this for line-level code, with code treated as text. But there is currently no such method of deterministic verification for the structure of the entire program, or more importantly the viability of the architecture when you throw in connectivity and IOPS.

Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.

How to do this for architecture: unsure.

gregwebs • today at 12:41 PM

This is the main thing that people are missing in their process of using AI. Telling it to use TDD is not enough.

After I created this skill for verification, it completely changed my level of trust with AI coding.

https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...

The overall process that I am using does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.

https://github.com/gregwebs/skills-sdlc/blob/main/skills/imp...

nelox • today at 1:00 PM

We are well on the way to recursive self improvement

nandanadileep29 • today at 11:13 AM

excellent content from the team. very concise and clear

adilbasari • today at 3:05 PM

[flagged]

arbor-group • today at 2:58 PM

[flagged]

makercfd • today at 3:02 PM

[flagged]

i_rutul_patel • today at 2:31 PM

[flagged]

Eunice99x • today at 3:06 PM

[flagged]

runtarmac • today at 11:40 AM

[flagged]

Yaojia666 • today at 12:15 PM

[flagged]

oceanrelay • today at 11:06 AM

[flagged]

yoy0721 • today at 12:01 PM

[flagged]

dotdi • today at 11:20 AM

> https://github.com/anthropics/claude-code/issues has over 16k open issues

this u?

➕ show 1 reply
Ampersander • today at 11:05 AM

Why is the team thinking? Their product should be doing that for them

➕ show 2 replies
lavendermon • today at 2:11 PM

At what point in the future could the title become: "How the Claude Code team's Claude Code thinks about verification"?

saagarjha • today at 1:19 PM

This is just a depressing read, and I’m not even sure why the team decided to post it because it just looks horrible. Like, you’re telling me that the team’s approach to verification is that they post Claudeslop blog posts talking about the most unimaginative ways to write tests? That their idea of making things work is this?

> You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again.

There are people out there who are doing all sorts of crazy neat things in the verification space using the tools you build and yet the people who could be helping seem to have no good ideas on how to help :/

titzer • today at 1:20 PM

I see a lot about adding skills to make Claude test things.

How about making Claude generate scripts that you can run?

Algorithms are better than agents. In fact, the whole point of AI is to reduce thinking to algorithmics. I didn't see a lot of evidence that the team understands this. If you need a magic box to test your other magic box, you have magic boxes all the way down.

whamp • today at 2:16 PM

This whole thing read like google not anthropic.

TheRoque • today at 12:52 PM

I can totally see that they don't review their product at all, Claude code is probably the thrashiest TUI I've ever used.

jordemort • today at 2:13 PM

we think about verification

we don’t actually do any of the work but we think about it

inamorty • today at 11:40 AM

Can they not afford someone with an engineering background?

➕ show 1 reply
matltc • today at 12:12 PM

Formal vibefication

ForHackernews • today at 11:25 AM

Ironic that this page loads partially styled with broken CSS:

"The resource at “https://claude.dev/_next/static/chunks/241gnzjfqbp9f.css” preloaded with link preload was not used within a few seconds. Make sure all attributes of the preload tag are set correctly."

dude250711 • today at 10:49 AM

> ...On the Claude Code team, the vast majority of our code (~90%) is written and reviewed by Claude...

I am more interested in the remaining 10%.

➕ show 1 reply
123ajhgF • today at 12:58 PM

Is the perfect verification the reason that Claude is down so often?

More snake oil and theatrical agent setups described in insufferable Claudish. I think the target group of Claude is people with an IQ below 100.

mococa • today at 11:34 AM

So a lot of slop...

sick_of_slop • today at 1:14 PM

[dead]

ahmedketata • today at 12:05 PM

[dead]

jonenst • today at 11:50 AM

[dead]

mohsen1 • today at 11:20 AM

[flagged]

➕ show 1 reply