logoalt Hacker News

xnickbtoday at 4:08 PM4 repliesview on HN

which part of that is engineering exactly?

Not trying to be snarky. I genuinely don't get it


Replies

thornewolftoday at 4:18 PM

Write a prompt, evaluate the prompt, understand that is succeeds 95% of the time.

Write a new prompt, evaluate, it now succeeds 99% of the time. Measure what changes between prompt #1 and prompt #2, understand what contributed to the performance jump.

Write a third prompt, this one succeeds 100% of the time. Increase the size of your evaluation set, find a 1/5000 error-class and a 1/10000 error-class, add some explicit code to correct for this cases.

Roll out to production, collecting usage metrics. You make some tweaks to your harness, your prompts. Eventually you have confidence that your system has fewer mistakes than 1 in 100k.

Now, multiply this iteration across all your different prompts and different ways that they might interact with one another.

show 3 replies
visargatoday at 4:15 PM

It might not be apparent from the start what are the best demands to put inside a skill, you can only know by evals. There are whole papers dedicated to changing a few details in a coding harness. https://arxiv.org/abs/2609.20519

rsalustoday at 4:13 PM

the evals? setting those up and empirically proving them is genuinely a lot of work.

CamperBob2today at 4:36 PM

Engineering is the use of mathematics to turn science into technology. Statistics is mathematics, comp sci is science, and technology is the end product.