What tooling makes this go?
Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
In this case, you need benchmarks, in addition to regular tests. You also need concise instructions (or better yet, following your language / ecosystem norms if applicable) to how to run this stuff
I'm convinced this is why there's a bifurcation in old-hand/new-hand results: the old hands include the steps they'd do and the tools they'd use, and the new-hands never learned those tools.
That'll get RL-d in, eventually.
The ability to run queries and get the metadata about the run, e.g. length of run, execution plan, engine, engine params, etc
The same tooling that you'd use. AI is not magic, it cannot know exactly how a program will run under your exact circumstances and data. Therefore, if it can use a profiler, test suite or application, or be able to read logs, just like you would, it then can make changes, evaluate them, and iterate. Personally, I've created test MCP servers that will perform actions like submit orders, have the AI capture and analyze PerfViews and Concurrency Visualizer, write out tracing data to logs... all the same stuff I was doing by hand for years.