alt Hacker NewsI would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like:
- Spend most time prioritizing/discussing what to do.
- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)
- Use Opus 5 or Sol Med to execute
- Auto-fix bugs and CI until green + thermonuclear review skill x3.
- Manual interrogation of change/nits
- Come up with QA plan and have Codex Computer Use execute on it
- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)
I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.
Replies
IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targeted surgical changes is far more ”productive” in the long term IMO.
This is very close to my workflow but you forgot one important step:
- Suggest a better approach that makes the AI say, “That’s much simpler. And you’re right. My original plan was over-engineered.”
My experience is that your description works for a certain time, since you're knowledgeable of the codebase and can guide it. But after too many iterations with not hand-holding the llm, it quickly gets unwieldy.
The output yes, but do you produce the impact and value of 3 engineers? I have seen this workflow being toyed with too, and I find it to produce massively overengineered stuff that actual people don't really wanna use
>Auto-fix bugs and CI until green + thermonuclear review skill x3.
Gotta love this loop, I have it running while I'm asleep all the time.
Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's not very good at testing in an organized way to even notice it made it worse despite repeated prompts to do so I mean if I prompt it to test performance in a handheld structured way (it is very bad at finding out what performance to test and why) before making changes I can usually figure it out but it usually takes insistence on the specifics to really ensure a good solution that will actually fix the problem
Do you have tips for generating clean productive output per dollar?
> essentially unlimited AI spend budget
> I probably spend like $80 a day
This doesn’t sound like “unlimited”, I spend more than this out of pocket per day and I have a strict budget.
but I produce the output of 3 or 4 2022 engineers and probably at better quality.
Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.
[dead]
In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered.
When I personally want tight code, I have to spend a considerable amount of time adjusting it manually:
- It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating
- I need to consider whether there are solutions based on higher-level assumptions, that AIs typically miss
- I need to check whether there are off-the-shelf solutions - AIs like to reinvent the wheel
IMO, software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.
(1) https://simpsons.fandom.com/wiki/The_Homer