alt Hacker News> I can imagine a future in which some or even most software is developed by witches, who construct elaborate summoning environments, repeat special incantations (“ALWAYS run the tests!”), and invoke LLM daemons who write software on their behalf.
This sort of prompting is only necessary now because LLMs are janky and new. I might have written this in 2025, but now LLMs are capable of saying "wait, that approach clearly isn't working, let's try something else," running the code again, and revising their results.
There's still a little jankiness but I have confidence LLMs will just get better and better at metacognitive tasks.
UPDATE: At this very moment, I'm using a coding agent at work and reading its output. It's saying things like:
> Ah! The command in README.md has specific flags! I ran: <internal command>. Without these flags! I missed that. I should have checked README.md again or remembered it better. The user just viewed it, maybe to remind me or themselves. But let's first see what the background task reported. Maybe it failed because I missed the flags, or passed because the user got access and defaults worked.
AI is already developing better metacognition.
Replies
[dead]
I'm concerned that developing better metacognition is really just throwing more finite resources at the problem. We surely don't have unlimited compute, or unlimited (V)RAM, and so there must be a wall here. If it could be demonstrated that this improved metacognition was coming without associated increases in resource utilization, I would find these improvements to be much more convincing... but as things stand, we're very much not there.
(There may be an argument here re: the move from dense to MoE models, but all research I am aware of suggests that MoE models are not dramatically more efficient than dense models - i.e., active parameter count is not the overriding factor, and total parameter count is still extremely important, though it does seem to roughly follow a power law.)