I think it's interesting because LLMs (at least if used in the terminal) quickly create their own scripts to do a certain task.
The issue is then, that those scripts are probably discarded after one use by the user, partly because the LLM doesn't make them generalized enough and focuses on solving the one task at that time. Partly because, when the issue next arises, it seems cheap to the user to just have the LLM do it again.
I think these models could be trained/fine tuned in a way to anticipate future issues or create general enough solutions but it would imply lowering their own use... so there might not be an incentive for that.
It's also because the LLM training loop is too short-lived to give them feedback on how well repurposed their code can be. I imagine the labs are working on this to make it better, but it's fundamentally a much more expensive environment than training on few-script rollouts.
> I think it's interesting because LLMs (at least if used in the terminal) quickly create their own scripts to do a certain task.
Let me just write a small Python script to decompile a section of this DLL so I can verify the constants for this function matches what the reference said.
With the images, I'll write a small Python script to compute the motion error using the Lucas-Kanade method[1].
Almost literally what Claude (Opus) casually has written as it chipped away at a couple of my problems.
[1]: https://en.wikipedia.org/wiki/Lucas%E2%80%93Kanade_method