alt Hacker NewsThe technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken, the arguments passed to it were wrong, or it's a newer/older version of the tool etc => more tokens. Sounds like just adding to the prompt to use `grep` and `tail` extensively will do the trick without any special tooling.
This was the problem I saw. I installed rtk when it came out and liked the idea of it. But over time with newer model generations I kept seeing the model get confused in the reasoning text and retry a command bypassing rtk. I didn't even need a benchmark to see it was regularly an impediment to the final outcome.