alt Hacker NewsAnd structured APIs are about 1e9x more expensive than not invoking an LLM in the first place compared to using deterministic code to do something ... it's not like any of this is rational based on compute.
And structured APIs are about 1e9x more expensive than not invoking an LLM in the first place compared to using deterministic code to do something ... it's not like any of this is rational based on compute.
It simply doesn't fit in the token/time budget to be useful. I don't think the purveyors of these technologies care about how expensive it is as long as it's "cheap enough"