alt Hacker NewsCool project! I haven't seen that OpenRouter workflow yet (sign into OpenRouter and it creates an API key that your app can use), that looks like an interesting pattern to investigate.
My company recently built a tool that is closer to your first category, but it's an API so it doesn't have the security (supply chain) concern of being embedded in your application.
It's built to help people manage the risk of LLMs changing underneath them and drifting from their designed behavior. Traditional deterministic testing probably won't be sufficient for apps that provide nondeterministic output, like a chatbot backed by an LLM.
The point in the linked article about the challenge of selling developer tools to developers is a good one. I think the first reaction to coding agents is "let's build everything ourselves!" but the long tail of maintenance is still there and the pendulum will probably swing back to "let's stick to our knitting."