I don't think refusal is the right approach. I would much prefer that it respond with something like:

> There is not enough information to make an accurate estimate, but if you'd like, I can take a stab at it. If so, how much effort to put into it?

> Yes, go ahead and spend up to 5mins and $1 to analyze it.

> Done, I've had 100 subagents analyze the image and have arrived at a 95% confidence interval of the portion containing ...

Indeed, I think any reasonable human might say “A few hundred calories but without measuring the ingredients I might be way off”. I think LLMs could get there, I don’t see anything stopping that. Though they have been notoriously bad at this so far.