alt Hacker NewsThe website says the model is for "tool calling, device use, and structured extraction". Your example just doesn't seem to be very relevant. FWIW, it did a pretty good job for tool calling when I tried it, and I think it could be pretty nice to have this running on locally and integrate with Home Assistant.
Replies
curious_cat_163 • today at 1:20 AM
I think the test above is about tool calling... That's how I read it. The issue here is known as "out of distribution detection" in the old-timey classification world.
I am not sure how a micro model will fundamentally solve it. Would love to understand what dannyw and team did there?
➕ show 1 reply
False positives are definitely relevant and worth measuring - natural language interfaces always have a discoverability problem, i.e., users not knowing what actions the system does and does not support. If the frontend of that system lacks the ability to reject unsupported commands, weird stuff happens.
Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evaluate your releases on some usability criteria (including false positives). Any serious integrator or adopter of these models would want to have that information available.