alt Hacker NewsThere is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends
Replies
LPisGood • yesterday at 8:28 PM
You can know what the user has installed if the OS developer offers something.
Knowing what's installed would have to be an OS API. If LLMs provide a standard API surface to the OS, likely including metadata related to feature support.