Burning weights onto a chip in an efficient way and exposing that via USB would be acceptable for a good enough model tbh
This is pretty close to what Taalas is doing.
alt Hacker News
This is pretty close to what Taalas is doing.