and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways.

why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever)