alt Hacker NewsI'm working on something from a crappy laptop, those numbers from jev can totally be matched:
Local Latency: 0.1813 secondsI'm working on something from a crappy laptop, those numbers from jev can totally be matched:
Local Latency: 0.1813 seconds
and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways.
why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever)