logoalt Hacker News

eadwutoday at 5:56 AM1 replyview on HN

Does the latency even matter?

You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.

The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).


Replies

brookman64ktoday at 7:11 AM

I tried TypeSafe’s Jev playground. It outputs the model latency and network latency separately. The model latency was 100-200ms in my tests.