alt Hacker NewsIf you are using an autoregressive decoder (which glm is) it is not “jev-like”. You lose all of the speed advantages that Jev has.
If you are using an autoregressive decoder (which glm is) it is not “jev-like”. You lose all of the speed advantages that Jev has.
> We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model.
> As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode.
turns out there is a trick to keeping the context filled and only evaluating a handful of choice tokens https://www.youtube.com/watch?v=bcGO7xre46o