alt Hacker NewsSol 6.1 is incredibly incredibly slow, 20t/s slow (vs the ~70-80 of Terra 6.1 and Luna 6). It wouldn't be so much slower if it was just a distillation, surely? There's something else going on there.
Replies
willy_k • yesterday at 8:50 PM
Is it using comparable tokens per task?
lnkdinsuxs • yesterday at 5:50 PM
> Sol 6.1 is incredibly incredibly slow, 20t/s slow
How are you getting these numbers?
➕ show 1 reply
lostmsu • yesterday at 5:03 PM
But is it slower than Astra?
There was some (admittedly flimsy) evidence from a microsoft model description card that it's doing multiple passes through its own weights before token generation to improve output; it might be an architecture change that's making it slow, moreso than raw size