alt Hacker NewsHi Antoine!
Interesting point about backend variance. Do you think serving layer should become part of standard LLM eval reporting?
Hi Antoine!
Interesting point about backend variance. Do you think serving layer should become part of standard LLM eval reporting?
Hi! Yes, I definitely think so. I've seen variance across all model families I looked at. The magnitude changes, but the presence of variance is a constant.