alt Hacker NewsIs my intuition correct in that the headline tok/s are for a single user?
Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users?
The chart of the actual tok/s vs concurrency is at the end. For a single user you get 107tok/s, which is about 2-3x as fast as you would get on 4x DGX spark.
The 607 tok/s is total tok/s across the 16 users, so a bit over 35tok/s per user.