logoalt Hacker News

janalsncmyesterday at 9:19 PM0 repliesview on HN

Providers amortize the compute across a batch.

If yours is the only request in the batch it will cost them one full pass through the model.

If yours is one of 1024 inputs in the batch the per token cost is 1024x less.