alt Hacker NewsTokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
Providers amortize the compute across a batch.
If yours is the only request in the batch it will cost them one full pass through the model.
If yours is one of 1024 inputs in the batch the per token cost is 1024x less.