alt Hacker NewsI wish there was a way to send compressed context to LLMs instead of plain text. This will reduce token size, performance & operational costs.
I wish there was a way to send compressed context to LLMs instead of plain text. This will reduce token size, performance & operational costs.
> This will reduce token size, performance & operational costs.
How? The models aren't trained on compressed text tokens nor could they be if I understand it correctly. The models would have to uncompress before running the raw text through the model.