alt Hacker NewsGood benchmarks!
> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.
Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.
I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.
Source: https://research.google/blog/turboquant-redefining-ai-effici...
in my experience it's very similar to what you typically need to do in order to get the best results in other fields (like audio), where being really particular/deliberate in how you manicure and format the input can pay orders-of-magnitude dividends
sometimes adding noise over a frequency range is better than removing it entirely, the opposite tends to be true for text and especially code, where you'll want lists of language and framework keywords, and then a pass on top of that to scan the codebase itself for its own slang. You can then 'double dip' and use these lists to weight the results after the fact