Faster Retrieval for Better, More Efficient RAG

Just yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.

From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).

[1] https://research.atspotify.com/2025/9/you-say-search-i-say-r...

Apple has a fascinating history of waiting and seeing how a market develops before entering with a better product and eventually dominating, the iPod and iPhone are two examples. It seems they may be building a strong position to pull that off again in the personal assistant AI category ChatGPT is trying to dominate.

I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.

I feel like retrieval approaches, despite how good your ranking mechanisms are, end up biasing LLMs into what a heuristics-based system with way worse understanding of the domain says. I guess is a trade off between speed and quality, but letting the LLM explore without a bias (based on my personal experience) seems to generally provide much better results in terms of quality for the final output.

as a recovering RAG-enthusiast, I will share a quote from a peer that has really stuck with me:

"why not use grep?"

Can't wait to see how they leverage this research to make Siri even worse!