alt Hacker NewsSearch has moved on from tf.idf + pagerank at this point. And absolutely they do use things like encoders to produce embeddings that are used in retrieval.
Search has moved on from tf.idf + pagerank at this point. And absolutely they do use things like encoders to produce embeddings that are used in retrieval.
It's not about whether it moves on or not.
Google ranked the best results, not the most commonly occurring ones.
AI is by default returning the highest statistical percentage of what it thinks you are looking for in a response, which may or may not be a search result.
A fun example is having the AI run a search, and comparing it to perplexity, or even gemini a little. OpenAI is improving, and Claude is pretty consistently not made for comprehensive search.