Also to note: (Not a fault of PGVector again just a limit of our algorithmic knowledge) PGVector does HSNW or IVFlat indices ... (there is nothing better persistent) however it breaks down with high latency at LARGE amounts of vectors ( 100MIO+ ) that seems like a high ceiling, but when designing production RAG systems, you tend to do per chunk embeddings, or even visual patch embeddings... e.g one page of a document becomes 1024 vectors in itself (for visual patch embeddings ) ... so you hit those limits at 100000 pages already.. something larger organizations definitly have.
I would keep a per-document summary, then dive down only into the filtered set. This is more production grade than selecting from billions of chunks.