alt Hacker NewsChunks can only be as large as the embedding model’s token limit, about 512-1024 tokens usually. Anything longer gets truncated.
Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.
(assuming English text)
512 tokens ≈ 350–400 words ≈ a long paragraph or two. 1024 tokens ≈ 700–800 words ≈ about a page and a half to two pages.
I would be very surprised if that amount of text is not enough to encode a general topic into the embedding (otherwise, what would be the whole point of them?).
So if there is a meaningful reference in C referring to A (assuming A-B-C consecutive 1-2 paragraphs), I would expect that the content of the two at least superficially resemble each other, and a vector query for one would return both. (And also, if I am including A in the context after retrieval, then I better give some context before-after as well -- and references tend to be local).
But feel free to prove me wrong, I'm mostly just guessing, not even an educated (in the given topic) guess here.