alt Hacker NewsAuthor here. Fair critique, thanks.
On the 512 tokens: that's the window of the model we benchmarked with (all-MiniLM-L6-v2), not a claim about embedding models in general. The article does mention text-embedding-3-small's 8,192 window, and the same setup works with 32K models like Qwen3-Embedding. If your documents fit the window, the advice stands: keep truncate.
A bigger window makes chunking easier, not irrelevant. max_tokens defaults to the model's own limit, so with an 8K model you get fewer, larger chunks and overlap matters a lot less.
Two reasons we still chunk even when the document would fit:
1. One vector per document is a summary of the whole thing, so a short, highly relevant section gets averaged away by everything around it. One vector per chunk turns the question into "does this document contain something close to the query?", with the doc scored by its best chunk. Your #include example is exactly that case: the first N tokens of every file look alike, and what distinguishes them is further down. That's the "deep content" split in the benchmark — truncate got 55% recall@5, recursive got 83%.
2. Cost. Transformer embedding time grows superlinearly with input length, so pushing a whole 8K or 32K document through a local model on CPU costs far more than embedding it as 512-token chunks. Remote APIs bill per token either way.
That said, you're right that our numbers only show the effect against a 512 window. We should rerun the same benchmark with an 8K and a 32K model. I'd expect the gap to shrink but not disappear, and that's worth measuring rather than assuming.
On "nothing anywhere told you": agreed, silent truncation is bad behavior. That line describes what Manticore used to do (and what most embedding pipelines still do by default), not a defense of it. truncate is still the default because multi-vector output needs a different column type, so it has to be opt-in.
On "what you already have": fair, that sentence reads badly. It means "the old default, unchanged", not "good enough for you". We'll reword it.
Replies
Just so you know, your comment was automatically hidden (“dead”) until I vouched for it now. Same for most of your recent submissions. Actually it’s probably because of your (exclusively self-promotional) submissions that your comments and submissions get hidden.
Did you really need to use an LLM to write this comment?
"If your documents fit the window, the advice stands: keep truncate." "Two reasons we still chunk even when the document would fit:"
Which advice do you stand by? Obviously, very short content doesn't need chunking, so let's consider a document that fills 75% of the input context.
When chunking, your cost overhead (per token) goes up as the number of new tokens per chunk goes down. That's an argument for longer chunks, although the averaging/smearing point argues for not going too long.
Embedding calculations are effectively prefill: on my cheapo local inference system (32 GB AMD R9700 + 8 GB AMD RX 7600), the older 8 GB card goes about 80% as fast as the bigger card for Qwen3-Embedding-4B (a bit over 19 chunks/second on my usual corpus, blog posts+comments that are mostly well under 32K tokens). So I would suggest that anyone who is limited by CPU embedding models could benefit from even a small local GPU.
For your blog post, I would suggest an explanation of the chunking modes, either in the blog post or as a hyperlink to the docs about them. "truncate" and "sentence" are fairly clear, whereas the others are not. (If "mean" just computes the mean of the embeddings, that seems like a poor choice. The arithmetic at https://www.johndcook.com/blog/2026/09/16/coffee-milk-latte/ might work for single words, but seems likely to break down at the document level. "recursive" and "fixed" are opaque, at least to me.)
If/when I index my team's documents, I will consider a content-aware chunking that fits as many sentences, paragraphs or sections as possible into each chunk, with overlap determined by the level at which the chunk finishes. Content-agnostic chunking is easier to code and more generic, but indexing should respect a document's internal structure.