alt Hacker NewsDoes anyone know if the average document length mentioned in the document length normalization is median? It seems like it would need to be to properly deweight excessively long documents, otherwise the excessively long documents would unfairly weight the average, right?
It’s the mean. At least in Lucene. Using median would be an interesting experiment.
Do you know of a search dataset with very large document length differences? MSMarco for example is pretty consistent in length.