logoalt Hacker News

codeulike • today at 7:22 PM • 3 replies • view on HN

What do they mean by "low frequency n-grams"


Replies

ethanj8011 • today at 7:28 PM

A combination of words that are infrequently observed in natural language. For example in the preceding sentence some of the 3-grams are "combination of words", "infrequently observed in", and "words that are". N-grams are just this for all values of n. They are saying that often times the exact sequence of a short fragment of text is not very common in human written text.

rustyhancock • today at 7:26 PM

Rarely used N length chain of words.

I.e. a high frequency 3-gram is "by the way".

A low frequency 3-gram is "elephants speak french".

➕ show 1 reply
antonvs • today at 7:48 PM

Unusual word combinations, basically. They’re word combinations that are rare in the training data, i.e. rare in normal language, but used more frequently by AI.

They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.

Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.