I find things like that hard to perfectly square with observations like the Flynn Effect (“the substantial and long-sustained increase in both fluid and crystallized intelligence test scores that were measured in many parts of the world over the 20th century”): https://en.wikipedia.org/wiki/Flynn_effect
Firstly, this is completely orthogonal. But it's also improper reasoning.
If Neanderthal had bigger brains (they did) or had different cognitive abilities, there's a chance they were baseline smarter than homo sapiens at the time.
Being perhaps a little smarter doesn't mean you win the evolutionary game. There are so many factors at play.
Why? Draw the line backwards, and in a couple of decades you are down at 0 IQ. That's clearly absurd, you can't draw any conclusions of IQ significantly before 1950 from how the line behaves after 1950.