I think calling the Norman invasion French was a simplification for children which outgrew its intended audience. They did absorb some of the French (Flemish) aristocracy by marriage and alliance though, and the Flemish played a large part of the colonization of Britain. I've also heard people refer to the subsequent invasions of Ireland and Wales as 'English'.