I noted the data point of the civil war in Syria: 20 times as many people moved as were killed. That's good news as far as it goes - to run is better than to die - but that doesn't make the Syrian civil war a good time for the people involved. They were running for a reason - the threat of death was too high if they stayed put. So they left. They left their homes, their belongings, their jobs, and ran to a very uncertain future somewhere else.
So I'm not sure that "the death toll wasn't that high" should be casually interpreted as "it wasn't that bad for regular people". Yes, most of them lived. That doesn't make it benign.
(Hmm, I seem to have used a lot of dashes in the first paragraph. No, I'm not an AI.)
A good remark, but also a word of caution.
The Syrian situation is not directly comparable. With modern logistics, it is way, way easier to feed even big masses of displaced people. We can produce food efficiently and we can move it over long distances before it spoils. If you run away from active fighting, chances are that you actually survive, even though the refugee camps are miserable.
Neither was true in the Early Middle Ages and if an invader displaced tens of thousands of people from somewhere, they would just die of hunger. The capacity to take care of sudden crowds of refugees just wasn't there.