It's a distasteful, but relevant, aspect of vikings that they were slavers as well as raiders. If you went viking, a large part of the booty you brought back walked on two legs and had genes to pass on. Perhaps the Norse liked their neighbours just enough not to make many of them "visitors".
doubtful given histories of the area, however maybe they disliked getting retaliation for a raid.
I think the explanation is much simpler, we know the Norse were a bit afraid of the Sami. They viewed them as a weird non-threatening neighbour people who had a weird language and weird magic. So you traded with them, you respected them, you said please and thank you, and then you were happy to see them gone because you didn't want them to curse you. (And I would assume the Sami were very happy to foster this belief since they were much weaker militarily)
Unlike the fat and rich continental Europeans that the Norse viewed as ripe for plunder, they did not fear them at all.