Contrary to what most people seem to think about the past, slavery was oft seen as naturally repulsive even thousands of years back. It required regular defense. In Aristotle's Politics [1], written some 2400+ years ago, he felt compelled to lay out just such a defense and it was, by far, his weakest argument. He clearly started at his conclusion and worked backwards from there, instead of working forward from first principles and he did in other topics. The reason it's relevant is that he did accurately predict the end of slavery:

"For if every instrument could accomplish its own work, obeying or anticipating the will of others, like the statues of Daedalus, or the tripods of Hephaestus, which, says the poet, 'Of their own accord entered the assembly of the Gods.' If, in like manner, the shuttle would weave and the plectrum touch the lyre without a hand to guide them, chief workmen would not want servants, nor masters slaves."

There were thousands of years of efforts to end slavery, some countries would occasionally succeed at such only to see it spring right back. Yet following the industrial revolution it began rapidly disappearing everywhere that had gone through industrialization + urbanization. The issue in your mental model is that you're only considering local effects over very immediate time frames. Think about the bigger picture.

Industrialization drove big money away from farms and into factories, away from rural scarcely populated rural areas into densely populated urban areas packed with very poor potential workers. As soon as the necessity argument for slavery became plainly absurd, to say nothing of the issue of industrialization also reducing the need for so many workers even on plantations, slavery wasn't long for this world. This says nothing about actual slave holders who, as you said, did not just go quietly into the night. But as their economic might relatively waned, so did their influence.

[1] - https://classics.mit.edu/Aristotle/politics.mb.txt