The amount of violence to keep the slavery running was huge. You cant pretend that all that violence does not count. That being said, war was more about south wanting war/leave the union, because the north did not wanted to expand the slavery to new territories. That threated the south.

It is not like north would march in there to stop the slavery. There was an anti slavery army - John Brown with his, like, 20 or so people attacking south.

African Americans as second class citizens were in fact much better off then them being slaves.