I understand your sentiment. Unfortunately, the history of America's legal system isn't simple. There are people in prison who never actually committed a crime, but who were convicted because they couldn't afford good legal representation during their trial. This disproportionately affects the poor, and there are correlations between poverty and minority status in America. Some people have been able to get their convictions overturned, but this typically requires very sympathetic people advocating for them.

There's also a very long history in America of laws and law enforcement being targeted against poor people and minorities. Vagrancy laws (https://en.wikipedia.org/wiki/Vagrancy#Post-Civil_War) and modern anti-homeless laws effectively criminalize homelessness, and the War on Drugs has had a major negative impact on poor people and minorities. Yes, in this situation those who have been imprisoned due to such laws did violate the law, but such laws, in my opinion, serve the function of kicking people while they are down rather than addressing the root causes of their poverty.

There's a good argument that having a system of convict labor creates a perverse incentive to fill that labor pipeline, similar to how well-meaning traffic laws (such as speed limits) can be abused (for example, "speed traps").