I work on software that Dr and Nurses use. The company I work for trains its own models to provide automated insight on patient history, etc. I've been pretty concerned about these kinds of problems you describe. I sat down with one of the data scientists and talked about biases a while ago. She walked me through the (high level) process they go through to try and detect common biases in the model before publishing.

For example, it is a common belief within the medical community that Black people have higher pain tolerance, and thus don't need as much or as strong of pain killers. People in pain sitting in beds in the ER will be delayed or refused pain killers because of this. This belief has been proven false, but it shows up in papers, notes, etc. and nurses and doctors still will treat their patients as if it's true.

This is just one example of many many things they test for. When training the model the data-science team actually tries to see if they can produce output that exhibit those identified biases/problems and block it from ever been used in production if they do.

I've had people internally tell me how frustrated they are that I or other people aren't letting them get their new-fangled AI-powered feature in front of customers fast enough... Without a lot of discipline and guardrails people would be using ChatGPT or Claude APIs instead.

My point is: I highly doubt that kind of care is being taken in many industries/companies. It's very concerning to me.

Yes, AI like organizations are very good for laundering accountability. Even before AI you could just hire Organization X or Consultancy Y and they can be as racist as they want, and boom you're just laundered all your accountability. Now go have fun, do whatever, who cares.

It's the same reason why Flock is so popular. It's just cameras, who cares right? Well Flock is popular for the organization, it's an accountability sink. The government can't just stand up a bunch of cameras, people will complain. Ahh, but Flock can. The entire economic value of Flock is purely the accountability laundering, the technology is just a footnote.

Really, many organizations in the US boil down to this. Producing value is hard, producing fake value is easy.

You don't even need there to be any kernel of reality statistically.

If a model is trained on data that shows some bias due to prior human decisions, it would continue to follow those biases while being a lot harder to prove/sue. There were examples of this with AI assisted resume screening & criminal sentencing for example.

In regulated industries like money lending, ML outputs have to be explainable (and explained for an ECOA case). I’m not an ML expert by any stretch, but IIUC this is why we end up using decision tree models.

On the contrary, you can feed the model counterfactuals and see what it does. Edit the person to be white then feed the image back into the model and see if its opinion changes.

>Some criminal behaviors are somewhat correlated with skin color, age or gender.

Well, yes, because so is crime.

Have you ever seen those AI videos where countries are competing in a farcical olympic event? Say, longest cold shower or something. And the Indian will just quit and fall to the ground and yell "I NEED CURRY!!!!!!!!"