It might be helpful to be a bit more specific about what's being automated. One story from the underlying report (https://assets.publishing.service.gov.uk/media/687f6f9dfdc19...):
> In another example, a Vietnamese national was initially given the benefit of the doubt at the first triage that took place in the waiting area. The CIO and social worker commented on his “soft face”, which they said was consistent with his claimed age of 17. However, his “developed shoulders” and “huge hands” cast doubt for them, as did a “tiny bit of stubble” that they noticed when they asked him to raise his chin. The CIO and social worker told inspectors afterwards that Vietnamese young people were typically difficult to assess because they “did not have the same ageing process”, and “did not show signs of ageing”. When asked where the evidence for this was, they said that it was knowledge gained through their own experience. The social worker said, “It is just genetics”, but was unable to support this with evidence.
If I had to choose between being judged by an AI model and being judged based on ad hoc stereotypes of what my race's shoulders and hands typically look like, I'd definitely pick the AI.
Isn't the AI going to use the same ad hoc stereotypes?