There's more egregious cases though that I think illustrate the problem at large: no one wants accountability.
A very famous and egregious example is the XBox user who got banned for listing "Fort Gay" as their place of residence[0]. This is a problem that was caused by automation and honestly, could have entirely been resolved with automation too[1]. But it was also a problem that could have been resolved in under a minute were a human given real power to do anything (or recognize that the cheapest labor usually isn't the cheapest labor).
Another is how there's a family suing Google for directing a man to drive off a bridge[2]. Hold your reservations because this is kinda like the McDonald's Coffee lawsuit[3]. The bridge had collapsed in 2013 and the man drove off in 2022. There's multiple parties that share some fault here (like city for not marking and barricading the bridge[4]), but the issue was reported many times and what kind of live map system isn't updating their maps within a decade?
I frequently report spam, phishing attacks, and all sorts of stuff. Nothing gets through. Same with Google maps. Same with literally any app. I can even send to dev channels with patches and things often do not go through. I can sit on a PR for months while others are asking for a merge and then a dev comes back and says "oh, change color to colour" or something, I'll repatch that night, and then the dev goes radio silent (seriously, it is more work to ask me to make that change than it is to do it yourself...).
I have so many frustrations, but the root of it all is that I can't fix problems I find. Even if I can create the fix myself, I can't get them upstream so I don't have to patch every fucking patch that comes down. I think a lot comes down to our mentality of "move fast and break things." This is fine for learning but not fine for production. Who cleans up all the mess left behind? The debt just grows and compounds. I know mitigating future costs is "invisible" but often we're talking about 15 minutes of work. If you don't have that kind of slack in your system then you're doomed. It's like having exactly the number of lifeboats on a ship such that you can accommodate every passenger. That's dumb. You have to over accommodate. Or else you get the Titanic (which underaccommodated, despite being capable of overaccomodating).
[0] https://kotaku.com/xbox-live-gamer-suspended-for-living-in-f...
[1] Step 1: Check user's location. If they aren't masking it, you'll find that they are located in "Fort Gay". Step 2: If it is masked, plug the fucking location into Google Maps or some database with a list of cities and check for a match. Done. Yay. 30 minutes of programming and you saved the company hundreds of dollars in customer service fees and millions of dollars in reputation rebuilding "fees".
[2] https://www.cnn.com/2023/09/21/us/father-death-google-gps-dr...
[3] https://en.wikipedia.org/wiki/Liebeck_v._McDonald%27s_Restau...
[4] I highly advocate citizen action here. If you live near there, put a pile of rocks or anything in the way to make a barricade. Law comes after you? Fuck the law. Besides, I'm sure it'll make a great news story. We have those for people filling in potholes, this seems much more sensational.
The Scunthorpe effect
XBox Fort Gay was a classic example of the Scunthorpe Problem[0]. I suppose we need a formal Scunthorpe Test, but this seems like you could solve the Problem with a popup checkbox and text field whenever your filter flags an account.
The seminal Falsehoods Programmers Believe About Names[1] looks at similar territory from a different perspective.
[0] https://en.wikipedia.org/wiki/Scunthorpe_problem
[1] https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...