We need more goat sacrifices

Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being? Either way, we know more about what's likely to work for future interventions. It's the scientific method.

These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?

Edit: I wrote the above before reaching the end of the article. By the end, the article reaches a similar conclusion, albiet with less science and more mysticism.

Maybe that means it's a good article, leading me to reach its conclusion before taking the stance itself. On the other hand, the article's leading sections gave such a strong impression that the author thought something needs to be drastically changed about the current system — the opposite stance from its conclusion — on seemingly shakey grounds, that I almost didn't finish reading it. I'm not sure what to think, aside from please leave innocent goats alone.

One very successful social intervention in the UK was the push to reduce drink-driving. Starting around the 80s I think, maybe earlier. There was a big media campaign with TV ads as well as implementing and enforcing stricter laws. What I find interesting is that many in the older generation still found it more acceptable to have a few drinks and drive, whereas those growing up during this period largely find it very socially unacceptable.

> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.

> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4

> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5

Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).

The article talks about measurement of impact as though it's a totally value-neutral idea, but it's absolutely not:

  * Measurement requires substantial logistical overhead; both up-front in creating a plan for measurement, and during intervention due to the extra record keeping involved. This is a (probably frequently large) drain on program efficacy. Just listen to any schoolteacher's take on standardized testing, evaluation rubrics, etc.
  * Some contexts, and some types of intervention are more amenable to measurement than others. Insisting that nothing which is not measured can be valuable biases our decision-making towards the most measurable actions, which are likely not the best. Strong correlation with simplistic actions, short timelines, and lack of responsiveness to individual context.
  * Goodhart's law.

this reminds me of the thought that a country run by scientists and engineers should be better. but there's none, and i think this can easily be understood by Daniel Kahneman's fast/slow thinking.

in other words, government is time-constrained, so when a problem is sufficiently complex, there's just not enough time to really flesh out a good solution.

also, humans can be selfish and will occasionally try to game the system, which changes your problem into an active adversary. and when you have active adversaries, you're playing a completely different game (indeed, when designing the policy in the first place you may not have even considered it a game with opponents!).

tough article. 90% is a tongue in cheek ridicule, with lots of data points, and then the bait and switch at the end, with far less argumentation.

mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.

maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.

fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.

> When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.

The author is assuming good faith on behalf of the Arkansas state government that the purpose of the requirement was to increase employment, rather than merely cutting down on the number of citizens successfully receiving Medicaid.

> Plain packaging — where you strip cigarette boxes of their attractive branding — seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more

This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.

Lots of examples in this article, but none of them go deep enough to matter. Just like people bringing up the cigarette issue, I see other factors in sone of the other topics, such as medical school (residency slots weren't mentioned, default assumption was motivators were solely economic, etc).

I'm not sure what to even discuss about this article due to the low quality. I guess there overall point is that policies and interventions don't necessarily scale, or perform the same across cultures/cohorts, or even accomplish their intended goals. I would largely agree with those points, but not due to the way the article got there.

In most of the examples given the aim is to be seen to be doing something. It is cheaper and/or avoids unwanted change than doing something that actually works.

For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.

> It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?

Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.

Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?

Seems like every social program should have a verifiable measure of success and a termination clause if the measure indicates that it isn't working.

But as software people we all know that turning a measure into a target yields problems.

This is a strange reaction to failed predictions. Very defeatist.

There's good evidence that we can improve accuracy on the predicted outcomes of policies, e.g. delphi method or prediction markets. See the "replication markets".

Interestingly, we do have quite a lot of material and studies on how to change human behavior and a whole lot of data supporting it: it's called marketing. I think this goes to show that when the entire economy is working on a problem we actually can make an impact, it's just unfortunately all going towards projects that make people poorer and more depressed.

So the proper way to do public policy is actually run interventions like clinical trials before allocating more funds. But this runs counter to political reality where politicians need to signal to constituents over realtively short time scales (2-4 years). In effect, the policy is the experiment.

It's unrealistic to run pilots, data collection, and followup in 4 years for many political decisions.

The best policy recommendation here is simply to enforce data collection for every decision. This would also incentivize politicans to only propose policies they truely believe will work.

Money will still be burned, but at the very least over long term our understanding will grow, and bad policies will gradually be disincentivized.

ive never had a presentation go well if i didnt place a "sacrificial slide" in the deck, or plan an "oops here is a bug, let me recover, and we're back on track" during the demo

it's the surprising surprises that get you

5 years later...

WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.

The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.

Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.

Problem: Graffiti

Solution:

Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.

Graffiti will be dead within ten years. :-)

> When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.

That... that's not a failure, that's exactly the point of this requirement

I realize the title is clickbait. Still, my immediate reaction is, “Yes, because that worked out so well for Haiti.”

I have no comment on the content, but man is it refreshing to read such a well-written article

I'm getting Malcolm Gladwell vibes.

I think it's quite funny how the "unintended consequences" people love to highlight every unintended consequence or ineffective intervention, but also have zero interest in finding out why the interventions did not work out as planned. Wouldn't this be a great starting point for scientific research?

If I'm investigating a bug and find that the program is still misbehaving after my fix, my usual reaction is to dig deeper, find out what I missed or how my fix changed the program's execution. My reaction would not be "eh, better not fix anything at all".

> Most of those are left-coded failures, so here are some right-coded failures:

"I'm neither left nor right, I'm just an apolitical neoliberal..."

Love the title, made me laugh :)

Viktor Pelevin has a book where Russian politicians sacrifice animals at oil wells to ensure Russia's continued prosperity.

> The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.

> For example:

> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.

> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.

> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1

> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2

> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.

Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:

* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems

* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority

* the Taliban. I hope I don't need to explain that one further.

* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others

I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.

[dead]

I just looked at the study for one claim "might be a fluke, but it also showed up in a study where researchers wiped out a bunch of people’s medical debt and it maybe made them more depressed".

The paper's abstract says "First, we find no impact of debt relief on credit access, utilization, and financial distress on average. [...] Third, we find no effect of medical debt relief on mental health on average, with detrimental effects for some groups in pre-registered heterogeneity analysis." which is exactly the kind of meaningless result you'd expect from doing ~20 statistical tests over random numbers. (The paper lists hundreds of p-values.)

Medical debt with a face value of $169 million was wiped out for 83,401 people, yet the study failed to find any benefit. I suspect this is representative of the quality of the reasoning in the rest of this article.

My Hindu friend sacrificed a gourd instead of a goat for the last full moon.