> but it's so utterly unlike how any other website works that unless you take a huge amount of effort to explain this to people, gently, then it's not at all surprising that people just bounce off of it.
It is, and that's fine.
It's incompatible with a site becoming large; and that, too, is fine.
A large site was not required to accomplish any of Atwood or Spolsky's goals aside from perhaps making a profit off of ads. Many other choices were also counterproductive to the goals aside from profit.
Everyone involved would have loved to "explain this to people gently". There was no space provided in which to do it. We had no access to modify the help pages or do anything about user onboarding flows. Downvotes were the tool provided to de-emphasize things and take them out of the way, and to rate quality; so they were used appropriately. (Despite this, and despite public perception, voting was historically very biased in the positive direction.)
Because new questions were open by default, they had to be closed explicitly and in public in order for them to be dealt with. People walked away with the completely incorrect impression that closing a question is inherently a rejection; the explicit purpose and sole effect of closing a question is to prevent answers unless and until it is fixed. It is done with an explicit expectation that fixable questions will get fixed; but the nature of some problems (in particular: missing information) is such that only the OP can do this. There was a period of time in which "closed" was rebranded as "on hold" in the interface, to try to impress upon people that no, they aren't being turned away, they are being asked to take further action. But this was reverted because it was somehow apparently making things even worse.
But also, yes, after telling countless people the same things daily, sometimes humans lose their ability to be gentle. But nobody who's dealt with roughly ever gets to see what the other side had to deal with earlier. The way that people talk about "toxicity" on Stack Overflow comes across like they think it was constantly full of insults and slurs. The evidence shows that the large majority of that sort of content (which explicitly violated the Code of Conduct, and there was a full system in place to deal with this, that pundits often ignore or even lie about) was directed at the curators. One of the most common questions on the meta site is "why aren't people required to explain downvotes?", with the main retort being "why don't you think they should be required to explain upvotes?". But the main answer to that question is because people who stuck out their necks to try to explain the site's standards politely, would frequently get sworn at and even threatened. (I was once openly called sexist for applying these standards, with no evidence beyond the fact that the OP was a woman, which I didn't know at the time).
I'm criticizing the design of the site (and to some extent the rules that were set up) more so than the moderators trying to operate within it. I'm aware of how thankless a job moderating can be, and how a Q&A site can quickly die under a mass of help vampires who will just suck up endless effort if you let them. I think a lot of the site's design really played into it, and realistically it was probably something that needed to evolve as it grew and matured, because the dynamics really change at different stages.
> the explicit purpose and sole effect of closing a question is to prevent answers unless and until it is fixed
This is a decision I would question. What is it about a bad question that makes answers undesirable until the question itself is fixed? Especially when the judgement of such badness might be faulty (even if it's encouraged to bias towards leaving it open). Again, the answer may be the moderators doing the best with the tools their given (e.g. lacking good ways to otherwise surface the better questions to potential answers), but this is exactly the kind of issue I think tends to underlie SO's problems.
> The way that people talk about "toxicity" on Stack Overflow comes across like they think it was constantly full of insults and slurs
I don't get that impression. The perceived toxicity comes from a somewhat inscrutable set of rules and also apparently inconsistent enforcement which created a lot of frustration with the site. None of this requires that the moderation is even impolite. In fact, people can get more frustrated when a result they consider wildly unreasonable is delivered with a cordial tone. The trope of a saccharine sweet but utterly evil bureaucrat exists for a reason, and I think that is the kind of impression people who describe SO as toxic are describing, whatever the underlying cause (which is almost certainly not that a bunch of people spending their spare time moderating a site are just evil for the sake of it).
(And it's not unique to SO. Wikipedia has a lot of the same problems, though with different causes and attempts to solve them)