alt Hacker NewsI've seen this happen in a wide range of production environments (both industrial and computing). Not this effect specifically, but "odd emergent behavior that occurs only at scale that is non-obvious and state-dependent". For example I work at a company that grows a lot of cells is massive reactors, and some folks who run the largest reactors commented that they saw slow changes in overall production that were not explainable by any observed variable (we speculated that slow genetic drift occurred in populations, but it may also have been seasonal, or due to unobserved variables). And when I worked at Google, there were definitely cluster-wide things that you'd only notice if you were very knowledgeable and attuned to their ongoing processes.
My guess is that this happens in nearly all large-scale production systems but goes mostly unobserved.
Replies
I think the disappearing polymorph stories are also pretty spooky. These have real-life impacts, like with ritonavir.
From experience with large scale clusters, yeah. Weird stuff happens. But it's very hard to setup a test cluster that is actually representative, and you can only do so much on a live cluster. Occasionally, I have been able to find explanations for some of the weird behavior, but usually it's like here's a bug in Linux packet forwarding that was fixed in Linus's tree 15 years ago, but apparently has never been deployed to some router, so it's just going to keep aggregating input packets because large receive offload, and then drop them with needs frag because the aggregated packet is too big to forward. sigh (that's not exactly a cluster scale issue, but it's the most relatable example of an investigation that comes to mind)
You're pretty unlikely to get academic papers when the required setup involves having 100M+ clients geographically dispersed. And it's going to be very hard for peers to reproduce your findings.
In case of this "invisible electrostatic wall", there were likely significant amount of people in that company who were at least somewhat into Star Trek[0], so I'd expect more than mere "meh, this happens" from people who had just seem to have accidentally invented a force field. It's not merely a weird emergent behavior, it's a behavior closely resembling a sci-fi technology, and therefore likely to have similar applications - so quite obviously a potential money and fame printer.
-- [0] - Which was well-known around the time of that event, and at its peak of popularity when the report in the article was filed!
It reminds me of that experiment where they had an audience of 1000 focus their attention upon a chair on a stage.
> My guess is that this happens in nearly all large-scale production systems but goes mostly unobserved.
Not unobserved. Unremarked maybe? It's expected behavior that leads us into personification of systems e.g. calling ships 'she' or talking about temperament between similar machines on a line.