Systems Thinking

  > you lay out a huge specification that would fully work through all of the complexity in advance, then build it.

But I do still think there's a lot of value into coming up with a good plan before jumping in. A lot of software people like to jump in and I see them portray the planning people as trying to figure everything out first. (I wonder if we reinforce the jumping in head first mentality because people figure out you can't plan everything) A good plan helps you prevent changing specs and prepares you for hiccups. It helps by having others but basically all you do is try to think of all the things that could go wrong. Write them down. Triage. If needed, elevate questions to the decision makers. Try a few small scale tests. Then build out. But building out you're always going to find things you didn't see. You can't plan forever because you'll never solve the unknown unknowns until you build, but also good prep makes for smoother processes. It's the reason engineers do the math before they build a bridge. Not because the math is a perfect representation and things won't change (despite common belief, it's not static) but because the plan is cheaper than the build and having a plan allows you to better track changes and helps you determine how off the rails you've gone.

It is also perplexing to me that people think they can just plan everything out and give it to a LLMs. Do you really believe your manager knows everything that needs to be done when they assign jobs to you? Of course not, they couldn't. Half the job is figuring out what the actual requirements are.

> There should be some balanced path in the middle somewhere, but I haven’t stumbled across a formal version of it after all these decades.

That's very simple. The balanced path depends directly on how much of the requirements and assumptions are going to change during the life time of the thing you are building.

Engineering is helpful only to the extent you can forsee the future changes. Anything beyond that requires evolution.

You are able to comment on the complexity of that large company only because you are standing in the future into 50 years from when those things started take shape. If you were designing it 50 years back, you would end up with same complexity.

The nature's answer to it is, consolidate and compact. Everything that falls onto earth gets compacted into a solid rock over time, by a huge pressure of weight. All complexity and features are flattened out. Companies undergo similar dynamics driven by pressures over time, not by big-bang engineering design upfront.

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.” Gall’s Law

> There are two main schools of thought in software development about how to build really big, complicated stuff.

> The most prevalent one, these days, is that you gradually evolve the complexity over time. You start small and keep adding to it.

> The other school is that you lay out a huge specification that would fully work through all of the complexity in advance, then build it.

I think AI will drive an interesting shift in how people build software. We'll see a move toward creating and iterating on specifications rather than implementations themselves.

In a sense, a specification is the most compact definition of your software possible. The knowledge density per "line" is much higher than in any programming language. This makes specifications easier to read, reason about, and iterate on—whether with AI or with peers.

I can imagine open source projects that will revolve entirely around specifications, not implementations. These specs could be discussed, with people contributing thoughts instead of pull requests. The more articulated the idea, the higher its chance of being "merged" into the working specification. For maintainers, reviewing "idea merge requests" and discussing them with AI assistants before updating the spec would be easier than reviewing code.

Specifications could be versioned just like software implementations, with running versions and stable releases. They could include addendums listing platform-specific caveats or library recommendations. With a good spec, developers could build their own tools in any language. One would be able to get a new version of the spec, diff it with the current one and ask AI to implement the difference or discuss what is needed for you personally and what is not. Similarly, It would be easier to "patch" the specification with your own requirements than to modify ready-made software.

Interesting times.

Lots of wisdom in this post about some of the realities of software development.

The core point they're trying to make is that agile (or similar) practices are the incorrect way to approach consolidation of smaller systems into bigger ones when the overall system already works and is very large.

I agree with their assertion that being forced to address difficult problems earlier on in the process results in ultimately better outcomes, but I think it ignores the reality that properly planning a re-write of monumentally sized and already in use system is practically impossible.

It takes a long time (years?) to understand and plan all the essential details, but in the interim the systems you're wanting to rewrite are evolving and some parts of the plan you thought you had completed are no longer correct. In essence, the goal posts keep shifting.

In this light, strangler fig pattern is probably the pragmatic approach for many of these re-writes. It's impossible to understand everything up front, so understand what you reasonably can for now, act on that, deliver something that works and adds value, then rinse and repeat. The problem is that for sufficiently large system, this will take decades and few software architects stick around at a single company long enough to see it through.

A final remark I want to make is that, after only a few years of being a full-time software developer, "writing code" is one of the easiest parts of the job. The hard part is knowing what code needs to be written, this requires skills in effective communication with various people, including other software developers and (probably more importantly) non-technical people who understand how the business processes actually need to work. If you want to be a great software developer, learn how to be good at this.

I find the title misleading. While the author is "thinking about systems", this is not about https://en.wikipedia.org/wiki/Systems_thinking

>In a sense, it is the difference between the way an entrepreneur might approach doing a startup versus how we build modern skyscrapers. Evolution versus Engineering.

There are core differences in software engineering that, unlike in construction work:

- making changes is often cheaper

- we might not know beforehand everything that is needed to be built, especially unknown unknowns

I would still agree that the truth is somewhere in between, but I would argue that, for software, it's closer to the evolutionary approach.

lol this article is like the waterfall manifesto.

Nah I’m good. I’ve watched system architecture framework views be developed. Years of prep and planning. System is released and half the employees that had requirements no longer work there and the business already pivoted to a new industry focus.

There’s a reason we went this way in software development a quarter century ago.

Software is not a skyscraper.

Wow that's a good piece. It propelled me into writing this: https://dev.to/itay-maman/the-elephant-in-the-room-systems-t...

In short: the tension described in "systems thinking" is the same one as the one between "spec driven" vs. "iterative prompting"

> Also, the other side of it is that evolutionary projects are just more fun. I’ve preferred them. You’re not loaded down with all those messy dependencies. Way fewer meetings, so you can just get into the work and see how it goes. Endlessly arguing about fiddly details in a giant spec is draining, made worse if the experience around you is weak.

IMO the problem isn't discussing the spec per se. It's that the spec doesn't talk back the way actual working code does. On a "big upfront design" project, there is a high chance you're spending a lot of time on moot issues and irrelevant features.

Making a good spec is much harder than making working software, because the spec may not be right AND the spec may not describe the right thing.

https://en.wikipedia.org/wiki/Conway%27s_law is very relevant here

Show me an example of a large complex software system built from spec rather than evolved.

The Evolution method outlined also seems born from the Continuous Delivery paradigm that was required for subscription business models. I would argue Engineering is the superior approach as the Lean/Agile methods of production were born from physical engineering projects whose end result was complete. Evolution seems to be even more chaotic because an improper paradigm of 'dev ops' was used instead of organically emerged as one would expect with an evolving method.

Ai assistance would seem to favor the engineering approach as the friction of teams and personalities is reduced in favor of quick feasibility testing and complete planning.

Is this philosophising about software or does it have something to do with real life?

> There are two main schools of thought in software development about how to build really big, complicated stuff.

That feels like a straw man to me. This is not a binary question. For each small design decision you have a choice about how much uncertainty you accept.

There are no "two schools". There is at least a spectrum between two extremes and no real project was ever at either of the very ends of it. Actually, I don't think spectrum is a proper word even because this is not just a single dimension. For example, speed and risk often correlate but they are also somewhat independent and sometimes they anti-correlate.

A major factor supporting evolution over big up-front design is the drift in system requirements over time. Even on large military like projects, apparently there's "discovery"--and the more years that pass, the more requirements change.

Software cannot be built like skyscrapers because the sponsors know about the malleability of the medium and treat it like a lump of clay that by adding water can be shaped to something else.

This missed the point that they are ignoring evolution is literally the way you build things. There is no other way. You don't know what is actually needed or what might work really. You try things and then compress later. If you can try bigger things, bigger leaps great.

[dead]