What Functional Programmers Get Wrong About Systems

The challenges mentioned on the article are challenges that are known to every larger distributed system. They are not challenges solved or to be solved by functional programming.

But the solutions and tools functional programming provides help to have higher verifiability, fewer side effects and more compile-time checks of code in the deployment unit. That other challenges exists is fully correct. But without a functional approach, you have these challenges plus all others.

So while FP is not a one-size-fits-all solution to distributed systems challenges, it does help solve a subset of the challenges on a single system level.

I went into this expecting it to be a criticism in the context of low-level systems, and was a bit surprised when it ended up being about distributed. The mismatch the author is describing, I think, is not really about functional programmers so much as everyone not working directly on modern distributed or web software. For being upfront about all the ways distributed programming is different, this is actually one of the best intros I have seen for the non-distributed programmer on what distributed programming is really like.

I still want to believe that future programming languages will be capable of tackling these issues, but exploring the design space will require being cognizant of those issues in the first place.

This is really, really good as an overview of system-level issues. I write only to encourage others to read it.

Ideas that helped me situate many of the issues we struggled with at various companies:

- distinguishing programs (the code as designed) from (distributed) systems (the whole mess, including maintenance and migration, as deployed)

- versioning code (reversibly) and data (irreversibly)

- walk-through of software solutions for distributed systems, and their blind spots wrt data versioning; temporal databases

- semantic drift: schema-less change

- Limitations of type systems for managing invariants

- co-evolving code and data; running old code on old data

The real benefit of systems thinking is recognizing whether an issue is unsolvable and/or manageable (with the current approach, or warrants a technology upgrade or directional shift). It's like O(n) evaluation of algorithms, not for space/time but for issue tractability.

> Static types, algebraic data types, making illegal states unrepresentable: the functional programming tradition has developed extraordinary tools for reasoning about programs.

But none of these things are functional programming? This is more the tradition of 'expressive static types' than it is of FP.

What about Lisp, Racket, Scheme, Clojure, Erlang / Elixir...

The version problem is solved well in FP by catching the old function and translating it cleanly. This ofc also works with data representation.

Im not sure what the point of the post is. Minimizing side effects in your code (keeping functions pure) is what gives you the flexibility he says FP misses?!

This is an absolutely wonderful article. I helped build a system very similar to the one quoted below:

> A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts.

that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this.

The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution.

I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type label: it makes a field required in the constructor but not the deserialiser. I would like to see these "asymmetric" types find their ways into IDLs like Protocol Buffers such that their compatibility checkers, like Buf's `buf breaking` (https://buf.build/docs/reference/cli/buf/breaking/), can formally reason about them. I feel like there is a rich design space of asymmetric types that are poorly described (enum values that can only be compared against but never constructed, paired validations, fallback values sent over the wire, etc.)

Another one that I think makes a pretty good attempt is the config language CUE and its type system (https://cuelang.org/docs/concept/the-logic-of-cue/), which allows you to do first-class version compatibility modelling. But I have yet to investigate whether CUE is fit for purpose for modelling compatibility over the wire / between service deploys.

Are there any other libaries / research / etc that sorta take a more functional approach to solving these sort of problems?

For db stuff - what if we flipped the way we write schemas around. A schema is its something that you derive at a current state, rather than starting by writing a schema and then generating migrations? So you can reason over all of it rather than a particular snapshot?

this website is incredibly unfriendly to scroll on firefox mobile. i could read it. it jumps every time i scroll down

With all due respect to Duncan... The reason why this is an issue at all is because the people who architected the internet were not thinking functionally. If they were, they would have come up with something like Nix.

But yes, interfacing between formally proven code and the wild wild West is always an issue. This is similar to when a person from a country with a strong legal system suddenly find themselves subject to the rule of the jungle.

The better question to ask is why did we choose to build a chaotic jungle and how do we get out of it