The Tokio/Rayon Trap and Why Async/Await Fails Concurrency

This reads an awful lot like the prompter had a semi-confusing time with some async, and had their favourite model write an upset blog post about it.

I don't think these systems are perfect, nor are they fit for every use-case, but some of the complaints ring a bit hollow.

> fetching a database record over the network, then immediately crunching the data. But what happens when that data crunching involves parsing a 10MB JSON payload

Mixing IO-sensitive code and blocking code causes issues, who knew? I'm not quite sure how these libraries are supposed to magically _save_ you from this?

> When these latency spikes occur, the answer is always the same: separate your runtimes.

Well yeah. "Why doesn't my daily-driver cut sick lap times around the Nurburgring?" If you want to run blocking, non-interleaved code, don't do it in an executor expecting small, interleaved, non-blocking tasks. The docs for Tokio even mention this, and provide a number of worked examples of integrating/bridging sync and async code.

> If a developer must manually partition I/O and compute, strictly police the boundaries to prevent deadlocks, and ferry data between two different runtimes with two different mental models, the async abstraction has failed.

Not necessarily. If I'm chasing a performance target, and the tool gets me 80-90% of the way there, to the point where my next task is optimising layout and caching, I call that a win. That's performance ground we'd have to address at some point if we want to go faster, so getting there easily means: - those people who don't need to go faster because their perf requirements are met are happy - those people who need to go further get to skip the intervening work, and go straight to these optimisations.

Edit: also wanted to make a point about the "memory blowup" the author mentions. It's entirely possible to do the _same_ load-shedding the author wants in Rust/Tokio/Monoio/etc, this is again, just a matter of building your application in a way that uses, and communicates back-pressure. I gather that they appear to want something which is a bit more opinionated and "batteries included" as far as functionality goes, which is totally fine-and-cool, but calling Tokio + Rayon _bad_ because they explicitly don't do this is a bit of a miss.

<When WhatsApp pushed the Erlang BEAM virtual machine to its limits on 100+ core machines, the system choked. As detailed by Robin Morisset, idle threads trying to steal work spent all their CPU cycles fighting over the global runq_lock5>

A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware.

Recently we built Dip, our in-house ephemeral + parallel database, and we went with may coroutines + work pinning to the same thread which also nicely becomes numa aware via architecture.

Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory (thread stack) used by the massive number of threads.

There are some fundamental assumptions in the Rust async system:

- The program is mostly I/O bound.

- All tasks have equal priority.

If your program isn't like that, the Tokio model is a bad match to the problem.

Real time control is not like that. MMO and metaverse game programs are not like that. Most web stuff is, but that's a special case. A big special case, but a special case.

I see two solid points here:

1. It's not reasonable to expect the application layer to carefully partition its work into "I/O heavy" and "CPU heavy" parts.

2. It's not reasonable to queue up an arbitrary amount of work without back-pressure.

I haven't used Tokio much, but if it falls prey to these pitfalls, it would make me pause before adopting it.

I think there are probably ways of using Rust async that don't fall prey to these. Maybe not so much with network servers (I haven't written that many of those), but models where you are evaluating a graph and have more control over how new work is added to the system.

I just had to rip out a lot of native Swift concurrency, and replace it with classic GCD.

Looks like tasks mess with the reference counter. My app was not releasing memory, and was constantly jetsam-crashing.

Also, the app was quite sluggish.

Reverting to GCD, fixed both these issues.

I was looking forward to checking out Project Tina until I realised it was a completely different language. Classic story. Surely you can build a thread-per-core message passing concurrency framework in Rust, the language is designed to allow such alternatives.

Is Project Tina a bit like the Actor Model, but having actors pinned to cores?

And I don't understand how Tina deals better with the problem of compute-heavy tasks blocking the thread. It looks to me like it is also cooperative concurrency per core, and if one Isolate runs for a long time the other Isolates in that core will not be able to handle their messages.

This is silly or just AI slop post? Because using a Go quote as an example of doing something right in the arena is laughable at best where it has the same problems, more magic, and worse observability.

The punchline seems to be something like the LMAX disruptor style which is genuinely good for some things, but if you have I/O loops like the illustration shows you can easily block that loop with some long running function.. so you have the same cognitive load as managing thread pools or async pools or disruptors..

I can hate Rayon and Tokio as much as the next guy generally I can empathize with problems they cause. But largely either it's a skill issue.

Or you are just trying to squeeze lemonade from stones, ala, they aren't meant to do what you are doing.

Tokio especially is extremely widely used for all kinds of things it doesn't work well for.

Sure I could improve it add or tune some primitives but I am honestly considering writing my own. And so should others.

I feel like we are all too bound in Rust ecosystem suddenly to Tokio and Rayon because we don't want to blame and acknowledge the libraries just don't work for what we want to use it for.

And library authors don't consider these usecases and bug important enough to actually fix it in a ergonomic way.

I am not really sure how much yet another post complaining about async/await that ends with “thread-per-core is the way to go” adds to this discussion. Granted I’m both an Erlang programmer and a big fan of Tokio and Rust’s async/await implementation in general and I think this post and many others like it betray a fundamental misunderstanding of these technologies so I am probably biased.

Rayon optimises for throughout, not latency. If you want to ensure a maximum latency cadence, use something like micropool: https://github.com/DouglasDwyer/micropool.

Here is an interesting video about it: https://youtu.be/QFQkqFSg8Z4.

“human in the loop scheduler” is very funny

The suggested alternative is interesting, I was expecting a Rust library though, but it's an Odin one. What's missing are any sort of benchmarks or proof of the claims that it will scale beyond the issues the article discuss.

I think it's important to understand how we got here and a lot of it has to do with serving network requests or RPCs.

The first Web servers used CGI (Common Gateway Interface). This spawned an entire program (process) per request and had obvious overheads. This led to some optimizations (eg FastCGI, ISAPI/NSAPI) to reduce the overhead. This was the era of Perl scripts being popular.

Then came the model of having a persistent state across requests. Java servlets were a big example of this. Given the cost of exec'ing a process, this was a big deal. But then you've immediately created an environment where multiple threads were accessing the same resource and you could leak resources. There were other variants like CORBA.

Now this was abotu the time of the birth of PHP. PHP was revolutionary because it had a stateless core that allowed shared hosting environments, which were exceptionally cheap for the time (even though they had security issues). But the idea is that you avoided the threading issues of a persistent environment and didn't really have resource leaks because everything got torn down. Of course PHP had other issues. But this was a big deal because things like the initialization of a JVM class loader, for example, was relatively expensive and you had to tune Java servers around performance and STW GC pauses.

None of the above really has anything to do with programming languages other than people learned (or didn't learn) just how hard writing multithreaded code is, something that is true to this day and you absoultely want to avoid it if possible. It is incredibly difficult to get right in an era of cores, threads, different L1/L2 caches, out-of-order proccessing, branch prediction, etc etc etc. And your code may have to run on multiple architectures.

Now Go chose to get around this issue with goroutines and channels. I personally think these are a bad abstraction, particularly because buffered channels are used without understanding the impact (leading to deadlocks), you can have exploding goroutine sttacks and using unbuffered channels is a strictly inferior (IMHO) async/await abstraction.

I actually think that Facebook's Hack has basically the almost perfect async/await. The whole idea of async/await is that you get the benefits of the PHP model of being single-threaded in your own application code and you can tear down your environment when you're done. Any IO goes through async API functions.

Now how does the scheduler manage threads, exhaustion, etc in this environment? Honestly? I have no idea. It just basically works. So maybe the Tokio issue is that the scheduler itself is blocked, which seems to be the case from this article. That does seem like a flaw but a fixable one.

I get the whole colored function criticism but the reality of using Hack to serve HTTP requests is that everything is async anyway so it seems to be a non-issue in practice. You can if you really need to call call an async function from a non-async function with a blocking function but that's not best practice.

I do know that thread pools, particularly multiple thread pools, is not the answer.

I've been thinking similar thoughts recently, in that I am not sure that a function call is the best way to model asynchronicity. Io-uring in particular feels much more suited to a req/res type model, which has the benefit that you can make a single threaded state machine the core of your app - very pleasant to test and reason about.

I'd draw the analogy to RPC; it's a leaky abstraction because HTTP is fundamentally a different thing than a function call. I'd argue that event loops are a different thing as well.

In a cooperative runtime like Rust’s Tokio or Node.js, the thread does not yield until it hits an await point.

This is just because JS is single threaded. Python has the Global Interpreter Lock, which makes it effectively single threaded too. That means you don’t have to deal with true parallelism, and critical sections, semaphores etc. It’s like Ethereum: only one thing happens at a time.

But you don’t have to parse JSON without yielding. You can make anything async by just using setTimeout once in a while. Here is one such implementation: https://www.npmjs.com/package/yieldable-json

The guy may as well have said while(1) locks up Node.

Now they get into multithreaded work-stealing, and isolates. But the solution in Node is to spin up multiple processes and pass messages between them. This is approaching the Erlang actor model, and is also shared-nothing. They even say "the schedule is a single-threaded loop per core" and "all cross-core communication occurs via the messaging subsystem".

This can also be achieved in most other single-threaded languages, too. Python with asyncio, for instance.

Tina may provide a nice opinionated implementation with bounded queues and deterministic scheduling, but those are architectural choices rather than evidence that async/await itself has failed.

Node has isolates, but it's more for sandboxing.

This is nonsense. Tokio was built for I/O, not crunching numbers. Most programmers know to use spawn_blocking to crunch 10MB JSON, if needed.

Additionally, the proposed workload per thread model will be orders of magnitude slower than Tokio for I/O-bound workloads, which most applications are.