Most async Rust programs rest on Tokio, which has earned a reputation for being difficult to debug and benchmark. Ahead of RustConf, Russell published a blog post with general principles for building fast Tokio applications, and the post is currently being discussed on Hacker News.
Russell has put together an early version of a document meant to grow into a collection of best practices. He has asked readers to raise concerns or make direct changes through issues or pull requests. He intends to include a working app soon, so people can see how the dial9 trace matches up with the guidelines.
The Hard-and-Fast Rules
Tokio runtimes don’t come with many fixed rules for building fast workloads, Russell points out. The response to so many questions is “it depends,” since how well a workload runs depends entirely on what other tasks are also running on the runtime at that very moment.
That’s why so many issues don’t appear until the system goes live. Crafting async software that runs smoothly requires striking a balance between fairness and batching, while also managing contention and ensuring isolation.
Splitting for Latency, Batching for Throughput
Optimizing for low latency across many requests depends on a single principle: yield more often. That principle demands fairness between connections, so no single request can hold up the rest.
Take Redis (or a similar application with support for request pipelining). A simple approach reads data straight from the connection before all data has arrived. With pipelined requests, the whole pipelined request (or nearly all of it) lands in an in-memory buffer instead. Reading frames off that buffer then produces Poll::Ready results without needing to return to the network at all.
The result is long polls and unequal treatment for users. Throughput is generally affected less — the total count of requests stays the same. But latency shifts greatly, since an entire chain of operations has to pause behind a single preceding one.
In this example, explicitly giving up control after each request cuts latency by about 10×. You can get an even greater reduction by waiting to give up control only after several reads in a row are ready right away.
Here is the example code Russell gives:
The handle_conn method runs an endless loop that keeps checking for incoming data on the connection until shutdown is triggered. Inside that loop, it calls tokio::select!, which waits for two possible outcomes at once. One outcome is a call to self.connection.read_frame() returning a result; the other is a signal arriving on self.shutdown. When read_frame completes successfully, its result is stored in frame, and the method moves on to execute the command contained within it using execute_command(&self.db, &mut self.connection, frame).await. If the shutdown signal arrives instead, the loop ends immediately and the method returns success. The comment about fairness suggests adding tokio::task::yield_now().await between iterations, though it isn’t included in the code here.
Batching to Amortize Overhead
Fairness comes at a cost, and the second principle is batching: the more useful work an application can do per runtime event—changing tasks, polling, moving between workers, or changing threads—the more efficient it becomes.
Tokio::fs serves as Russell’s prime illustration, and he occasionally pushes his point further by saying that “tokio::fs is considered harmful.” — that is, without io_uring, Tokio runs each filesystem operation on the blocking pool. Every call to spawn_blocking carries its own cost, and the blocking pool is shared across the entire runtime.
Before you start performing a series of filesystem operations—or any other kind of blocking work—consider bundling those operations together into the largest possible single batch. Sometimes, rather than trying to handle all of it at once within your application, a dedicated OS thread is actually the more efficient approach.
The rule holds true no matter where you engage with Tokio. Whenever you know a task will reach the global queue, combining several tasks into one batch can ease that coordination burden across the board.
Spawning a task comes at no cost either, yet even so it is not free. The creation of a task remains inexpensive, but when you spawn 100s or 1000s of tasks, each instance demands attention from the runtime individually. Every one introduces fresh opportunities for scheduling delay, adds another poll for the runtime to manage, and contributes to the overhead in general.
Before you spawn a task, take stock of what you are really scheduling: putting a 10-microsecond unit of work on its own task is likely to hinder more than it helps. You can use tools such as dial9 or tokio-metrics to monitor the lifespan of your tasks.
How to Know You Have a Problem
In his work, Russell presents two sets of markers that signal problems. The first set concerns latency.
- P99 is much greater than P50.
- Polls take longer than the work inside them should require.
- Many spans fall inside a single poll.
The second is about throughput:
- Tokio APIs such as spawn_blocking consume noticeable time in flamegraphs.
- A tight loop performs many individually small filesystem or blocking operations.
- Throughput improves when the same work is grouped into larger units.
The Exceptions and the Metrics
Tokio isn’t always where the problem lies, according to Russell, who points out that dial9 has shed light on the library through extensive visibility. As often as dial9 uncovers an actual Tokio issue, it just as frequently shows the absence of one, giving people the confidence to look elsewhere for answers.
Of course, sometimes it is a Tokio problem.
The most helpful Tokio metric is the schedule latency histogram, which was recently added. Schedule latency measures how long it takes for Tokio to poll a future after your task becomes ready to run — say, when the socket has data. This metric does not reveal the cause, but it is the clearest sign of problems between Tokio and your code.
Our View
These guidelines prove helpful. The examples are specific, and the difference between latency and throughput is explained clearly. Working backward from a real metric you are trying to improve is solid advice.
This entry is not a recipe for cooking. Instead, it presents a collection of guiding ideas, and Russell himself admits that the response to numerous inquiries often comes down to “it depends.” That modesty is what makes the whole piece worth reading.
Russell’s promise of a sample app will prove useful. Having dial9 traces sit beside these principles in action will make it easier for readers to grasp how to put them into practice.
Anyone building a Tokio application will find the post a solid place to begin. It does not replace careful profiling and testing, but it points you toward what to examine.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.