Design and internals¶
Here we’ll discuss Trio’s overall design and architecture: how it fits together and why we made the decisions we did. If all you want to do is use Trio, then you don’t need to read this – though you might find it interesting. The main target audience here is (a) folks who want to read the code and potentially contribute, (b) anyone working on similar libraries who want to understand what we’re up to, (c) anyone interested in I/O library design generally.
There are many valid approaches to writing an async I/O library. This is ours.
High-level design principles¶
Trio’s two overriding goals are usability and correctness: we want to make it easy to get things right.
Of course there are lots of other things that matter too, like speed, maintainability, etc. We want those too, as much as we can get. But sometimes these things come in conflict, and when that happens, these are our priorities.
In some sense the entire rest of this document is a description of how
these play out, but to give a simple example: Trio’s
KeyboardInterrupt handling machinery is a bit tricky and hard to
test, so it scores poorly on simplicity and maintainability. But we
think the usability+correctness gains outweigh this.
There are some subtleties here. Notice that it’s specifically “easy to get things right”. There are situations (e.g. writing one-off scripts) where the most “usable” tool is the one that will happily ignore errors and keep going no matter what, or that doesn’t bother with resource cleanup. (Cf. the success of PHP.) This is a totally valid use case and valid definition of usability, but it’s not the one we use: we think it’s easier to build reliable and correct systems if exceptions propagate until handled and if the system catches you when you make potentially dangerous resource handling errors, so that’s what we optimize for.
It’s also worth saying something about speed, since it often looms large in comparisons between I/O libraries. This is a rather subtle and complex topic.
In general, speed is certainly important – but the fact that people sometimes use Python instead of C is a pretty good indicator that usability often trumps speed in practice. We want to make trio fast, but it’s not an accident that it’s left off our list of overriding goals at the top: if necessary we are willing to accept some slowdowns in the service of usability and reliability.
To break things down in more detail:
First of all, there are the cases where speed directly impacts
correctness, like when you hit an accidental
O(N**2) algorithm and
your program effectively locks up. Trio is very careful to use
algorithms and data structures that have good worst-case behavior
(even if this might mean sacrificing a few percentage points of speed
in the average case).
Similarly, when there’s a conflict, we care more about 99th percentile latencies than we do about raw throughput, because insufficient throughput – if it’s consistent! – can often be budgeted for and handled with horizontal scaling, but once you lose latency it’s gone forever, and latency spikes can easily cross over to become a correctness issue (e.g., an RPC server that responds slowly enough to trigger timeouts is effectively non-functional). Again, of course, this doesn’t mean we don’t care about throughput – but sometimes engineering requires making trade-offs, especially for early-stage projects that haven’t had time to optimize for all use cases yet.
And finally: we care about speed on real-world applications quite a bit, but speed on microbenchmarks is just about our lowest priority. We aren’t interested in competing to build “the fastest echo server in the West”. I mean, it’s nice if it happens or whatever, and microbenchmarks are an invaluable tool for understanding a system’s behavior. But if you play that game to win then it’s very easy to get yourself into a situation with seriously misaligned incentives, where you have to start compromising on features and correctness in order to get a speedup that’s totally irrelevant to real-world applications. In most cases (we suspect) it’s the application code that’s the bottleneck, and you’ll get more of a win out of running the whole app under PyPy than out of any heroic optimizations to the I/O layer. (And this is why Trio does place a priority on PyPy compatibility.)
As a matter of tactics, we also note that at this stage in Trio’s lifecycle, it’d probably be a mistake to worry about speed too much. It doesn’t make sense to spend lots of effort optimizing an API whose semantics are still in flux.
User-level API principles¶
Trio is very much a continuation of the ideas explored in this blog post, and in particular the principles identified there that make curio easier to use correctly than asyncio. So trio also adopts these rules, in particular:
- The only form of concurrency is the task.
- Tasks are guaranteed to run to completion.
- Task spawning is always explicit. No callbacks, no implicit concurrency, no futures/deferreds/promises/other APIs that involve callbacks. All APIs are “causal” except for those that are explicitly used for task spawning.
- Exceptions are used for error handling;
withblocks for handling cleanup.
Cancel points and schedule points¶
The first major place that trio departs from curio is in its decision to make a much larger fraction of the API use sync functions rather than async functions, and to provide strong conventions about cancel points and schedule points. (At this point, there are a lot of ways that trio and curio have diverged. But this was really the origin – the tipping point where I realized that exploring these ideas would require a new library, and couldn’t be done inside curio.) The full reasoning here takes some unpacking.
First, some definitions: a cancel point is a point where your code
checks if it has been cancelled – e.g., due to a timeout having
expired – and potentially raises a
Cancelled error. A schedule
point is a point where the current task can potentially be suspended,
and another task allowed to run.
In curio, the convention is that all operations that interact with the run loop in any way are syntactically async, and it’s undefined which of these operations are cancel/schedule points; users are instructed to assume that any of them might be cancel/schedule points, but with a few exceptions there’s no guarantee that any of them are unless they actually block. (I.e., whether a given call acts as a cancel/schedule point is allowed to vary across curio versions and also depending on runtime factors like network load.)
But when using an async library, there are good reasons why you need to be aware of cancel and schedule points. They introduce a set of complex and partially conflicting constraints on your code:
You need to make sure that every task passes through a cancel point regularly, because otherwise timeouts become ineffective and your code becomes subject to DoS attacks and other problems. So for correctness, it’s important to make sure you have enough cancel points.
But… every cancel point also increases the chance of subtle
bugs in your program, because it’s a place where you have to be
prepared to handle a
Cancelled exception and clean up
properly. And while we try to make this as easy as possible,
these kinds of clean-up paths are notorious for getting missed
in testing and harboring subtle bugs. So the more cancel points
you have, the harder it is to make sure your code is correct.
Similarly, you need to make sure that every task passes through a schedule point regularly, because otherwise this task could end up hogging the event loop and preventing other code from running, causing a latency spike. So for correctness, it’s important to make sure you have enough schedule points.
But… you have to be careful here too, because every schedule point is a point where arbitrary other code could run, and alter your program’s state out from under you, introducing classic concurrency bugs. So as you add more schedule points, it becomes exponentially harder to reason about how your code is interleaved and be sure that it’s correct.
So an important question for an async I/O library is: how do we help the user manage these trade-offs?
Trio’s answer is informed by two further observations:
First, any time a task blocks (e.g., because it does an
sock.recv() but there’s no data available to receive), that
has to be a cancel point (because if the I/O never arrives, we
need to be able to time out), and it has to be a schedule point
(because the whole idea of asynchronous programming is that
when one task is waiting we can switch to another task to get
something useful done).
And second, a function which sometimes counts as a cancel/schedule point, and sometimes doesn’t, is the worst of both worlds: you have put in the effort to make sure your code handles cancellation or interleaving correctly, but you can’t count on it to help meet latency requirements.
With all that in mind, trio takes the following approach:
Rule 1: to reduce the number of concepts to keep track of, we collapse cancel points and schedule points together. Every point that is a cancel point is also a schedule point and vice versa. These are distinct concepts both theoretically and in the actual implementation, but we hide that distinction from the user so that there’s only one concept they need to keep track of.
Rule 2: Cancel+schedule points are determined statically. A trio primitive is either always a cancel+schedule point, or never a cancel+schedule point, regardless of runtime conditions. This is because we want it to be possible to determine whether some code has “enough” cancel/schedule points by reading the source code.
In fact, to make this even simpler, we make it so you don’t even have to look at the function arguments: each function is either a cancel+schedule point on every call or on no calls.
Observation: since blocking is always a cancel+schedule point, rule 2 implies that any function that sometimes blocks is always a cancel+schedule point.
So that gives us a number of cancel+schedule points: all the functions
that can block. Are there any others? Trio’s answer is: no. It’s easy
to add new points explicitly (throw in a
sleep(0) or whatever) but
hard to get rid of them when you don’t want them. (And this is a real
issue – “too many potential cancel points” is definitely a tension
while trying to build things like task supervisors in curio.) And we
expect that most trio programs will execute potentially-blocking
operations “often enough” to produce reasonable behavior. So, rule 3:
the only cancel+schedule points are the potentially-blocking
And now that we know where our cancel+schedule points are, there’s the
question of how to effectively communicate this information to the
user. We want some way to mark out a category of functions that might
block or trigger a task switch, so that they’re clearly distinguished
from functions that don’t do this. Wouldn’t it be nice if there were
some Python feature, that naturally divided functions into two
categories, and maybe put some sort of special syntactic marking on
with the functions that can do weird things like block and task
switch…? What a coincidence, that’s exactly how async functions
work! Rule 4: in trio, only the potentially blocking functions are
async. So e.g.
Event.wait() is async, but
Summing up: out of what’s actually a pretty vast space of design possibilities, we declare by fiat that when it comes to trio primitives, all of these categories are identical:
- async functions
- functions that can, under at least some circumstances, block
- functions where the caller needs to be prepared to handle
- functions that are guaranteed to notice any pending cancellation
- functions where you need to be prepared for a potential task switch
- functions that are guaranteed to take care of switching tasks if appropriate
This requires some non-trivial work internally – it actually takes a fair amount of care to make those 4 cancel/schedule categories line up, and there are some shenanigans required to let sync and async APIs both interact with the run loop on an equal footing. But this is all invisible to the user, we feel that it pays off in terms of usability and correctness.
There is one exception to these rules, for async context
managers. Context managers are composed of two operations – enter and
exit – and sometimes only one of these is potentially
async with lock: can block when entering but
never when exiting;
async with open_nursery() as ...: can block
when exiting but never when entering.) But, Python doesn’t have
“half-asynchronous” context managers: either both operations are
async-flavored, or neither is. In Trio we take a pragmatic approach:
for this kind of async context manager, we enforce the above rules
only on the potentially blocking operation, and the other operation is
allowed to be syntactically
async but semantically
synchronous. And async context managers should always document which
of their operations are schedule+cancel points.
Exceptions always propagate¶
Another rule that trio follows is that exceptions must always propagate. This is like the zen line about “Errors should never pass silently”, except that in every other concurrency library for Python (threads, asyncio, curio, …), it’s fairly common to end up with an undeliverable exception, which just gets printed to stderr and then discarded. While we understand the pragmatic constraints that motivated these libraries to adopt this approach, we feel that there are far too many situations where no human will ever look at stderr and notice the problem, and insist that trio APIs find a way to propagate exceptions “up the stack” – whatever that might mean.
This is often a challenging rule to follow – for example, the call soon code has to jump through some hoops to make it happen – but its most dramatic influence can seen in trio’s task-spawning interface, where it motivates the use of “nurseries”:
async def parent(): async with trio.open_nursery() as nursery: nursery.spawn(child)
(See Tasks let you do multiple things at once for full details.)
If you squint you can see the influence of erlang’s “task linking” idea here, but it’s quite different in detail, exactly because Python has exceptions and Erlang doesn’t. Erlang’s links are symmetric and optional; to support exceptions we need ours to be asymmetric and mandatory.
This design also turns out to enforce a remarkable, unexpected invariant.
In the blog post
I called out a nice feature of curio’s spawning API, which is that
since spawning is the only way to break causality, and in curio
spawn is async, this means that in curio sync functions are
guaranteed to be causal. One limitation though is that this invariant
is actually not very predictive: in curio there are lots of async
functions that could spawn off children and violate causality, but
most of them don’t, but there’s no clear marker for the ones that do.
Our API doesn’t quite give that guarantee, but actually a better one. In trio:
- Sync functions can’t create nurseries, because nurseries require an
- Any async function can create a nursery and spawn new tasks… but creating a nursery allows task spawning without allowing causality breaking, because the children have to exit before the function is allowed to return. So we can preserve causality without having to give up concurrency!
- The only way to violate causality (which is an important feature, just one that needs to be handled carefully) is to explicitly create a nursery object in one task and then pass it into another task. And this provides a very clear and precise signal about where the funny stuff is happening – just watch for the nursery object getting passed around.
Introspection, debugging, testing¶
Tools for introspection and debugging are critical to achieving usability and correctness in practice, so they should be first-class considerations in trio.
Similarly, the availability of powerful testing tools has a huge impact on usability and correctness; we consider testing helpers to be very much in scope for the trio project.
Specific style guidelines¶
As noted above, functions that don’t block should be sync-colored, and functions that might block should be async-colored and unconditionally act as cancel+schedule points.
Any function that takes a callable to run should have a signature like:
def call_the_thing(fn, *args, kwonly1, kwonly2, ...):: ...
fn(*args)is the thing to be called, and
kwonly2, … are keyword-only arguments that belong to
call_the_thing. This applies even if
call_the_thingdoesn’t take any arguments of its own, i.e. in this case its signature looks like:
def call_the_thing(fn, *args):: ...
This allows users to skip faffing about with
functools.partial()in most cases, while still providing an unambiguous and extensible way to pass arguments to the caller. (Hat-tip to asyncio, who we stole this convention from.)
Whenever it makes sense, trio classes should have a method called
statistics()which returns an immutable object with named fields containing internal statistics about the object that are useful for debugging or introspection (examples).
Functions or methods whose purpose is to wait for a condition to become true should be called
wait_<condition>. This avoids ambiguities like “does
await readable()check readability (returning a bool) or wait for readability?”.
Sometimes this leads to the slightly funny looking
await wait_.... Sorry. As far as I can tell all the alternatives are worse, and you get used to the convention pretty quick.
If it’s desirable to have both blocking and non-blocking versions of a function, then they look like:
async def OPERATION(...): ... def OPERATION_nowait(...): ...
trio.WouldBlockif it would block.
…we should, but currently don’t, have a solid convention to distinguish between functions that take an async callable and those that take a sync callable. See issue #68.
A brief tour of trio’s internals¶
If you want to understand how trio is put together internally, then
the first thing to know is that there’s a very strict internal
trio._core package is a fully self-contained
implementation of the core scheduling/cancellation/IO handling logic,
and then the other
trio.* modules are implemented in terms of the
API it exposes. (If you want to see what this API looks like, then
import trio; print(trio._core.__all__)). Everything exported from
trio._core is also exported as part of either the
trio.hazmat namespaces. (This is managed through the use of a
@_hazmat decorator that marks which items in
trio._core.__all__ should go into
Rationale: currently, trio is a new project in a novel part of the
design space, so we don’t make any stability guarantees. But the goal
is to reach the point where we can declare the API stable. It’s
unlikely that we’ll be able to quickly explore all possible corners of
the design space and cover all possible types of I/O. So instead, our
strategy is to make sure that it’s possible for independent packages
to add new features on top of trio. Enforcing the
trio._core split is a way of eating our own dogfood: basic
actually implemented solely in terms of public APIs. And the hope is
that by doing this, we increase the chances that someone who comes up
with a better kind of queue or wants to add some new functionality
like, say, file system change watching, will be able to do that on top
of our public APIs without having to modify trio internals.
There are three notable sub-modules that are largely independent of the rest of trio, and could (possibly should?) be extracted into their own independent packages:
MultiErrorand associated infrastructure.
_ki.py: Implements the core infrastructure for safe handling of
The most important submodule, where everything is integrated, is
_run.py. (This is also by far the largest submodule; it’d be nice
to factor bits of it out with possible, but it’s tricky because the
core functionality genuinely is pretty intertwined.) Notably, this is
where cancel scopes, nurseries, and
Task are defined; it’s
also where the scheduler state and
The one thing that isn’t in
_run.py is I/O handling. This is
delegated to an
IOManager class, of which there are currently
_io_epoll.py(used on Linux, Illuminos)
_io_kqueue.py(used on MacOS, *BSD)
_io_windows.py(used on Windows)
The epoll and kqueue backends take advantage of the epoll and kqueue
wrappers in the stdlib
select module. The windows backend uses
CFFI to access to the Win32 API directly (see
trio/_core/_windows_cffi.py). In general, we prefer to go directly
to the raw OS functionality rather than use
- Controlling our own fate: I/O handling is pretty core to what trio
is about, and
selectorsis (as of 2017-03-01) somewhat buggy (e.g. issue 29587, issue 29255). Which isn’t a big deal on its own, but since
selectorsis part of the standard library we can’t fix it and ship an updated version; we’re stuck with whatever we get. We want more control over our users’ experience than that.
- Impedence mismatch: the
selectorsAPI isn’t particularly well-fitted to how we want to use it. For example, kqueue natively treats an interest in readability of some fd as a separate thing from an interest in that same fd’s writability, which neatly matches trio’s model.
selectors.KqueueSelectorgoes to some effort internally to lump together all interests in a single fd, and to use it we’d then we’d have to jump through more hoops to reverse this. Of course, the native epoll API is fd-centric in the same way as the
selectorsAPI so we do still have to write code to jump through these hoops, but the point is that the
selectorsabstractions aren’t providing a lot of extra value.
- (Most important) Access to raw platform capabilities:
selectorsis highly inadequate on Windows, and even on Unix-like systems it hides a lot of power (e.g. kqueue can do a lot more than just check fd readability/writability!).
IOManager layer provides a fairly raw exposure of the capabilities
of each system, with public API functions that vary between different
backends. (This is somewhat inspired by how
os works.) These
public APIs are then exported as part of
higher-level APIs like
trio.socket abstract over these
system-specific APIs to provide a uniform experience.
Currently the choice of backend is made statically at import time, and there is no provision for “pluggable” backends. The intuition here is that we’d rather focus our energy on making one set of solid, official backends that provide a high-quality experience out-of-the-box on all supported systems.