Exploring Generators and Coroutines

Le­t’s re­vi­sit the idea of ge­ne­ra­tors in Py­tho­n, in or­der to un­ders­tan­d how the su­pport for co­rou­ti­nes was achie­ved in la­test ver­sions of Py­thon (3.6, at the ti­me of this wri­tin­g).

By re­viewing the mi­les­to­nes on ge­ne­ra­tor­s, ch­ro­no­lo­gi­ca­ll­y, we can get a be­tte­r i­dea of the evo­lu­tion that lead to as­yn­ch­ro­nous pro­gra­m­ming in Py­tho­n.

We wi­ll re­view the main chan­ges in Py­thon that re­la­te to ge­ne­ra­tors an­d a­s­yn­ch­ro­nous pro­gra­m­min­g, star­ting wi­th PE­P-255 (Sim­ple Ge­ne­ra­tor­s), PE­P-342 (­Co­rou­ti­nes via Enhan­ced Ge­ne­ra­tor­s), PE­P-380 (S­yn­tax for de­le­ga­ting to a ­Su­b-­Ge­ne­ra­to­r), and fi­nis­hing wi­th PE­P-525 (A­s­yn­ch­ro­nous Ge­ne­ra­tor­s).

Simple Generators

PE­P-255 in­tro­du­ced ge­ne­ra­tors to Py­tho­n. The idea is that when we pro­ce­ss so­me ­da­ta, we do­n’t ac­tua­lly need all of that to be in me­mo­ry at on­ce. ­Most of the ti­me­s, ha­ving one va­lue at the ti­me is enou­gh. La­zy eva­lua­tion is a ­good trait to ha­ve in so­ftwa­re, be­cau­se in this ca­se it means that le­ss ­me­mo­ry is us­e­d. It’s al­so a key con­cept in other pro­gra­m­ming lan­gua­ges, and one of the main ideas be­hind func­tio­nal pro­gra­m­min­g.

The new yield keyword was added to Python, with the meaning of producing an element that will be consumed by another caller function.

The mere presence of the yield keyword on any part of the function, automatically makes that a generator function. When called, this function will create a generator object, which can be advanced, producing its elements, one at the time. By calling the generator successive times with the next() function, the generator advances to the next yield statement, producing values. After the generator produced a value, the generator is suspended, waiting to be called again.

Take the ran­ge buil­t-in func­tio­n, for exam­ple. In Py­thon 2, this func­tio­n ­re­turns a list wi­th all the num­bers on the in­ter­va­l. Ima­gi­ne we want to co­me up wi­th a si­mi­lar im­ple­men­ta­tion of it, in or­der to get the sum of all num­bers up ­to a cer­tain li­mi­t.

LIMIT = 1_000_000
def old_range(n):
    numbers = []
    i = 0
    while i < n:
        numbers.append(i)
        i += 1
    return numbers

print(sum(old_range(LIMIT)))

Now le­t’s see how mu­ch me­mo­ry is us­e­d:

$ /usr/bin/time -f %M python rangesum.py
499999500000
48628

The first num­ber is the re­sult of the prin­t, whilst the se­cond one is the ou­tput of the ti­me co­m­man­d, prin­ting out the me­mo­ry us­ed by the pro­gram (~48 ­Mi­B).

No­w, what if this is im­ple­men­ted wi­th a ge­ne­ra­tor ins­tea­d?

We just ha­ve to get rid of the lis­t, and pla­ce the yield sta­te­ment ins­tea­d, in­di­ca­ting that we want to pro­du­ce the va­lue on the ex­pres­sion that fo­llo­ws the ke­ywor­d.

LIMIT = 1_000_000
def new_range(n):
    i = 0
    while i < n:
        yield i
        i += 1

print(sum(new_range(LIMIT)))

This ti­me, the re­sult is:

$ /usr/bin/time -f %M python rangesum.py
499999500000
8992

We see a hu­ge di­ffe­ren­ce: the im­ple­men­ta­tion that hol­ds all num­bers in a lis­t in me­mo­r­y, uses ~48­Mi­B, whe­reas the im­ple­men­ta­tion that just uses one num­ber at ­the ti­me, uses mu­ch le­ss me­mo­ry (< 9 Mi­B) 1.

We see the idea: when the yield <ex­pres­sio­n> is rea­che­d, the re­sult of the ex­pres­sion wi­ll be pa­ss­ed to the ca­ller co­de, and the ge­ne­ra­tor wi­ll re­mai­n sus­pen­ded at that li­ne in the meanwhi­le.

>>> import inspect
>>> r = new_range(1_000_000)
>>> inspect.getgeneratorstate(r)
'GEN_CREATED'
>>> next(r)
0
>>> next(r)
1
>>> inspect.getgeneratorstate(r)
'GEN_SUSPENDED'

Generators are iterable objects. An iterable is an object whose __iter__ method, constructs a new iterator, every time is called (with iter(it), for instance). An iterator is an object whose __iter__ returns itself, and its __next__ method contains the logic to produce new values each time is called, and how to signal the stop (by raising StopIteration).

The idea of iterables is that they advance through values, by calling the built-in next() function on it, and this will produce values until the StopIteration exception is raised, signalling the end of the iteration.

>>> def f():
...     yield 1
...     yield 2

>>> g = f()
>>> next(g)
1
>>> next(g)
2
>>> next(g)
StopIteration:

>>> list(f())
[1, 2]

In the first case, when calling f(), this creates a new generator. The first two calls to next(), will advance until the next yield statement, producing the values they have set. When there is nothing else to produce, the StopIteration exception is raised. Something similar to this, is actually run, when we iterate over this object in the form of for x in iterable: …. Only that Python internally handles the exception that determines when the for loop stops.

Be­fo­re wra­pping up the in­tro­duc­tion to ge­ne­ra­tor­s, I want to make a qui­ck ­co­m­men­t, and hi­gh­li­ght so­me­thing im­por­tant about the ro­le of ge­ne­ra­tors in the ­lan­gua­ge, and why the­y’­re su­ch a neat abs­trac­tion to ha­ve.

Ins­tead of using the ea­ger ver­sion (the one that sto­res eve­r­y­thing in a lis­t), ­you mi­ght con­si­der avoi­ding that by just using a lo­op and coun­ting in­si­de it. I­t’s like sa­ying “a­ll I need is just the coun­t, so I mi­ght as we­ll jus­t ac­cu­mu­la­te the va­lue in a lo­op, and tha­t’s it”. So­me­thing sli­gh­tly si­mi­lar to:

total = 0
i = 0
while i < LIMIT:
    total += i
    i += 1

This is so­me­thing I mi­ght con­si­der doing in a lan­gua­ge that does­n’t ha­ve ­ge­ne­ra­tor­s. Do­n’t do this. Ge­ne­ra­tors are the ri­ght way to go. By using a ­ge­ne­ra­to­r, we’­re doing mo­re than just wra­pping the co­de of an ite­ra­tio­n; we’­re ­crea­ting a se­quen­ce (whi­ch could even be in­fi­ni­te), and na­ming it. This ­se­quen­ce we ha­ve, is an ob­ject we can use in the rest of the co­de. It’s an a­bs­trac­tio­n. As su­ch, we can com­bi­ne it wi­th the rest of the co­de (for exam­ple ­to fil­ter on it), reu­se it, pa­ss it along to other ob­jec­ts, and mo­re.

For example, let’s say we have the sequence created with new_range(), and then we realize that we need the first 10 even numbers of it. This is as simple as doing.

>>> import itertools
>>> rg = new_range(1_000_000)
>>> itertools.islice(filter(lambda n: n % 2 == 0, rg), 10)

And this is so­me­thing we could not so ea­si­ly ac­com­plis­h, had we cho­sen to­ ig­no­re ge­ne­ra­tor­s.

For year­s, this has been all pre­tty mu­ch about ge­ne­ra­tors in Py­tho­n. Ge­ne­ra­tor­s we­re in­tro­du­ced wi­th the idea of ite­ra­tion and la­zy com­pu­ta­tion in min­d.

La­ter on, the­re was ano­ther enhan­ce­men­t, by PE­P-342, adding mo­re me­tho­ds to­ ­the­m, wi­th the goal of su­ppor­ting co­rou­ti­nes.

Coroutines

Rou­gh­ly speakin­g, the idea of co­rou­ti­nes is to pau­se the exe­cu­tion of a ­func­tion at a gi­ven poin­t, from whe­re it can be la­ter re­su­me­d. The idea is tha­t whi­le a co­rou­ti­ne is sus­pen­ded, the pro­gram can swi­tch to run ano­ther part of ­the co­de. Ba­si­ca­ll­y, we need func­tions that can be pau­s­e­d.

As we have seen from the previous example, generators have this feature: when the yield <expresson>, is reached, a value is produced to the caller object, and in the meantime the generator object is suspended. This suggested that generators can be used to support coroutines in Python, hence the name of the PEP being “Coroutines via Enhanced Generators”.

The­re is mo­re, thou­gh. Co­rou­ti­nes ha­ve to su­pport to be re­su­med fro­m ­mul­ti­ple en­try poin­ts to con­ti­nue their exe­cu­tio­n. The­re­fo­re, mo­re chan­ges are ­re­qui­re­d. We need to be able to pa­ss da­ta ba­ck to the­m, and hand­le ex­cep­tion­s. ­For this, mo­re me­tho­ds we­re added to their in­ter­fa­ce.

  • sen­­d(<­­va­­lue>)

  • th­ro­w(ex_­ty­pe[, ex_­va­lue[, ex_­­tra­­ce­­ba­­ck]])

  • clo­se()

The­se me­tho­ds allow sen­ding a va­lue to a ge­ne­ra­to­r, th­ro­wing an ex­cep­tio­n in­si­de it, and clo­sing it, res­pec­ti­ve­l­y.

The send() method implies that yield becomes an expression, rather than a statement (as it was before). With this, is possible to assign the result of a yield to a variable, and the value will be whatever it was sent to it.

>>> def gen(start=0):
...     step = start
...     while True:
...         value = yield step
...         print(f"Got {value}")
...         step += 1
...
>>> g =  gen(1)
>>> next(g)
1
>>> g.send("hello")
Got hello
2
>>> g.send(42)
Got 42
3

As we can see from this previous code, the value sent by yield is going to be the result of the send, (in this case, the consecutive numbers of the sequence), while the value passed in the send(), the parameter, is the result that is assigned to value as returned by the yield, and printed out on the next line.

Before sending any values to the generator, this has to be advanced to the next yield. In fact, advancing is the only allowed operation on a newly-created generator. This can be done by calling next(g) or g.send(None), which are equivalent.

War­ning

Re­mem­ber to alwa­ys ad­van­ce a ge­ne­ra­tor that was just create­d, or you wi­ll ­get a Ty­peE­rror.

With the .throw() method the caller can make the generator raise an exception at the point where is suspended. If this exception is handled internally in the generator, it will continue normally and the return value will be the one of the next yield line that reached. If it’s not handled by the generator, it will fail, and the exception will propagate to the caller.

The .close() method is used to terminate the generator. It will raise the GeneratorExit exception inside the generator. If we wish to run some clean up code, this is the exception to handle. When handling this exception, the only allowed action is to return a value.

Wi­th the­se addi­tion­s, ge­ne­ra­tors ha­ve now evol­ved in­to co­rou­ti­nes. This mean­s our co­de can now su­pport con­cu­rrent pro­gra­m­ming, sus­pend the exe­cu­tion of ­ta­sks, com­pu­te no­n-­blo­cking I/O, and su­ch.

Whi­le this wo­rks, hand­ling many co­rou­ti­nes, re­fac­tor ge­ne­ra­tor­s, and or­ga­ni­zin­g ­the co­de be­ca­me a bit cum­ber­so­me. Mo­re wo­rk had to be do­ne, if we wanted to­ keep a Py­tho­nic way of doing con­cu­rrent pro­gra­m­min­g.

More Coroutines

PE­P-380 added mo­re chan­ges to co­rou­ti­nes, this ti­me wi­th the goal of su­ppor­tin­g ­de­le­ga­tion to su­b-­ge­ne­ra­tor­s. Two main things chan­ged in ge­ne­ra­tors to make ­them mo­re use­ful as co­rou­ti­nes:

  • Ge­­ne­­ra­­tors can now re­­turn va­­lues.

  • The yield from syn­ta­x.

Return Values in Generators

The keyword def, defines a function, which returns values (with the return keyword). However, as stated on the first section, if that def contains a yield, is a generator function. Before this PEP it would have been a syntax error to have a return in a generator function (a function that also has a yield. However, this is no longer the case.

Remember how generators stop by raising StopIteration. What does it mean that a generator returns a value? It means that it stops. And where does that value do? It’s contained inside the exception, as an attribute in StopIteration.value.

def gen():
        yield 1
        yield 2
        return "returned value"

>>> g = gen()
>>> try:
...     while True:
...         print(next(g))
... except StopIteration as e:
...     print(e.value)
...
1
2
returned value

No­ti­ce that the va­lue re­tur­ned by the ge­ne­ra­tor is sto­red in­si­de the ex­cep­tio­n, in Sto­pI­te­ra­tio­n.­va­lue. This mi­ght sound like is not the most ele­gan­t ­so­lu­tio­n, but doing so, pre­ser­ves the ori­gi­nal in­ter­fa­ce, and the pro­to­co­l ­re­mains un­chan­ge­d. It’s sti­ll the sa­me kind of ex­cep­tion sig­na­lling the end of ­the ite­ra­tio­n.

yield from

Ano­ther syn­tax chan­ge to the lan­gua­ge.

In its most basic form, the construction yield from <iterable>, can be thought of as:

for e in iterable:
    yield e

Ba­si­ca­lly this means that it ex­ten­ds an ite­ra­ble, yiel­ding all ele­men­ts tha­t ­this in­ter­nal ite­ra­ble can pro­du­ce.

For example, this way we could create a clone of the itertools.chain function from the standard library.

>>> def chain2(*iterables):
...:     for it in iterables:
...:         yield from it

>>> list(chain2("hello", " ", "world"))
['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']

Ho­we­ve­r, saving two li­nes of co­de is not the rea­son why this cons­truc­tion wa­s a­dded to the lan­gua­ge. The rai­son d’e­tre of this cons­truc­tion is to ac­tua­ll­y ­de­le­ga­te res­pon­si­bi­li­ty in­to sma­ller ge­ne­ra­tor­s, and chain the­m.

>>> def internal(name, limit):
...:     for i in range(limit):
...:         got = yield i
...:         print(f"{name} got: {got}")
...:     return f"{name} finished"

>>> def gen():
...:     yield from internal("A", 3)
...:     return (yield from internal("B", 2))

>>> g = gen()
>>> next(g)
0
>>> g.send(1)
A got: 1
1

>>> g.send(1)   # a few more calls until the generator ends
B got: 1
------------------------------------------------------
StopIteration        Traceback (most recent call last)
... in <module>()
----> 1 g.send(1)
StopIteration: B finished

Here we see how yield from handles proper delegation to an internal generator. Notice that we never send values directly to internal, but to gen, instead, and these values end up on the nested generator. What yield from is actually doing is creating a generator that has a channel to all nested generators. Values produced by these will be provided to the caller of gen. Values sent to it, will be passed along to the internal generators (the same for exceptions). Even the return value is handled, and becomes the return value of the top-level generator (in this case the string that states the name of the last generator becomes the resulting StopIteration.value).

We see now the real va­lue of this cons­truc­tio­n. Wi­th this, it’s ea­sier to­ ­re­fac­tor ge­ne­ra­tors in­to sma­ller pie­ce­s, com­po­se them and chain them to­ge­the­r whi­le pre­ser­ving the be­ha­viour of co­rou­ti­nes.

The new yield from syntax is a great step towards supporting better concurrency. We can now think generators as being “lightweight threads”, that delegate functionality to an internal generator, pause the execution, so that other things can be computed in that time.

Because syntactically generators are like coroutines, it was possible to accidentally confuse them, and end up placing a generator where a coroutine would have been expected (the yield from would accept it, after all). For this reason, the next step is to actually define the concept of coroutine as a proper type. With this change, it also followed that yield from evolved into await, and a new syntax for defining coroutines was introduced: async.

async def / await

A qui­ck no­te on how this re­la­tes to as­yn­ch­ro­nous pro­gra­m­ming in Py­tho­n.

On asyncio, or any other event loop, the idea is that we define coroutines, and make them part of the event loop. Broadly speaking the event loop will keep a list of the tasks (which wrap our coroutines) that have to run, and will schedule them to.

On our coroutines we delegate the I/O functionality we want to achieve, to some other coroutine or awaitable object, by calling yield from or await on it.

Then the event loop will call our coroutine, which will reach this line, delegating to the internal coroutine, and pausing the execution, which gives the control back to the scheduler (so it can run another coroutine). The event loop will monitor the future object that wraps our coroutine until is finished, and when it’s needed, it will update it by calling the .send() method on it. Which in turn, will pass along to the internal coroutine, and so on.

Before the new syntax for async and await was introduced, coroutines were defined as generators decorated with asyncio.coroutine (types.coroutine was added in Python 3.5, when the coroutine type itself was created). Nowadays, async def creates a native coroutine, and inside it, only the await expression is accepted (not yield from).

The following two coroutines step and coro are a simple example, of how await works similar to yield from delegating the values to the internal generator.

>>>  @types.coroutine
...: def step():
...:     s = 0
...:     while True:
...:         value = yield s
...:         print("Step got value ", value)
...:         s += 1

>>>  async def coro():
...:     while True:
...:         got = await step()
...:         print(got)


>>> c = coro()
>>> c.send(None)
0
>>> c.send("first")
Step got value  first
1

>>> c.send("second")
Step got value  second
2

>>> c.send("third")
Step got value  third
3

Once again, like in the yield from example, when we send a value to coro, this reaches the await instruction, which means that will pass to the step coroutine. In this simple example coro is something like what we would write, while step would be an external function we call.

The fo­llo­wing two co­rou­ti­nes are di­ffe­rent wa­ys of de­fi­ning co­rou­ti­nes.

# py 3.4
@asyncio.coroutine
def coroutine():
    yield from asyncio.sleep(1)

# py 3.5+
async def coroutine():
    await asyncio.sleep(1)

Basically this means that this asynchronous way of programming is kind of like an API, for working with event loops. It doesn’t really relate to asyncio, we could use any event loop (curio, uvloop, etc.), for this. The important part is to understand, that an event loop will call our coroutine, which will eventually reach the line where we defined the await, and this will delegate the function to an external function (in this case asyncio.sleep). When the event loop calls send(), this is also passed, and the await gives back control to the event loop, so a different coroutine can run.

The co­rou­ti­nes we de­fi­ne are the­re­fo­re in be­tween the event lo­op, and 3r­d-­par­ty ­func­tions that know how to hand­le the I/O in a no­n-­blo­cking fas­hio­n.

The event loop works then by a chain of await calls. Ultimately, at the end of that chain there is a generator, that pauses the execution of the function, and handles the I/O.

In fact if we check the type of asyncio.sleep, we’ll see that is indeed a generator:

>>> asyncio.sleep(1)
<generator object sleep at 0x...>

So with this new syntax, does this mean that await is like yield from?

Only with respect to coroutines. It’s correct to write await <coroutine>, as well as yield from <coroutine>, the former won’t work with other iterables (for example generators that aren’t coroutines, sequences, etc.). Conversely, the latter won’t work with awaitable objects.

The rea­son for this syn­tax chan­ge is for co­rrec­tness. Ac­tua­lly it’s not just a s­yn­tax chan­ge, the new co­rou­ti­ne ty­pe is pro­per­ly de­fi­ne­d.:

>>> from collections import abc
>>> issubclass(abc.Coroutine, abc.Awaitable)
True

Given that coroutines are syntactically like generators, it would be possible to mix them, and place a generator in an asynchronous code where in fact we expected a coroutine. By using await, the type of the object in the expression is checked by Python, and if it doesn’t comply, it will raise an exception.

Asynchronous Generators

In Python 3.5 not only the proper syntax for coroutines was added (async def / await), but also the concept of asynchronous iterators. The idea of having an asynchronous iterable is to iterate while running asynchronous code. For this new methods such as __aiter__ and __anext__ where added under the concept of asynchronous iterators.

However there was no support for asynchronous generators. That is analogous to saying that for asynchronous code we had to use iterables (like __iter__ / __next__ on regular code), but we couldn’t use generators (having a yield in an async def function was an error).

This chan­ged in Py­thon 3.6, and now this syn­tax is su­pporte­d, wi­th the se­man­ti­cs of a re­gu­lar ge­ne­ra­tor (la­zy eva­lua­tio­n, sus­pend and pro­du­ce one e­le­ment at the ti­me, etc.), whi­le ite­ra­tin­g.

Con­si­der this sim­ple exam­ple on whi­ch we want to ite­ra­te whi­le ca­lling so­me I/O­ ­co­de that we do­n’t want to blo­ck upo­n.

async def recv(no, size) -> str:
    """Simulate reading <size> bytes from a remote source, asynchronously.
    It takes a time proportional to the bytes requested to read.
    """
    await asyncio.sleep((size // 512) * 0.4)
    chunk = f"[chunk {no} ({size})]"
    return chunk


class AsyncDataStreamer:
    """Read 10 times into data"""
    LIMIT = 10
    CHUNK_SIZE = 1024

    def __init__(self):
        self.lecture = 0

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.lecture >= self.LIMIT:
            raise StopAsyncIteration

        result = await recv(self.lecture, self.CHUNK_SIZE)
        self.lecture += 1
        return result

async def test():
    async for read in AsyncDataStreamer():
        logger.info("collector on read %s", read)

The test function will simply exercise the iterator, on which elements are produced, one at the time, while calling an I/O task (in this example asyncio.sleep).

Wi­th as­yn­ch­ro­nous ge­ne­ra­tor­s, the sa­me could be rew­ri­tten in a mo­re com­pac­t wa­y.

async def async_data_streamer():
    LIMIT = 10
    CHUNK_SIZE = 1024
    lecture = 0
    while lecture < LIMIT:
        lecture += 1
        yield await recv(lecture, CHUNK_SIZE)

Summary

It all started wi­th ge­ne­ra­tor­s. It was a sim­ple way of ha­ving la­zy com­pu­ta­tio­n in Py­tho­n, and run­ning mo­re effi­cient pro­gra­ms, that use le­ss me­mo­r­y.

This evol­ved in­to co­rou­ti­nes, taking ad­van­ta­ge of the fact that ge­ne­ra­tors can ­sus­pend their exe­cu­tio­n. By ex­ten­ding the in­ter­fa­ce of ge­ne­ra­tor­s, co­rou­ti­nes ­pro­vi­ded mo­re po­wer­ful fea­tu­res to Py­tho­n.

Coroutines were also improved to support better patterns, and the addition of yield from was a game changer, that allows to have better generators, refactor into smaller pieces, and reorganize the logic better.

The addition of an event loop to the standard library, helps to provide a referential way of doing asynchronous programming. However, the logic of the coroutines and the await syntax it not bound to any particular event loop. It’s an API 2 for doing asynchronous programming.

As­yn­ch­ro­nous ge­ne­ra­tor was the la­test addi­tion to Py­thon that re­la­tes to­ ­ge­ne­ra­tor­s, and they help build mo­re com­pact (and effi­cien­t!) co­de fo­r a­s­yn­ch­ro­nous ite­ra­tio­n.

In the end, behind all the logic of async / await, everything is a generator. Coroutines are in fact (technically), generators. Conceptually they are different, and have different purposes, but in terms of implementation generators are what make all this asynchronous programming possible.

Slides

Notes

1

Nee­d­le­ss to sa­y, the re­sul­ts wi­ll va­ry from sys­tem to sys­te­m, but we ge­t an idea of the di­ffe­ren­ce be­tween bo­th im­ple­men­ta­tion­s.

2

This is an idea by Da­vid Bea­z­le­y, that you can see at http­s://­you­tu.­be­/­Z­z­fH­j­y­tDceU