15.06.2018 by Jonatan Zint

async/await with Python

A practical example with asynchronous Python

Web applications that have to keep open websocket connections, need to be asynchronously executed. If you would do so by creating a new thread for each connection that would be pretty inefficient and hard for programmers to cope with all that multithreading.
Asynchronicity was a key feature for Javascript to become so popular for server side web applications. The runtime is by default an event loop, allowing constructs like these:

1
2
3
4
  fs.readFile('foo.txt', function(err, data) {
    if (!err)
      console.log('data was read')
  });

Reading a file relies on the filesystem, and you can’t really be sure how long that is gonna take, so here you pass a callback which will be picked up by the NodeJS event loop (a very nice detailed explanation to be found here). Eventually this Syntax evolved. At first it became Promises, to get out of callback hell, and recently with the ECMAScript® 2017 Language Specification async/await found its place in the language specification, while being available in NodeJS since version 8. It serves the purpose of making your execution of you program feel synchronous. Consider this example with promises:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
  let data;
  fs.readFile('foo.txt')
    .then((data) => {
      console.log('data was read')
      data = data;
    })
    .catch((err) => {
      console.err(err.message)
    })
    .then(() => {
      return anotherAsyncThing(data)
    })
  });

You now have three scopes, which are seperated, and if you want access a variable across them you have to specify it in its parent. Works though, but it becomes more concise and elegant with async/await:

1
2
3
4
5
6
  try {
    const data = await fs.readFile('foo.txt')
    anotherAsyncThing(data)
  } catch (err) {
    console.log(err.message)
  }

We can make use of proper error handling! Although i would argue that Javascript does not have proper error handling, but that’s another story.

Now we got the situation in Javascript, but I wanted to write about python, which also has implementations of an event loop. There are quite a couple around but never became “first-class-citizen” until version 3.4 when asyncio was put into the standard library. You may argue it is slower than the Javascript event loop implementation, but actually there is an implementation relying on the same event loop library as NodeJS: uvloop. It is a simple drop in replacement being 100% API compatible, so you can continue to use the official docs of python which helps a lot.

But now: How do you do async with python? Let me show you a little piece:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import asyncio
import random

TREES = [
    'OAK',
    'ALDER',
    'CATALPA',
    'CYPRESS',
    'WALNUT'
]

CARS = [
    'TRUCK',
    'COMBI',
    'SEMI',
    'ROADSTER',
    'COUPE'
]

async def printTree():
    while True:
        await asyncio.sleep(0.5)
        print(random.choice(TREES))
        await asyncio.sleep(0.5)

async def printCar():
    while True:
        print(random.choice(CARS))
        await asyncio.sleep(1)

loop = asyncio.get_event_loop()
all_tasks = asyncio.wait([printTree(), printCar()])
loop.run_until_complete(all_tasks)

This program won’t terminate because it executes two while True loops indefinitly, where printTree and printCar run concurretly! You might notice the three lines on the bottom, where we explicitly start an event loop. That’s because python does not run in an event loop by default like NodeJS does. So you have to create one explicitly put your futures/awaitables inside. Await won’t work outside an event loop since python still is synchronous by default.

You could argue that you could avoid that little extra overhead of managing your own event loop and just go with NodeJS, but I would say it is definitely worth it. I would say since I learned how to actively manage concurrency myself I find my self in race condition fuckups a lot less. Also I always would trade an extra bit of complexity for the slim syntax and the excellent exception system of Python, which in Javascript is pretty poor. But like I said, that is a whole different story.