Python: why do we need async for?

In Python,  async for uses asynchronous generator to iterate over values. But how is it different from a regular for that uses a generator that yields awaitables?
TL;DR: not much, except some peculiarities when ending the loop. The answer is in this StackOverflow post, but it lacks examples. I give concrete examples in my blog post.

Suppose we iterate over values from an async generator:

async def asyncGetValues():
    await ...something...
    yield value

async for x in asyncGetNextValues():
    print (value)

How is it different from:

def getAwaitables():
    async def asyncGetValue():
        await ...something...;
        return value
    yield asyncGetValue()

for get_x in getAwaitables():
    print(await get_x)

Besides having to write a little bit more code, the difference is in stopping the loop. Suppose we want to iterate over numbers that come from a remote source, and want to stop when we received a zero. We need to wait for each number to come, and we don’t know in advance whether the result would be zero. I will emulate it using sleep() and random numbers:

async def asyncGetNumberStream():
    while True:
        await asyncio.sleep(1)
    n = random.randint(0, 4)
    if n==0:
        return
    yield n

We can use it as follows:

async for n in asyncGetNumberStream():
    print(f"Received {n}")

The loop will exit when we receive a zero, but zero will never be returned to the user and printed.

Let’s rewrite it using a regular generator:

# generates a stream of Awaitable[int]
def getNumberStream():
    async def getNextNumber():
        await asyncio.sleep(1)
        return random.randint(0, 4)
    while True:
        yield getNextNumber()

Do you see the problem? It’s an infinite generator now, because we don’t know when to stop. We must return the next awaitable before we know whether it will return a zero or some other number. It’s up to the loop’s user now to stop at zero:

for get_n in getNumberStream():
    n = await get_n
    if n==0:
        break
    print(f"Received {n}")

This illustrates the main difference between regular for over awaitables and async for. If we know the number of items in advance, or if the sequence is infinite, there is not much difference. Regular for is a little bulkier, but functionally it’s the same. But if we need to stop on some dynamically calculated condition, with async for we can embed this condition into the generator, while with regular for the loop will be technically ‘infinite’, and it is the user who will have to determine when to stop.

Leave a Reply

Your email address will not be published. Required fields are marked *