How to Use asyncio to Run Shell Commands Concurrently

When a script runs hundreds of short commands, threads work but cost memory and complexity. Asyncio runs them all on one thread, waiting for output without blocking, and the API is compact. The cost is that everything in the path must be async-aware. This post shows the working pattern and the limits that keep it from overwhelming a machine. It is the alternative to concurrent futures. Examples use Python 3.13 on Linux.

One command

import asyncio

async def run(*cmd):
    proc = await asyncio.create_subprocess_exec(
        *cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    out, err = await proc.communicate()
    return proc.returncode, out.decode(), err.decode()

code, out, err = asyncio.run(run("uname", "-sr"))
print(code, out.strip())

0 Linux 6.12.0

communicate() reads both pipes while waiting for the process to end, which prevents the deadlock that occurs when a child fills a pipe nobody reads. The arguments are separate strings, so nothing is parsed by a shell. There is also create_subprocess_shell, with the injection risks described in shell injection in subprocess; prefer exec.

Many at once, with a limit

import asyncio

HOSTS = [f"web{n}.example.org" for n in range(1, 41)]
limit = asyncio.Semaphore(10)

async def probe(host):
    async with limit:
        code, out, err = await run("ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5",
                                   host, "uptime")
        return host, code, out.strip()

async def main():
    results = await asyncio.gather(*(probe(h) for h in HOSTS))
    for host, code, out in results:
        print(f"{host:22} {'ok' if code == 0 else 'fail'} {out}")

asyncio.run(main())

All forty coroutines start, but the semaphore lets only ten into the command at a time. Without it, forty SSH processes launch together, which can exhaust file descriptors or trip rate limits on the far side. gather returns results in input order. Add return_exceptions=True so that one failure does not hide the other results.

Deadlines and cleanup

async def run_limited(*cmd, seconds=20):
    proc = await asyncio.create_subprocess_exec(*cmd, stdout=asyncio.subprocess.PIPE)
    try:
        async with asyncio.timeout(seconds):
            out, _ = await proc.communicate()
        return proc.returncode, out.decode()
    except TimeoutError:
        proc.kill()
        await proc.wait()
        return None, ""

asyncio.timeout is available from Python 3.11; earlier versions use asyncio.wait_for. A timeout cancels the waiting, not the process, so the handler must kill the child and then wait() to reap it. Skipping that leaves a zombie. For process trees, start with start_new_session=True and use os.killpg as in subprocess timeouts.

Where this does not apply

Asyncio does not speed up CPU-bound work, and one blocking call inside a coroutine stalls everything. For a handful of commands the overhead is not worth it; a plain loop or a thread pool is simpler to read. Windows needs the proactor event loop for subprocess support, which is the default since Python 3.8. Inside notebooks, an event loop already runs, so use await directly instead of asyncio.run.

Why the event loop changes how you read output

With asyncio.create_subprocess_exec the child runs while your coroutines continue, and its pipes are read through the event loop. The danger is the same as with threads: if you start a process with stdout=PIPE and await process.wait() without reading, a child that writes a lot fills the pipe and blocks, and wait() never returns. Use await process.communicate(), which reads both streams until the process ends, or read the streams in tasks you start before waiting. For large output, read line by line with async for line in process.stdout so memory stays small. Pass arguments as separate strings to create_subprocess_exec, which does not use a shell. create_subprocess_shell exists, and it brings the quoting and injection problems back, so use it only for fixed strings.

On Windows, subprocess support requires the proactor event loop, which is the default on current Python versions. Older code or custom policies that select the selector loop fail with NotImplementedError when creating a subprocess.

Limits, cancellation and cleaning up children

Starting a thousand processes at once exhausts file descriptors or memory, so bound the concurrency with an asyncio.Semaphore or a fixed number of worker tasks pulling from a queue. Pick the limit from what the machine and the programs can take, not from the number of jobs. A timeout is applied with asyncio.wait_for around the communicate call, and on expiry the task is cancelled but the child process is still running. Cancelling the coroutine does not kill the child. Catch the TimeoutError and call process.kill(), then await process.wait() to reap it, so no zombie remains. In a finally block, check process.returncode is None and terminate whatever is still alive, which covers cancellation from outside, such as Control-C that cancels the main task.

If a command spawns its own children, kill the process group. Start the child with start_new_session=True and send the signal with os.killpg, because killing only the direct child can leave the rest running.

Collecting results and failures

asyncio.gather with return_exceptions=True collects every result and exception in order, so one failure does not hide the others, and without that flag the first exception propagates while the rest keep running unobserved. From Python 3.11, asyncio.TaskGroup offers structured concurrency: if any task fails, the others are cancelled and the errors are raised together as an exception group, which makes the cleanup path more predictable. Return a small record for each command with its arguments, exit status, output and duration, and print a summary at the end. A summary that lists failures with their command lines is far more useful than a stream of interleaved output from fifty processes. Remember to read standard error as well, since a child that writes only to stderr and has its stderr piped but unread will block exactly as described above.

When asyncio fits and when it does not

Running subprocesses concurrently is a good use of asyncio, because starting and waiting on a process is waiting work, and one event loop can supervise hundreds of them with little overhead. It fits scripts that already use asyncio for network calls and want to run commands in the same flow, and tools that stream output from several commands at once. It is a poor fit for a script that runs three commands in a row. A plain subprocess.run is simpler and easier to debug, and a thread pool gives the same parallelism with ordinary blocking code. Pick asyncio when you need many concurrent operations with fine control over timeouts and cancellation, and pick threads when you want to keep the code simple.

Mixing styles causes trouble. Calling a blocking subprocess.run inside a coroutine freezes the whole loop until it returns, so every other task stalls. If you must call blocking code, run it with asyncio.to_thread, available from Python 3.9, which moves it to a worker thread and keeps the loop responsive.

Debugging helps with the debug mode. Run the program with PYTHONASYNCIODEBUG=1 or asyncio.run(main(), debug=True), and the loop logs slow callbacks and coroutines that were never awaited, which are the two mistakes that account for most broken asyncio scripts. A warning about a coroutine that was never awaited usually points to a missing await in front of a call, and the fix is a single word.

Finally, test the failure paths on purpose: a command that does not exist raises FileNotFoundError when the process is created, a command that exits with an error returns a nonzero code that you must check yourself, since the asyncio API does not raise for it, and a command that never ends needs the deadline logic described above.

See the asyncio subprocess documentation and asyncio timeouts.