How to Use Python Subprocess With Timeouts and Retries

A script that calls an external command and waits forever is a script that has to be killed by hand. Python's subprocess has a timeout built in, with details that matter: what is killed, what is raised and what is left behind. This post shows the pattern with retries. It extends running shell commands from Python. Examples use Python 3.13 on Linux.

Timeout and check

import subprocess

try:
    result = subprocess.run(
        ["curl", "-fsS", "https://example.org/health"],
        capture_output=True, text=True, timeout=10, check=True,
    )
    print(result.stdout)
except subprocess.TimeoutExpired as exc:
    print(f"timed out after {exc.timeout}s")
except subprocess.CalledProcessError as exc:
    print(f"exit {exc.returncode}: {exc.stderr.strip()}")

When the timeout passes, run kills the child with SIGKILL, waits for it and raises TimeoutExpired. With check=True, a non-zero exit raises CalledProcessError carrying the output, which is the information you need in the log. The command is a list, so no shell is involved and an argument cannot be reinterpreted as syntax. See shell injection in subprocess for why.

Retry with backoff

import time

def run_with_retry(cmd, attempts=4, delay=1.0, timeout=15):
    for n in range(1, attempts + 1):
        try:
            return subprocess.run(cmd, capture_output=True, text=True,
                                  timeout=timeout, check=True)
        except (subprocess.TimeoutExpired, subprocess.CalledProcessError) as exc:
            if n == attempts:
                raise
            wait = delay * 2 ** (n - 1)
            print(f"attempt {n} failed ({exc.__class__.__name__}), retry in {wait:.0f}s")
            time.sleep(wait)

The delay doubles: one second, two, four. The last failure is re-raised so the caller still sees the real error. Retrying hides bugs when the failure is permanent, such as a wrong flag, so restrict it by exit code: if exc.returncode in (1, 2): raise for codes you know are not transient. Add random jitter to the wait when many scripts retry against one service, so they do not return in lockstep.

Kill the whole tree

import os, signal

proc = subprocess.Popen(["bash", "-c", "sleep 300 & wait"], start_new_session=True)
try:
    proc.wait(timeout=5)
except subprocess.TimeoutExpired:
    os.killpg(proc.pid, signal.SIGTERM)
    try:
        proc.wait(timeout=5)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGKILL)

The timeout in run kills only the direct child. A shell script that started other processes leaves them running as orphans. start_new_session=True puts the child in its own process group, and os.killpg signals the group, SIGTERM first to let programs clean up, then SIGKILL. This is Unix-only.

Streaming output with a deadline

run with capture_output buffers everything in memory, which is fine for small output. For a long-running command whose output you want live, use Popen and read lines while checking a deadline with time.monotonic(). See streaming output.

Where this does not apply

A timeout is wall-clock time, so a command that is merely slow is killed along with one that is stuck. Choose a value from measurements, not a guess. Commands that must not be interrupted halfway, such as a database migration, need a graceful path rather than a kill. For many commands at once, concurrent futures and asyncio apply the same ideas in parallel.

What timeout actually does

The timeout argument of subprocess.run starts a timer, and when it expires Python kills the child process and raises TimeoutExpired. The kill is sent to the process it started, and to that process alone. If the command is a shell script that launched other programs, or you used shell=True so the child is a shell, the grandchildren keep running after the shell dies, and may keep the output pipes open, which can make the call hang even after the timeout fires, since Python waits for the pipes to close. This is the most common surprise with timeouts. The dependable approach is to start the child in its own session or process group and kill the group on timeout, or to avoid the shell and run the program directly so there is no layer in between.

Distinguish the timeout from a failing exit. check=True raises CalledProcessError for a nonzero exit, and TimeoutExpired is a separate exception. Catch each on purpose, and decide whether the work is worth retrying: a timeout may be transient, a syntax error never is.

Which failures deserve a retry

Retries help against problems that go away by themselves, such as a busy server, a brief network failure or a lock held for a moment. They hurt against permanent errors, since repeated attempts waste time and may repeat a side effect. Look at the exit status and the output. A command that reports "connection refused" or exits with a documented temporary failure code is a candidate. One that reports bad arguments or a missing file is not. Make the operation idempotent when you can, so a retry after a timeout, where you cannot know whether the first attempt did its work, does no harm. Appending to a file twice or creating the same record twice is the kind of bug that retries introduce.

Use exponential backoff with jitter: wait a short time after the first failure, double it each time up to a cap, and add a random amount so many clients do not retry in step. Limit the attempts and the total time, and log each failure with its reason so a flaky dependency shows up in the logs.

Output, deadlocks and long-running commands

If you capture output with a pipe and the child writes more than the pipe buffer holds, around 64 KB on Linux, it blocks until you read. run() reads while waiting, so it is safe, but code that starts a process with Popen, waits on it and only then reads the pipes will deadlock on large output. Use communicate(), which reads both streams while waiting, and pass the timeout there. After a timeout from communicate(), kill the process and call communicate() again to collect the remaining output and reap the child, otherwise a zombie remains. For commands that run for a long time and print progress, read the pipe line by line in a loop with a separate deadline check, or use a thread or asyncio, and avoid buffering all the output in memory.

Set an environment, a working directory and arguments as a list rather than a string. That keeps the call independent of the caller's state and avoids quoting errors, as the shell-injection post in this series explains.

Putting the pieces together

A reliable wrapper has a small shape. Run the command with a list of arguments and no shell, set a timeout, capture output as text, and check the exit status. On TimeoutExpired, kill the process group, log what was captured so far, and decide whether to retry. On CalledProcessError, include the command, the status and the last lines of standard error in the message, because a bare "returned non-zero exit status 1" costs an hour of guessing. Return a result object that callers inspect, and raise only for conditions the caller cannot handle. Keep the policy, meaning the number of attempts and the delays, in one place, so changing it later touches a single function.

Test the wrapper against a command that sleeps, one that exits with a failure, one that prints a lot of output and one that ignores SIGTERM, since those four behaviors cover the paths where wrappers break. Use kill() after a short grace period when terminate() does not work.

See subprocess.run and the Popen reference.