Python Subprocess Pipe Deadlock and Hanging Processes

A script that works with small output and hangs on big output is the classic symptom of a pipe deadlock. The process is not slow: it is blocked on a write that nobody will ever read. Three causes cover most hangs, and each has a short fix. The examples use Python 3.12 on Linux; the related timeouts are in subprocess timeouts.

Cause 1: waiting while the pipe is full

import subprocess

p = subprocess.Popen(["yes"], stdout=subprocess.PIPE)   # produces endless output
p.wait()      # never returns: the pipe buffer is full and nobody reads it

A pipe has a fixed buffer, 64 KB on Linux by default. The child blocks when it is full, and wait() blocks until the child exits, so neither side moves. The documentation warns about this. Use communicate(), which reads both pipes while waiting, or run(), which calls it for you:

r = subprocess.run(["make", "-j8"], capture_output=True, text=True, timeout=1800)
print(r.stdout[-2000:])

If the output is huge, do not capture it in memory: send it to a file with stdout=open("build.log", "wb"), or read it line by line as it is produced, described in streaming subprocess output.

Cause 2: the child is waiting for input

subprocess.run(["apt-get", "install", "foo"], stdin=subprocess.DEVNULL, timeout=600)
subprocess.run(["ssh", "host", "uptime"], stdin=subprocess.DEVNULL,
               env={**os.environ, "GIT_TERMINAL_PROMPT": "0"})

A program that asks a question, for a password, a confirmation or a host key, waits on its inherited stdin, which in a script has no one behind it. stdin=subprocess.DEVNULL makes reads fail immediately, so the program errors out. Tools often have their own flags, such as -y or -o BatchMode=yes. For SSH, BatchMode disables prompts. To answer prompts on purpose, see Pexpect.

Cause 3: a grandchild holds the pipe

import os, signal, subprocess

p = subprocess.Popen(["sh", "-c", "sleep 300 & echo started"],
                     stdout=subprocess.PIPE, start_new_session=True, text=True)
try:
    out, _ = p.communicate(timeout=5)
except subprocess.TimeoutExpired:
    os.killpg(p.pid, signal.SIGKILL)
    out, _ = p.communicate()
print(out)

The shell exits at once, but the backgrounded sleep inherited the write end of the pipe, so communicate() waits until it too closes. run(timeout=...) kills only the direct child, which would not help here. start_new_session=True puts the child in a new session and process group, so os.killpg signals the child and everything it started. Background daemons started from a script should redirect their output to a file to avoid holding the pipe.

Diagnose a hang that is happening now

pgrep -af 'make|yes'
ls -l /proc/<pid>/fd
cat /proc/<pid>/wchan; echo
strace -p <pid>

A process blocked in write on a pipe is waiting for the reader, and one blocked in read on fd 0 is waiting for input. The tools are covered in debugging scripts with strace, and the symptoms in debugging a hanging script.

The communicate method and why it exists

The standard library has a method built to avoid the deadlocks above. communicate() writes the given input, reads standard output and standard error until both reach end of file, using threads or select internally, and then waits for the process. It cannot deadlock on pipe buffers, which is why run() uses it. The limit is memory, since all output is collected into one object, and the fact that it returns only after the child closes its streams, which a grandchild can delay, as the third case shows. For a command with output too large to hold, read the stream in a loop: for line in p.stdout: processes lines as they come, while a second thread or a redirect to a file handles standard error, because reading one pipe and ignoring the other can reproduce the original deadlock in reverse.

Merging the streams avoids the second pipe altogether. Passing stderr=subprocess.STDOUT sends error output into the same pipe as standard output, so one reading loop sees both in the order the child wrote them. The price is that you can no longer tell which line was an error, which is acceptable for a build log and wrong for a program whose standard output is data.

Timeouts that actually stop the work

When the timeout passes, run() kills the child with SIGKILL and raises TimeoutExpired. That stops the direct child and leaves its descendants running, and they keep the pipe open. A tool such as make, a shell script or a browser starts many children, so a timeout on the parent leaves the real work going. Start the command in its own session with start_new_session=True, and on timeout kill the whole group with os.killpg(p.pid, signal.SIGTERM), wait briefly, then send SIGKILL. Python 3.11 and later also kill the process group in some cases, but depending on that is less clear than doing it explicitly. Log which command timed out and how much output it produced before the limit, since that is the first clue to why.

Buffering inside the child

Sometimes the program is not stuck at all and only appears silent, because it buffers its output. Most programs use full buffering when writing to a pipe instead of a terminal, so a long-running tool shows nothing for minutes, then dumps a block. A Python child prints nothing until its buffer fills unless run with -u or with PYTHONUNBUFFERED=1. For C programs, stdbuf -oL command asks for line buffering, and it works for programs that use the standard C library buffering and not for ones that manage their own. When a pseudo-terminal is acceptable, running the child under pexpect or the pty module makes it believe it is interactive, so it flushes per line. Before concluding that a command is hung, check whether it is producing output into a file or a socket with strace.

A checklist when a call hangs

Work through five questions. Is something reading every pipe that was created? Is the child waiting for input that will never come, and is stdin closed or redirected? Does the command start background children that hold the pipes? Is there a timeout, and does it kill the group? Is the child buffering and merely slow to show output? Answering them in order locates almost every hang, and the answer to the first usually points to a one-line fix.

Making hangs visible in production

Add a timeout to every external call, even one that has never hung, and log the command name when the timeout fires. Export a counter of timeouts per command to your metrics system. A growing count is the early warning that a dependency has changed or slowed, and it points straight at the command involved, which turns an overnight mystery into a morning fix. Pair timeouts with a limit on retries so that a persistent hang becomes a clear failure instead of a loop.

Where this does not apply

If a process is just doing a lot of work, a timeout will kill good work; set it from measured run times. With asyncio, use asyncio.create_subprocess_exec and read the streams concurrently, as in asyncio subprocesses. On Windows, process groups and signals work differently, so start_new_session and killpg do not apply. The warnings are in the Popen.wait documentation and Popen.communicate.