Python Subprocess Called Process Error and Exit Status

The traceback ends in subprocess.CalledProcessError: Command '[...]' returned non-zero exit status 1 and tells you little else. The exception object has more, if you ask for it. This post explains what each field holds and how to decide between letting it fail and handling it. The code is Python 3.12 on Linux; the broader topic is in capturing subprocess output.

The exception and its fields

import subprocess

try:
    subprocess.run(
        ["ls", "/nonexistent"],
        check=True, capture_output=True, text=True,
    )
except subprocess.CalledProcessError as e:
    print("code:", e.returncode)
    print("cmd:", e.cmd)
    print("stderr:", e.stderr.strip())

code: 2
cmd: ['ls', '/nonexistent']
stderr: ls: cannot access '/nonexistent': No such file or directory

Without capture_output=True, e.stdout and e.stderr are None, and the program's error message went straight to the terminal, which is fine interactively and invisible in logs. Capturing makes it available for your own log line or for a retry decision. If you capture only stdout, use stderr=subprocess.STDOUT to merge streams.

Exit codes as answers

r = subprocess.run(["grep", "-q", "ERROR", "app.log"])
if r.returncode == 0:
    print("errors found")
elif r.returncode == 1:
    print("no errors")
else:
    raise RuntimeError(f"grep failed with {r.returncode}")

For some programs a non-zero code is not a failure. grep and diff return 1 for "no match" and "different", and 2 or above for real errors, and test-like tools behave similarly. Use check=False and interpret the code yourself. The meaning of codes is covered in the Linux exit codes list, and the shell behaviour in Bash exit codes.

Signals show up as negative numbers

import signal
r = subprocess.run(["sh", "-c", "kill -9 $$"])
print(r.returncode, signal.Signals(-r.returncode).name)

-9 SIGKILL

When the child dies from a signal, Python reports -N instead of the shell's 128+N. A -9 often means the kernel's out-of-memory killer, see the OOM killer, and -11 is a segmentation fault. A shell wrapper reports the same events as 137 and 139 instead.

Where to catch it

def run(cmd, **kw):
    try:
        return subprocess.run(cmd, check=True, capture_output=True, text=True, **kw)
    except subprocess.CalledProcessError as e:
        log.error("command failed: %s (exit %s)\n%s", e.cmd, e.returncode, e.stderr)
        raise

A thin wrapper that logs and re-raises keeps the stack trace and the evidence. Catch the exception at the level that can do something about it, such as retrying or skipping an item, and let the rest propagate, so a script fails loudly rather than continuing with bad data. A timeout is a different exception, TimeoutExpired, covered in subprocess timeouts.

What the exception does not tell you

A CalledProcessError says the child exited with a non-zero status, and nothing else. It does not say why, and it does not say whether the failure is permanent. The same exit code 1 can mean "no matches found", "file not found", "permission denied" or "internal error" depending on the program, because the code is a convention between the tool and its authors and not a shared vocabulary. Reading stderr is how you find out, and that is why the capture options matter more than the exception class itself.

Capturing has a cost. With capture_output=True, Python reads everything the child prints into memory, so a command that writes gigabytes of output turns a diagnostic aid into a memory problem. For commands with large or unbounded output, redirect stdout to a file object, and capture only stderr, which is normally short. For long-running commands that should show progress, do not capture at all: let the output go to the terminal, and accept that the exception will carry only the command and the code.

Text mode, encoding and the second exception

With text=True, Python decodes output using the locale's encoding, and a program that prints bytes that are not valid in that encoding raises UnicodeDecodeError instead of returning. That error is raised from inside run, before your except clause for CalledProcessError has any say, so a handler for one does not cover the other. Pass errors="replace" when the output is only for logging, or set encoding explicitly when the tool documents one. Working in bytes and decoding later is also valid when the output is binary, such as a tarball from tar -c.

A single call can raise several exception types, and it helps to know which is which. FileNotFoundError means the executable was not found. PermissionError means it was found but not executable. CalledProcessError means it ran and returned non-zero. TimeoutExpired means it ran too long and was killed. A handler that catches only the third will let the other three escape, which is often right, since they signal a broken environment rather than a failed task.

Deciding between check=True and returncode

The two styles express different intentions. check=True says "failure is exceptional here", and it shortens the code because success is the only path written out. Checking returncode says "failure is one of the expected outcomes", and the code lists them. Choose by asking whether a person reading the script would be surprised by a non-zero status. A git command in a deployment script should raise. A grep used as a test should not.

A hybrid is useful in wrappers. Call run(..., check=False), look at the code, and raise your own exception with a clearer message when it is outside the list you expect. The custom error can carry the command, the code and the last lines of stderr, which gives the on-call engineer something to read at three in the morning. Truncate the stderr text to a few kilobytes before putting it into an exception or a log line, because a runaway tool can otherwise produce a message that is larger than the logging system accepts.

Retries and idempotence

Catching the exception to retry is reasonable only if the command is safe to repeat. Fetching a file is. Creating a database row is not, unless the command is written to be idempotent. Retry only on codes or messages that mean "try again later", such as a network timeout, and give up at once on the ones that mean "your arguments are wrong". A loop that retries every failure three times with a sleep only makes a typo take longer to report.

A wrapper that keeps the evidence

In a larger program, wrapping the call in one helper makes the policy visible in a single place. The helper decides what is captured, how much output is kept, which codes are acceptable and what the error says. Callers then pass the command and, when needed, a list of codes that count as success. The sketch below follows that idea.

import subprocess

class CommandError(RuntimeError):
    def __init__(self, cmd, code, stderr):
        super().__init__(f"{cmd[0]} exited {code}: {stderr[-500:].strip()}")
        self.cmd, self.code, self.stderr = cmd, code, stderr

def run(cmd, ok=(0,), timeout=60):
    r = subprocess.run(cmd, capture_output=True, text=True,
                       errors="replace", timeout=timeout)
    if r.returncode not in ok:
        raise CommandError(cmd, r.returncode, r.stderr)
    return r.stdout

The helper keeps only the last 500 characters of stderr in the message, which is where tools usually print the reason, while the full text stays on the exception object for anyone who wants it. Callers that treat code 1 as a normal outcome pass ok=(0, 1). The timeout argument is always set, because a command with no limit is a way to hang a service on a stuck dependency.

Where this does not apply

If the command could not be started at all, the exception is FileNotFoundError or PermissionError, not this one, see No such file or directory from subprocess. check_output and check_call raise it too, and run(check=True) is the modern form. See the CalledProcessError documentation and the signal module.