subprocess vs sh vs plumbum for Running Commands in Python

A script that calls other programs from Python has three common options: the standard library's subprocess, and the third-party sh and plumbum that make commands feel like Python code. They differ in verbosity, error handling and portability. The same task, counting lines of Python files, is written three ways below. The versions are Python 3.12, sh 2.0 and plumbum 1.8; the standard approach is explained in running commands with subprocess.

subprocess

import subprocess

files = subprocess.run(
    ["git", "ls-files", "*.py"],
    check=True, capture_output=True, text=True,
).stdout.split()
total = subprocess.run(
    ["wc", "-l", *files], check=True, capture_output=True, text=True,
).stdout.splitlines()[-1]
print(total)

Everything is explicit: arguments as a list, no shell, a decision about errors with check=True, and text mode. It is longer than the alternatives, and a pipeline needs two calls or the Popen class with pipes, as in piping with subprocess. Its strength is that it is in the standard library, works on Windows and macOS, and has no surprises: what you pass is what runs.

sh

import sh

files = sh.git("ls-files", "*.py").split()
print(sh.wc("-l", *files).splitlines()[-1])

try:
    sh.false()
except sh.ErrorReturnCode as e:
    print("exit", e.exit_code, e.stderr)

print(sh.sort(sh.uniq(sh.cat("data.txt"), "-c"), "-rn"))

sh resolves attribute names to programs on PATH, so sh.git(...) runs git, and the output comes back as a string-like object. A non-zero exit raises ErrorReturnCode_N with the output attached, which makes errors loud by default. Nested calls form a pipeline. It is concise and works only on Unix. Names that are not valid Python identifiers need sh.Command("ssh-keygen"), and a program that shares a name with an attribute of the module can collide.

plumbum

from plumbum import local

git = local["git"]
files = git("ls-files", "*.py").split()
chain = local["cat"]["data.txt"] | local["sort"] | local["uniq"]["-c"] | local["sort"]["-rn"]
print(chain())
(git["log", "--oneline"] > "log.txt")()

plumbum builds command objects you can store, combine and call later. The | operator pipes, > redirects, and local.cwd and local.env are context managers for working directory and environment. It runs on Windows and supports remote execution over SSH with the same API. The operator style takes a little getting used to, and the library is less actively developed than the other two.

Security and errors

None of the three uses a shell unless you ask, which is why the argument lists above are not vulnerable to injection through file names. The danger is subprocess.run(cmd_string, shell=True) with user input, explained in subprocess shell injection. All three can hang on a program that waits for input or fills its pipe; use timeouts, covered in subprocess timeouts.

Streaming, large outputs and timeouts

The convenience calls capture the whole output in memory, which is fine for a short listing and a problem for a command that prints gigabytes. With subprocess, read from Popen.stdout in a loop, line by line or in chunks, and process each piece as it arrives. Set a timeout on run and handle TimeoutExpired, which kills the child. The sh library offers _iter=True to iterate over output lines and _timeout for limits, and plumbum has popen() returning an object like the standard one. For commands that run for a long time, use a thread or asyncio.create_subprocess_exec, which lets a program wait on several processes at once without blocking.

A subtle trap is a full pipe. If a child writes a lot to standard error while the parent reads only standard output, the child's pipe buffer fills and it blocks, and the program hangs. run with capture_output handles both streams, and a hand-written Popen loop must either merge the streams with stderr=STDOUT or read them from separate threads.

Environment, working directory and shells

Each library lets you set the working directory and environment for a call. In subprocess use cwd= and env=, remembering that passing env replaces the whole environment, so copy os.environ and add to it: env={**os.environ, "LC_ALL": "C"}. Without PATH the child may not find programs. Plumbum's context managers set both for a block of commands. Run through a shell only when you need shell features such as globbing or pipes, and then quote values with shlex.quote. If you can use a list of arguments, do so.

Portability and dependencies

The standard library is always there and behaves the same on every platform with minor differences. The third-party libraries are pure Python and install quickly, but they are extra dependencies to pin, update and audit. The sh library does not support Windows, since it relies on Unix process features, which rules it out for tools that run on developer laptops of mixed types. Plumbum works on Windows with limitations. If the script is an internal tool on Linux servers, that is acceptable. If it ships to others, the dependency-free choice is safer.

Testing code that runs commands

To test logic that calls programs, put the calls behind one small function and replace it in tests. Replacing the library's call with a mock tells you only that the call was made. A better test runs a real command that exists everywhere, such as echo or python -c, and asserts on the real result, including a failing command to exercise the error path. Keep tests that depend on a particular tool, like git, marked and skipped when it is absent, and run them in CI where it is installed.

Choosing for a team

Pick one approach per codebase and document why. For infrastructure scripts maintained by several people, the standard library keeps the dependency list short, and a ten-line helper that wraps run gives the same convenience as the third-party packages. Choose sh or plumbum when the script is mostly command orchestration and the team values the shorter syntax, and accept that every new contributor learns one more API. Whatever the choice, make failures loud, log the command and the status, and keep user-provided values out of shell strings.

Migrating from shell strings

Older scripts often build a command string and run it with shell=True. To migrate, split the string into a list of arguments, with shlex.split as a first step for constant commands, and replace each embedded variable with a list element. Pipelines become either two calls connected by passing the first call's output as the input of the second, or a Popen pair sharing a pipe. Work through the script one command at a time, comparing the output before and after with the same inputs. The change removes the quoting problems and makes each argument visible to a reviewer, which is where injection bugs hide in the old form.

Handling return codes uniformly

Whichever library you use, agree on how a non-zero status is treated. A wrapper that logs the command, the status and the last lines of standard error, then raises a single project exception type, lets callers handle failure in one way. It also gives a place to add a retry or a redaction rule for secrets later without touching every call.

Where this does not apply

If the script is mostly command glue with little Python logic, a Bash script is shorter than any of these, as in Python versus Bash for DevOps scripts. For deployment to many machines, a configuration tool replaces ad hoc commands. A new dependency also means packaging and supply-chain review; for a small tool, the standard library is easier to keep. Read the subprocess documentation, the sh documentation and plumbum documentation.