How to Use Python Pexpect for Interactive Automation

Pexpect is the Python counterpart of Expect: it starts a program in a pseudo-terminal, reads what it prints, and types replies. Python makes the logic around the prompts easier than Tcl, with real error handling and data structures. The examples use Pexpect 4.9 on Python 3.12; the idea and the Tcl original are described in Expect for interactive programs.

A first session

import pexpect

child = pexpect.spawn("python3 -q", encoding="utf-8", timeout=10)
child.expect(r">>> ")
child.sendline("2 + 2")
child.expect(r">>> ")
print(child.before.strip().splitlines()[-1])
child.sendline("exit()")
child.expect(pexpect.EOF)
child.close()
print("exit status:", child.exitstatus)

4
exit status: 0

expect() waits until the pattern, a regular expression, appears in the output. Everything printed before the match is in child.before, and the matched text is in child.after. The pattern is a regex, so characters such as . and $ need escaping, which trips up many first scripts. encoding="utf-8" makes the values strings instead of bytes.

Branch on what the program says

import os, pexpect

child = pexpect.spawn("ssh [email protected] uptime", encoding="utf-8", timeout=20)
i = child.expect([
    "Are you sure you want to continue connecting",
    "[Pp]assword:",
    pexpect.EOF,
    pexpect.TIMEOUT,
])
if i == 0:
    child.sendline("yes")
    child.expect("[Pp]assword:")
    i = 1
if i == 1:
    child.sendline(os.environ["SSH_PASS"])
    child.expect(pexpect.EOF)
elif i == 3:
    raise SystemExit("timed out")
print(child.before.strip())

expect accepts a list and returns the index of the pattern that matched, which turns an unpredictable conversation into a small state machine. The list always contains TIMEOUT and EOF, or a missing prompt raises an exception. Reading the password from the environment keeps it out of the source; for SSH specifically, keys are better, as discussed in SSH config shortcuts.

Debug with a log

import sys
child = pexpect.spawn("./installer.sh", encoding="utf-8")
child.logfile_read = sys.stdout        # echo what the program prints
child.logfile_send = open("sent.log", "w")   # keep a record of what we typed

When a script hangs, the first question is what the program last printed. logfile_read answers it. Keep logfile_send out of shared directories if secrets are typed, since it records them. Most hangs are a pattern that does not match because of an extra space, a colour code or a prompt that changed.

Helpers for common cases

out = pexpect.run("ls -l /etc/hostname", encoding="utf-8")
from pexpect import pxssh

pexpect.run() runs a command and returns its output, optionally answering prompts through an events dictionary. pxssh wraps a login and prompt handling for SSH. The standard subprocess module is a better fit when no terminal is needed, see subprocess versus sh and plumbum.

How the pseudo-terminal changes the conversation

A program that detects a terminal behaves differently from one that reads a pipe. It prints prompts, enables colour, reads the password without echo and may page its output. Pexpect gives the child a pseudo-terminal, so the program believes a person is typing, and this is the reason the library can drive passwd, sudo and installers that refuse to read from a pipe. It is also the reason the output contains carriage returns and escape sequences. Lines end in \r\n, so a pattern anchored with $ may not match where you expect, and the text in child.before often starts or ends with a stray \r. Strip it, or match with \r\n in the pattern.

Echo is the other effect. The terminal driver echoes what the script sends, so the next expect sees the typed command in the stream before the program's own response. A pattern that matches a word the script just typed will match the echo and return too early. Anchor on text that only the program prints, such as a prompt, or turn echo off with child.setecho(False) when the program does not need it.

Timeouts and what a failure looks like

Every expect call has a timeout, 30 seconds by default, and when it passes without a match Pexpect raises pexpect.TIMEOUT. The exception message includes the buffer and the patterns tried, which is usually enough to see the mismatch. Treat a timeout as information: it means the program printed something unexpected or nothing at all. Raising the timeout to a large number to make the error go away hides the cause and makes the next failure slow.

Setting timeout=None waits forever, which is the right choice only when the program legitimately takes a long time, such as a build, and even then a generous finite value gives a clean failure instead of a stuck job. If one step is slow, pass a larger timeout to that single expect call and keep the default elsewhere.

At the end of a session, call child.close() and read child.exitstatus and child.signalstatus. If the child is still running, close() sends a hangup and then kills it if needed. Skipping this step leaves zombie processes in a long-running service. When the program is expected to exit by itself, finish with child.expect(pexpect.EOF) so that the exit status is available.

Interactive hand-off and terminal size

child.interact() connects the child to the user's terminal, which is handy for a script that automates the login and then hands control to a person, for example an SSH session after the password step. Press the escape character, Ctrl-] by default, to return to the script. The child's window size matters too. Many full-screen programs wrap output at the terminal width, and a default of 80 columns can split a long path across two lines, so that a pattern that expects it on one line fails. Pass dimensions=(24, 200) to spawn to give it a wider screen.

Secrets, logs and testing

Anything typed with sendline goes through the pseudo-terminal, not through the process's command line, so it does not appear in ps. That is better than putting a password in an argument, though the script still needs to obtain the secret from somewhere safe, such as an environment variable supplied by a secrets manager. Do not enable logfile or logfile_send in production with prompts that contain secrets, and rotate any log that was written during debugging.

Testing a Pexpect script is easier when the target program is replaceable. Point the script at a small stand-in that prints the same prompts, run the conversation in a unit test, and assert on the captured transcript. That catches the common failures, such as a changed prompt string after an upgrade, without needing the real service, and it documents the expected exchange in code that the next maintainer can read.

Common failure patterns

Three failures account for most broken Pexpect scripts. The first is a pattern that is almost right: a prompt that ends in a space, a trailing colon the script forgot, or colour codes between words. Print child.before and child.buffer at the point of failure and compare them with the pattern byte for byte. The second is a race with output that arrives in pieces; expect returns as soon as the pattern matches, so the rest of the text may still be in transit and is picked up by the next call. Always expect the last thing a step prints, not the first. The third is sending input too early, before the program is ready to read it, which loses characters when the program flushes its input buffer. Waiting for the prompt before every sendline avoids it.

Where this does not apply

If the program has flags for non-interactive use, use them: subprocess.run([...], input=...) is simpler and more reliable. Pexpect depends on pseudo-terminals, so it does not work in the same way on Windows, where pexpect.spawn is unavailable. Scripts that type secrets into prompts are only as safe as the account that holds them. The API is documented in the Pexpect API reference, and the subprocess module documentation covers the alternative.