How to Run a Command on Many Servers With pdsh or ssh

You often need an answer from every machine: which kernel is running, how full is the disk, did the config change reach them. A loop is serial and slow, and the better tools are older than the cloud. This post covers the common ones, ordered by setup cost, and continues the story in replacing ssh loops with Ansible. The examples assume OpenSSH 9.x and pdsh 2.34.

Prepare ssh for batch use

ssh -o BatchMode=yes -o ConnectTimeout=5 web1 hostname

web1

BatchMode=yes fails immediately if key authentication does not work, instead of waiting for a password. ConnectTimeout=5 gives up on a silent host in five seconds. Put both under a Host * entry in your SSH config, along with ControlMaster so repeated runs reuse connections.

pdsh: parallel with host labels

export PDSH_RCMD_TYPE=ssh
pdsh -w web[1-4] 'uname -r'
pdsh -w web[1-4],db1 -f 8 'df -h / | tail -1' | dshbak -c

web1: 6.8.0-45-generic
web2: 6.8.0-45-generic
web3: 6.8.0-41-generic
web4: 6.8.0-45-generic
----------------
web[1-2,4]
----------------
6.8.0-45-generic
----------------
web3
----------------
6.8.0-41-generic

Without PDSH_RCMD_TYPE=ssh, pdsh defaults to rsh on some builds. The host range syntax web[1-4] expands the names. -f limits the fan-out, and the default of 32 is enough to overload a jump host. dshbak -c collapses identical output, so the odd host, here web3 with an older kernel, stands out.

xargs -P: no extra software

printf '%s\n' web1 web2 web3 web4 |
  xargs -P 4 -I{} sh -c 'ssh -o BatchMode=yes -o ConnectTimeout=5 {} "uptime -p" 2>&1 | sed "s/^/{}: /"'

-P 4 runs four at a time. Prefixing each output line with the host name keeps results readable when they interleave. Keep in mind that xargs exits non-zero if any invocation fails, so a failed host marks the whole run; it does not say which one, so capture status per host inside the command. See parallel xargs for tuning the process count.

Report failures

failed=()
for h in $(<hosts.txt); do
  ssh -o BatchMode=yes -o ConnectTimeout=5 "$h" 'systemctl is-active nginx' >/dev/null 2>&1 &
  pids[$!]=$h
done
for pid in "${!pids[@]}"; do
  wait "$pid" || failed+=("${pids[$pid]}")
done
printf 'failed: %s\n' "${failed[*]:-none}"

Background jobs plus wait PID gives you each exit status without extra tools. End with a list of failures, because a host that timed out leaves no output, and its silence is easy to read as success.

Choosing a tool for the job

Three families of tools run commands on many hosts: shell loops over ssh, parallel launchers such as pdsh and xargs -P, and configuration managers such as Ansible. A plain loop is the shortest to write and the slowest to run, because it handles one host at a time. A parallel launcher keeps the same commands and finishes in about the time of the slowest host. A configuration manager adds state, idempotence and reporting, at the cost of learning a tool. Pick the lightest option that fits the job, and move up when a task is repeated often enough that the loop's weaknesses start to hurt.

The question to ask is whether the task changes state. A read-only command such as checking disk space or a package version is safe to run again and again and belongs in a launcher. A change such as editing a configuration file belongs in something that can describe the end state and detect when it is already there.

Timeouts and the unreachable host

The failure mode of batch SSH is the host that hangs, not the host that refuses. Without limits, a connection to a powered-off machine waits for the TCP timeout, which can be over two minutes, and a loop over fifty hosts stalls on each dead one in turn. Set ConnectTimeout to five or ten seconds, ServerAliveInterval so a stuck session is dropped, and BatchMode=yes so a missing key produces an error instead of a password prompt that waits forever. Put these in a dedicated block of the SSH configuration used by the batch job, not on the command line of each call.

Control the connection overhead as well. With ControlMaster auto, ControlPath and ControlPersist, repeated commands to one host reuse a single authenticated connection, so the second command costs milliseconds instead of a full handshake. That matters when a job sends several commands per host.

Output that can be read afterwards

Parallel output interleaves unless each line is labelled. pdsh prefixes every line with the host name; with xargs, wrap the remote command so it prints the host in front, or write each host's output to its own file in a results directory and combine them at the end. Files are the more reliable option for long output, since they survive the run and can be searched, compared and attached to a ticket. Record the exit status for each host in a separate file or line so a summary can be printed, listing exactly which hosts failed and with what code.

Sort the summary by status, not by completion time. A reader wants the failures together at the bottom of the terminal, not scattered among successes. Exit with a non-zero status if any host failed, so that calling scripts and CI notice.

Limits on concurrency and blast radius

Parallelism has side effects on the target environment. Running a package update on all hosts at once can overwhelm a mirror, and restarting a service everywhere at once takes down the service. Cap concurrency with -P in xargs or -f in pdsh, start with a single host as a canary, and wait for a health check before continuing to the rest. Keep the host list in a file under version control, and use groups so that a command can target the staging set before production. Finally, avoid putting secrets into the command string, which is visible in the process list on both ends; read them from a file on the target or pass them on standard input.

Test the whole procedure on a harmless command before using it on a risky one. Run hostname across the fleet and check that every host answers once, that the labels match, and that a deliberately wrong host name shows up in the failure summary with a clear reason. Only then run the real command. This costs a minute and finds configuration mistakes, such as a missing key on one host, while they are still harmless.

Keep a record of each batch run: the command, the host list, the time, the person and the results. When a change causes trouble later, that record answers which machines were touched and in what order.

Where this does not apply

These tools run a command; they do not manage state, handle retries or keep an audit trail. Past a few dozen hosts, or for anything that changes configuration, use Ansible or a similar tool. Never run a destructive command across all hosts at once: try one host, then a group, then the rest. pdsh is usually in distribution repositories and unmaintained on some; check that your build supports -R ssh. See the pdsh project page and the ssh_config manual.