Debugging Shell Scripts From set x All the Way to strace

Debugging a shell script is the same discipline as debugging anything else: reproduce the failure, shrink it, form a hypothesis, test it and fix the cause rather than the symptom. What is different about shell is the tooling and the kinds of mistakes. The language has no compiler, so syntax errors surface only when the line is reached. Types do not exist, so a number and a filename are both strings. And the real work happens in other programs, so many failures are not in the shell code at all. This guide works through the tools in the order you should reach for them, using one real buggy script traced to its cause, on Bash 5.2.15, ShellCheck 0.9.0 and strace 6.1 in Debian 12. The scripts being debugged are written as in the Bash scripting tutorial.

The sections cover the debugging mindset, static checks, tracing with set -x, structured error reporting with trap and caller, the classes of bug that recur, system-level tracing with strace, timing and performance, and debugging scripts that run under cron, over SSH and in containers.

Start with a failing case you can repeat

A bug you cannot reproduce is a bug you cannot fix. Before touching the code, capture a command line, an input and an environment that make the script fail every time, and write them down as a script of their own. A small reproduction turns a vague "it breaks sometimes" into a test. It also tells you whether the failure depends on the input, the environment, the time or the machine, which decides where to look.

Shrink the case. Remove arguments, then lines of input, then whole sections of the script, running it after each cut and keeping the cut only if the failure remains. When the failure disappears after a cut, the removed part was involved. This is bisection by hand, and it is faster than reading a long script and guessing. For failures that appear in a long pipeline, run the pipeline stage by stage and look at the output of each. For failures that depend on state, such as a file that exists on one host and not another, record that state with ls -l, env and id alongside the failing output.

The sample bug

This short script is meant to total the sizes of all log files in the current directory. It has a bug that shows up only for one kind of file name, which is how most shell bugs behave.

#!/usr/bin/env bash
count=0
for f in *.log; do
  size=$(stat -c %s $f)
  count=$((count + size))
done
echo "total: $count"

With a directory containing c.log (six bytes) and an empty file named a b.log, the script prints this.

stat: cannot statx 'a': No such file or directory
stat: cannot statx 'b.log': No such file or directory
total: 6

The error mentions files called a and b.log, neither of which exists, and the total looks plausible, so the script reports a wrong answer without failing. We will use this case in each section to see what each tool shows.

Check syntax with bash -n

bash -n reads the script and reports syntax errors without running any command. It catches unclosed quotes, missing fi or done and misplaced parentheses. It cannot catch anything that depends on runtime values.

printf 'if true; then\necho hi\n' > bad.sh
bash -n bad.sh; echo "rc=$?"

bad.sh: line 3: syntax error: unexpected end of file
rc=2

The line number points at the end of the file because that is where Bash noticed the if was never closed, not where you forgot the fi. For an unclosed quote or a missing done, work backwards from the reported line to the last construct that opened a block. An editor with bracket matching and a ShellCheck plugin shows the same problems as you type. Run bash -n in a pre-commit hook; it takes milliseconds and prevents committing a script that cannot parse.

Lint with ShellCheck

ShellCheck is a static analyzer for shell. It finds the bugs that are valid syntax and wrong in practice, and it explains each in a page of the project wiki. Running it on the sample script identifies the bug before any tracing.

shellcheck buggy.sh

In buggy.sh line 4:
  size=$(stat -c %s $f)
                    ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
  size=$(stat -c %s "$f")

That is the whole diagnosis: the unquoted $f was split on the space, so stat received two arguments. ShellCheck reports severities of error, warning, info and style, and -S warning hides the lower ones. The codes worth knowing by heart are SC2086 (unquoted expansion), SC2046 (unquoted command substitution), SC2164 (cd without a guard), SC2155 (declare and assign together mask a return value), SC2181 (testing $? instead of the command) and SC2034 (unused variable, often a typo). Its output depends on the shell named in the shebang or given with -s, so a wrong shebang yields wrong advice. The tool and its checks are described on the ShellCheck wiki, and the testing guide shows how to run it in CI.

Trace with set -x

When static checks find nothing, watch the script run. set -x makes Bash print each command to standard error, after expansion, with a prefix. Run the whole script with bash -x script.sh, or turn it on for a region with set -x and set +x.

bash -x buggy.sh 2>&1 | head -12

+ count=0
+ for f in *.log
++ stat -c %s a b.log
stat: cannot statx 'a': No such file or directory
stat: cannot statx 'b.log': No such file or directory
+ size=
+ count=0
+ for f in *.log
++ stat -c %s c.log
+ size=6
+ count=6
+ echo 'total: 6'

The trace shows precisely what happened. The line ++ stat -c %s a b.log has two words after %s, though the loop variable held one file name, so the splitting is visible. Each plus sign is a nesting level: one for a command in the script, two for a command inside a command substitution. Note that size= is empty after the failure, and the next line adds nothing to the count; the script carried on because nothing stopped it. This is the main benefit of tracing: it turns an invisible expansion into a visible line.

Make the trace readable with PS4

The prefix is the value of the variable PS4, expanded for each line, so it can hold the script name, the line number, the function and the time. A longer script produces hundreds of trace lines, and a prefix that says where each came from is the difference between a usable trace and a wall of text.

PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '
set -x
echo "starting"

+ t.sh:3:main: echo starting
starting

Assign PS4 inside the script, before set -x. In this Debian 12 container, setting PS4 in the environment of bash -x script.sh, whether inline or exported, left the prefix at a plain plus sign, while the assignment within the shell took effect, so do not rely on passing it from outside. Put the assignment behind a debug switch, such as [[ ${DEBUG:-} ]] && { PS4=...; set -x; }, so the trace turns on with DEBUG=1 ./script.sh and costs nothing otherwise. Add $EPOCHREALTIME to the prefix to get microsecond timestamps for finding slow steps.

Send the trace to a file

By default the trace goes to standard error, mixed with the script's own error messages and with whatever the caller does to that stream. Bash lets you redirect it to a dedicated file descriptor through BASH_XTRACEFD, which keeps the trace separate and complete.

exec 5>trace.log
BASH_XTRACEFD=5
set -x
echo hi
ls /nonexistent_dir

hi
ls: cannot access '/nonexistent_dir': No such file or directory

+ echo hi
+ ls /nonexistent_dir

The first block is what the terminal showed and the second is the content of trace.log. This matters for scripts run under cron or from other programs that capture standard error, where an unsolicited trace would pollute output that something else parses. Remember that a trace prints the expanded values of every variable, so it will contain any secret that passed through a command line. Never leave tracing on in production logs; the security guide covers why in the shell script security guide.

Print lines as they are read with set -v

set -v (verbose) prints each line of input as the shell reads it, before expansion. Used with -x, you see the original text and then the expanded command, which helps when you suspect the shell parsed a line differently from how you read it.

bash -v -c 'x=1
echo "$x"' 2>&1

x=1
echo "$x"
1

It is most useful for here documents and multi-line constructs, and for checking which lines of a sourced file were read. For everyday work -x carries the information you need; reach for -v when the question is what the shell saw, not what it did.

Report failures with trap ERR

A script that fails with only an exit status is hard to diagnose, especially when it runs unattended. The ERR trap fires when a command fails, and inside the handler $LINENO and $BASH_COMMAND give the line and the command. Add -E (errtrace) so that functions and subshells inherit the trap.

#!/usr/bin/env bash
set -eEuo pipefail
trap 'echo "failed: line $LINENO: $BASH_COMMAND (exit $?)" >&2' ERR
f() { false; }
f
echo unreachable

failed: line 4: false (exit 1)

The handler named the exact command and line, then set -e ended the script with status 1. Put this in a shared header for every script you run unattended and the next failure explains itself in the log. Capture $? as the first thing the handler does, because any command in the handler resets it. The trap fires only for failures that set -e would act on, so the exceptions of -e (conditions, && lists, and the like) are exceptions here too.

Print a stack with caller

The caller builtin prints the line, function and file of a call at a given depth in the call stack. Looping over increasing depths prints a full stack trace, which is invaluable once a script is split into functions and the failing line is three calls deep.

trace() { local i=0; while caller "$i"; do (( i++ )); done; }
a() { b; }
b() { trace; }
a

4 b trace.sh
3 a trace.sh
5 main trace.sh

Each row is the line number, the function name and the file: b was called from line 4, a from line 3, and main at line 5 made the first call. Call this from your ERR handler to turn "failed at line 42" into the path that led there. Related arrays are available directly: FUNCNAME, BASH_SOURCE and BASH_LINENO hold the names, files and lines of the active call stack, in the order innermost first.

Step through with the DEBUG trap

The DEBUG trap runs before each simple command. It is the closest thing the shell has to single-stepping, and with a handler that pauses for input it gives you an interactive stepper in a few lines.

trap 'echo "[$LINENO] $BASH_COMMAND"' DEBUG
x=1
y=2

[2] x=1
[3] y=2

trap 'read -r -p "[$LINENO] $BASH_COMMAND > " _ < /dev/tty' DEBUG

The second handler waits for Enter before each command, reading from the terminal so it does not consume the script's own input. Combine it with shopt -s extdebug for finer control. For a full debugger with breakpoints there is bashdb, but it is rarely installed and ties you to a specific Bash version; most problems yield to set -x and the traps above.

Pipelines and PIPESTATUS

The exit status of a pipeline is that of its last command, so a failure earlier in the chain disappears. Bash records every stage's status in the array PIPESTATUS, which you must read immediately, before running another command.

false | true | false; echo "${PIPESTATUS[@]}"
bash -c 'false | true; echo "${PIPESTATUS[*]}"'

1 0 1
1 0

With set -o pipefail the pipeline fails if any stage does, but you lose the detail of which one, and some stages legitimately exit non-zero, such as grep finding nothing or a producer killed by SIGPIPE (status 141) when head closes the pipe early. When debugging, print PIPESTATUS after the pipeline in question. Split long pipelines into steps writing to temporary files to look at each intermediate result; it is slower, and it removes guesswork.

Variables that vanish in subshells

A frequent puzzle is a variable that is set in a loop and empty afterwards. Pipelines, command substitutions and parentheses run in subshells, whose changes are lost when they end.

n=0
echo x | { n=5; }
echo "n=$n"

n=0

To find it, add echo "$BASHPID $$" at the top of the suspect region: $$ stays the main shell's PID in subshells, while $BASHPID changes, so differing values prove a subshell. The fix is to avoid the pipe, with a redirection from a file or process substitution, or to enable shopt -s lastpipe where job control is off. A related symptom is a variable that works in the shell but is empty in a child process, which means it was never exported.

V=1; bash -c 'echo "[$V]"'
export V; bash -c 'echo "[$V]"'

[]
[1]

Exit codes that mislead

An exit status can tell you where to look, if you know the conventions. 0 is success. 1 is a general error, and tools use it for their own purposes. 2 usually means misuse of the command. 126 means the file exists but cannot be executed: a missing executable bit, a directory, or a binary for another architecture. 127 means the command or interpreter was not found. Values above 128 mean a signal: 130 is SIGINT, 137 is SIGKILL (often the out-of-memory killer), and 143 is SIGTERM.

./crlf.sh; echo "rc=$?"
./ns.sh; echo "rc=$?"
bash -c 'exit 7'; echo "rc=$?"
dmesg 2>/dev/null | grep -i 'killed process' | tail -2

bash: line 59: ./crlf.sh: cannot execute: required file not found

The first message is the classic symptom of Windows line endings: the shebang is read as #!/bin/bash followed by a carriage return, so the kernel looks for an interpreter literally named with that character and reports that a required file was not found. The script file exists and the interpreter exists, which is why the message confuses people. Convert the file with tr -d '\r' < in > out or dos2unix, and prevent it with a .gitattributes rule. A script with no shebang at all runs, but under whatever shell the caller used to start it, which is a common cause of "different results from different launchers".

Where set -e does not help

set -e is no substitute for debugging because it has gaps that hide the failing command. A function called in a condition runs with -e disabled, so errors inside it do not stop it.

set -e; f() { false; echo after; }; f && echo ok

after
ok

The false failed and execution went on, because the function sat on the left of &&. Another trap is local x=$(cmd), where the exit status of local, always 0, replaces the status of the substitution.

f() { local x=$(false); echo "rc=$?"; }; f
g() { local x; x=$(false); echo "rc=$?"; }; g

rc=0
rc=1

Declare and assign on separate lines whenever the status matters; ShellCheck reports this as SC2155. When a script keeps going after a visible error, check these two before anything else.

Loops that stop after one iteration

A loop that reads lines and runs ssh or another command that reads standard input will stop after the first line, because the command consumes the rest of the input.

printf 'h1\nh2\nh3\n' > hosts
while read -r h; do cat >/dev/null; echo "got $h"; done < hosts

got h1

Here cat swallowed the remaining two lines, as ssh would. Give the command its own input: ssh -n, or < /dev/null on the command, or read the list with a different file descriptor using while read -r h <&3; do ...; done 3< hosts. The symptom, a loop that runs once with no error, is easy to recognize once seen and easy to spend an hour on if not.

Glob and word-splitting bugs

Many failures reduce to the shell's expansion rules. When a command behaves as if it received the wrong arguments, print them exactly. A tiny function that echoes each argument in brackets shows the real word boundaries.

args() { printf '[%s]\n' "$@"; }
x="a b"
args $x
args "$x"
declare -p x

[a]
[b]
[a b]
declare -- x="a b"

Use it in place of the suspect command to see what it would receive. declare -p prints a variable with its attributes in a form that shows hidden characters, quotes and array structure, which beats echo for finding a trailing space or an array that is really a string. For unprintable characters in a value, printf '%q' and od -c reveal them, such as the carriage return left by a file with Windows line endings.

Find which command is really running

A command may not be the one you think it is. An alias, a function or an earlier directory on PATH can shadow it, and a script runs with different aliases and a different PATH from your interactive shell. type -a lists every candidate in the order the shell would pick, and command -v prints the one that wins.

type -a ls
command -v python3
hash -r

ls is /usr/bin/ls
ls is /bin/ls

The shell caches the location of commands it has run, so after installing or moving a program, hash -r clears the cache; a stale cache explains "command not found" for a program that is plainly present. In a script, print the resolved path and the version of every tool it depends on at startup in debug mode. A surprising number of "bugs" turn out to be a different version of sed, date or grep on the machine where it failed.

Reproduce under the failing environment

Many scripts work interactively and fail under cron, a systemd unit, a CI runner or ssh host command, because the environment differs. Reproduce the failure by running the script in an environment that is as bare as the failing one.

env -i HOME="$HOME" PATH=/usr/bin:/bin /usr/local/bin/backup.sh
env -i bash --noprofile --norc ./script.sh
ssh host 'env | sort' > remote-env.txt
diff <(env | sort) remote-env.txt

env -i clears the whole environment, so anything the script silently relied on, such as a tool in /usr/local/bin or a variable set in ~/.bashrc, is now missing, and the error appears. Comparing the working and failing environments with diff shows the difference directly. Also compare the working directory, the user, the umask, the locale and whether standard input is a terminal: [[ -t 0 ]] is false under cron, and scripts that prompt then hang or fail. The scheduler-specific differences are described in the guide to cron and systemd timers.

When the problem is outside the shell: strace

If the shell code looks right and a command still fails, the cause is often in how the programs interact with the system: a file that is not where it should be, a permission refusal, a library that cannot load, a network call that hangs. strace records the system calls a process makes, and with -f it follows the children, which for a shell script means every command it runs.

strace -f -e trace=execve,openat -o st.txt bash slow.sh
grep -E 'execve|nonexistent' st.txt | cut -c1-120

147   execve("/usr/bin/bash", ["bash", "slow.sh"], 0xffffd4ec3810 /* 8 vars */) = 0
148   execve("/usr/bin/cat", ["cat", "/etc/hostname"], 0xaaaaf46bf3e0 /* 8 vars */) = 0
149   execve("/usr/bin/ls", ["ls", "/nonexistent"], 0xaaaaf46c22b0 /* 8 vars */) = 0
150   execve("/usr/bin/grep", ["grep", "-q", "root", "/etc/passwd"], 0xaaaaf46c22b0 /* 8 vars */) = 0

Each line begins with the process ID, so you can see which child ran which command, with the exact arguments after expansion and the exact path chosen from PATH. This is a definitive answer to "which program ran and with what", independent of anything the shell prints. Restrict the trace with -e trace= to the calls you care about or the output is overwhelming. The manual is strace(1). In containers, tracing requires the SYS_PTRACE capability and often a relaxed seccomp profile, as it did for these examples.

Find a missing file with strace

When a program reports a vague failure, a trace of file opens shows precisely which path it tried and what the kernel answered. An ENOENT means the path does not exist, EACCES means permission denied.

strace -f -e trace=openat bash -c 'cat /etc/missing.conf' 2>&1 | grep -E 'missing|ENOENT'

openat(AT_FDCWD, "/etc/missing.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
cat: /etc/missing.conf: No such file or directory

The same technique finds which configuration file a tool actually reads out of several candidates, which library it fails to load, and which socket it tried to connect to. For a program with a long list of attempts, filter on ENOENT\|EACCES. The absence of an expected openat is informative too: if the program never tries to open your config file, the problem is how it is told where the file is, not what the file contains.

Diagnose a hang

A script that hangs is waiting on something. Attach to the stuck process, or the one under it, and see which system call it sits in. First find the deepest child with pstree -p, then trace it. A process blocked in read on descriptor 0 is waiting for input; one in wait4 is waiting for a child; one in connect or recvfrom is waiting on the network; one in futex is waiting on a lock or thread.

pstree -p "$(pgrep -f stuck.sh | head -1)"
sudo strace -p 4242 -f -tt
ls -l /proc/4242/fd
cat /proc/4242/wchan; echo

171   05:58:40.256267 wait4(-1,  <unfinished ...>
172   05:58:40.256522 execve("/usr/bin/sleep", ["sleep", "1"], 0xaaab0df644b0 /* 8 vars */) = 0
172   05:58:41.259335 +++ exited with 0 +++
171   05:58:41.259389 <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 172

The timestamps from -tt show the parent shell waiting a full second in wait4 while a child ran sleep 1, which is a complete account of where the time went. /proc/PID/fd lists what the process has open, which reveals a pipe whose other end nobody is reading, a classic deadlock. A hang at the very start of an unattended script is very often an interactive prompt, from sudo, ssh asking about a host key or apt asking to continue, waiting for input from a terminal that does not exist.

Count the work: strace -c

The summary mode counts system calls and the time in each, which locates cost without reading a trace. A script that is slow because it starts too many processes shows up as a large count of execve and clone.

strace -c -f -o summary.txt bash slow.sh
head -8 summary.txt

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 58.45    0.004132         688         6         3 wait4
  5.52    0.000390           9        43           mmap
  4.51    0.000319          79         4           execve
  4.05    0.000286           5        55        12 newfstatat
  3.03    0.000214          71         3           clone
  2.65    0.000187           7        24           mprotect

This three-line script ran four execve calls: bash itself and one for each of cat, ls and grep. Scaled up, a loop over ten thousand lines that calls grep or sed on each line would show forty thousand. The remedy is to replace the loop's subprocesses with shell builtins or to process all the input in a single awk, as discussed in the grep, sed and awk guide. The wait time is mostly the parent waiting for children, so look at the counts and the errors, not just the percentages.

Measure time inside the script

Before optimizing, measure. The shell keyword time reports real, user and system time for a command or a block. To find which section of a script is slow, record timestamps at key points, using the builtin EPOCHREALTIME (Bash 5.0 and later) which avoids starting date.

time (for i in $(seq 1 2000); do /bin/true; done)
time (for i in $(seq 1 2000); do :; done)
t0=$EPOCHREALTIME
./step-one.sh
t1=$EPOCHREALTIME
printf 'step one: %.3fs\n' "$(echo "$t1 - $t0" | bc -l)"

real	0m0.623s
real	0m0.003s

Starting /bin/true two thousand times cost 0.62 seconds in the container, while the same loop with the builtin : cost 3 milliseconds: roughly 300 microseconds per process, about two hundred times the cost of a builtin. That ratio explains most shell performance problems. When the numbers are noisy, repeat the measurement and take the minimum. Time the whole script from outside as well, with time ./script.sh, so you do not optimize something that is a tiny share of the total.

Put debug support into the script

Make the script help you debug itself. A DEBUG or -v switch should turn on tracing and verbose logging without code changes. A --dry-run option should print the destructive commands instead of running them. Log with levels to standard error, so that standard output carries only results. And assert the assumptions that the script relies on, early, with clear messages.

: "${DRY_RUN:=0}" "${DEBUG:=0}"
(( DEBUG )) && { PS4='+ ${BASH_SOURCE##*/}:${LINENO}: '; set -x; }

run() {
  if (( DRY_RUN )); then
    printf 'DRY: %q ' "$@"; printf '\n'
  else
    "$@"
  fi
}
log() { printf '%s [%s] %s\n' "$(date +%T)" "$1" "${*:2}" >&2; }
need() { command -v "$1" >/dev/null || { log ERROR "missing: $1"; exit 127; }; }

need rsync
run rsync -a --delete -- "$src/" "$dest/"

$ DRY_RUN=1 ./sync.sh
DRY: rsync -a --delete -- /srv/site/ /backup/site/

The %q format prints each argument quoted so you can paste the line into a shell and get exactly what the script would have run. A dry run is also the safest way to test a change to a destructive script. Keep the debug code in the script permanently; it is cheap and it will be needed again.

Debug scripts over SSH and in containers

A remote command runs with a non-interactive shell that reads no profile, so its environment differs from your login. To debug, run the same command with tracing on the remote side and capture both streams. A containerized script has another set of differences: a minimal image may lack the tool, use a different shell as sh, or run as a different user.

ssh host 'bash -x /opt/app/deploy.sh' 2>&1 | tee deploy-trace.txt
ssh host 'echo "$PATH"; command -v node; id'
docker run --rm -it --entrypoint sh myimage -c 'id; echo "$PATH"; command -v bash'
docker exec -it mycontainer sh

Overriding the entrypoint with a shell is the quickest way into an image whose normal entrypoint fails at once. Check which shell the script's shebang resolves to in the image: a script that uses Bash features and starts with #!/bin/sh fails on Alpine, as the portable shell guide explains. For hosts you reach through a bastion or with unusual options, the SSH guide shows how to inspect the effective client configuration.

Other tools worth knowing

ltrace traces library calls, useful for programs that fail inside a shared library. lsof shows what files and sockets a process has open. /proc/PID/environ, /proc/PID/cmdline and /proc/PID/status expose a running process's environment, arguments and state, with NUL separators that tr '\0' '\n' turns readable. dmesg and journalctl -k show the kernel's view, including out-of-memory kills. bash --posix and dash test portability. inotifywait watches for file changes. script records an entire terminal session. And tee in a pipeline saves an intermediate result without disturbing the flow.

tr '\0' ' ' < /proc/4242/cmdline; echo
tr '\0' '\n' < /proc/4242/environ | sort | head
dmesg | grep -i 'out of memory'
cmd1 | tee /tmp/stage1.out | cmd2

Each of these answers a specific question quickly, and none needs installation beyond a standard distribution. Learn two or three well and reach for them by habit, rather than adding echo statements and rerunning, which is slower and changes the program you are testing.

Common mistakes while debugging

• Changing code before reproducing the failure. You cannot tell whether the change fixed it.
• Debugging in your login shell. Test with env -i and the same user as the failing run.
• Reading $? after another command. Capture it on the line that follows the command.
• Trusting set -e. It skips conditions, lists and local assignments.
• Printing with echo. It hides whitespace and treats some data as options; use printf '[%s]\n' or declare -p.
• Leaving tracing on. It leaks secrets and floods logs.
• Fixing a symptom with a retry or sleep. A timing fix hides a race.
• Ignoring stderr. Redirecting it to /dev/null while debugging removes the evidence.
• Optimizing without measuring. Count processes and time sections first.

The common thread is replacing belief with observation. Every tool above exists to show what the shell and the kernel actually did, so that the next change addresses a fact.

Read the error message properly

Shell error messages are terse, but each one narrows the cause sharply when you know its usual meaning. The wording below is from Bash 5.2; other shells phrase the same faults differently.

Two of these, integer expression expected and unary operator expected, almost always point at an unquoted or empty variable in a single-bracket test, and quoting plus a check that the value is set resolves them. The message also names the line where the shell noticed the problem, which for syntax errors can be well after the cause. When the line number looks wrong, read upward to the last opened block.

A second case: the script that never ran

Some bugs leave no output at all, because the script did not run when expected. A nightly job that has been silently failing for a month is more common than a loud one. Suppose a report is scheduled at 02:00 and no reports have appeared. Work through the layers in order, ruling each out with evidence, not assumption.

systemctl list-timers --all | grep report
journalctl -u report.service --since '3 days ago' --no-pager | tail -20
grep -i report /var/log/syslog | tail
ls -l --time=ctime /srv/reports | tail -3
sudo -u reports env -i HOME=/srv/reports PATH=/usr/bin:/bin /usr/local/bin/report.sh; echo "rc=$?"

The first command checks that the schedule exists and when it last fired. The second reads what the job printed, which is the first place a real error message appears. The third checks cron and the mail system if cron is used. The fourth looks at whether the output files are being produced at all. The last runs the job as the right user in a bare environment, the most informative single test, because it reproduces the commonest causes: a missing PATH entry, a permission problem or a relative path. Only when each layer has been cleared does it make sense to read the script line by line. The reasons these layers fail are laid out in the cron and systemd timers guide.

Write a regression test as you fix

A bug that has been found once will return unless something checks for it. The moment you have a reproduction, turn it into an automated test, run it to confirm that it fails, apply the fix, and run it again to confirm that it passes. The test records the lesson in executable form. For the sample bug the test creates a file name with a space and asserts that the total is correct.

#!/usr/bin/env bats
@test "totals sizes of files whose names contain spaces" {
  cd "$BATS_TEST_TMPDIR"
  printf 'abc' > "a b.log"
  printf 'de'  > "c.log"
  run bash "$BATS_TEST_DIRNAME/../total.sh"
  [ "$status" -eq 0 ]
  [ "$output" = "total: 5" ]
}

Before the fix this test fails, because the script prints a smaller total and complains on standard error; after quoting "$f" it passes. Keeping such tests turns each debugging session into a permanent improvement. The framework, with setup, mocking and CI, is covered in the testing guide.

Handle flaky failures

A failure that happens one run in twenty is the hardest kind. The causes are few: a race between background jobs or between two copies of the script, a dependency that is slow or intermittently down, input that varies (a file arriving while the script reads it), or state left behind by an earlier run. The approach is to make the failure frequent enough to study, then record enough to tell the causes apart.

for i in $(seq 1 200); do
  ./flaky.sh > "run.$i.out" 2>&1 || echo "run $i failed (rc=$?)"
done
ls run.*.out | wc -l
grep -L 'ok' run.*.out | head

Run it a few hundred times in a loop, with the trace going to a per-run file, and compare a failing run's trace to a passing one with diff. Look at what differs first: timestamps, ordering of output from parallel jobs, an environment variable. If two copies might overlap, add a lock and see whether the failure vanishes, which also proves the diagnosis. If a dependency is the cause, log its response time and status on every call so that a slow or failing moment is on the record. Resist the temptation to add a sleep: it moves the race and does not remove it.

Read the code in the order the shell does

Many bugs come from reading a line the way a human does instead of the way the shell does. The shell processes a command line in a fixed sequence: it splits the input into tokens and applies quoting, expands braces, then tilde, parameters, command substitution and arithmetic, then splits the results into words, then expands globs, and only then runs the command with its redirections. A rule that follows from that order is that a redirection target is expanded last and a brace expansion happens before variables are read, so {1..$n} does not count to n. When a line does something you did not expect, walk through it in that order and write down the value after each step. Doing this on paper once teaches you to see the steps at a glance, and it explains the whole family of surprises around quoting, splitting and ordering without a tool. The Bash manual's description of shell expansions lists the order.

n=3
echo {1..$n}
eval echo {1..$n}
seq 1 "$n" | paste -sd' '

The first command prints {1..3}: brace expansion ran before the variable was replaced, found no number to count to, and left the braces for the later step to leave alone; the second works only because eval forces a second round of expansion, which is why it is both a fix and a hazard; the third is the plain answer. The point is not this particular trick but the habit of asking in which phase a value becomes known.

Keep a short debugging routine

Having an order for the first ten minutes saves more time than any single tool. Read the error message completely and note the line. Run shellcheck and bash -n. Reproduce with the smallest input and the failing environment. Turn on tracing with a useful PS4 and read the trace around the first sign of trouble, not the last. Print the real arguments of the failing command with the bracket helper. If the shell looks innocent, trace system calls with strace -f and look for ENOENT, EACCES and the call where a hung process sits. Fix the cause. Add a test. Remove any temporary debugging code, and keep the permanent switches. The routine is short on purpose, and following it every time is what turns debugging from a mood into a method.

Debugging pipes, buffering and output order

Output that arrives late, in the wrong order or all at once is usually a buffering issue, not a bug in the logic. When a program writes to a terminal, its output is line buffered, so each line appears as it is produced. When it writes to a pipe or a file, the C library switches to block buffering, holding output until a buffer of several kilobytes fills or the program exits. So ./slow-tool | grep ready shows nothing for minutes, and a log that looks fine on the terminal is empty when redirected. The streams standard output and standard error are also buffered separately, so their lines can interleave in surprising order when both go to the same file.

./slow-tool | stdbuf -oL grep ready
stdbuf -oL ./slow-tool | tee run.log
./slow-tool 2>&1 | cat
unbuffer ./slow-tool | grep ready
python3 -u script.py | tee out.log

stdbuf from coreutils changes the buffering of programs that use the C library by preloading a shim, so it works for most commands and not for statically linked ones. unbuffer from the expect package gives the program a pseudo-terminal so that it believes it is interactive. Many tools have their own flag, such as grep --line-buffered, awk with fflush() and Python's -u. When two streams interleave badly, merge them with 2>&1 on the command so they share one buffer and keep their order.

Finding what a script does to the filesystem

A script that modifies files in ways you cannot predict is easier to understand if you watch it. Three tools help. strace -e trace=file lists every path the process and its children touch. inotifywait watches a directory for changes and prints each event. A before-and-after comparison of a directory tree, using find with checksums or a version control commit of the directory, shows exactly what changed.

strace -f -e trace=openat,unlink,rename,mkdir -o fs.txt ./deploy.sh
grep -v ENOENT fs.txt | awk '{print $2, $3}' | sort | uniq -c | sort -rn | head
inotifywait -m -r -e create,modify,delete,move /srv/app &
git -C /etc init -q 2>/dev/null; git -C /etc add -A; ./configure-server.sh; git -C /etc diff --stat

The last line uses a repository in /etc as a recorder: commit the state, run the script, and the diff is the full list of changes. Tools such as etckeeper do this automatically. The approach answers "what did that installer actually do?" without reading it, which is also a good way to review a script from someone else before you run it as root.

Bisecting a script

When a script that used to work stops working and the cause is not obvious, bisection finds the change responsible. If the script is in Git, git bisect automates it: mark a known good and a known bad commit, give it a command that returns 0 for good and non-zero for bad, and it narrows to the first bad commit in a logarithmic number of steps. The test command can be a short script that runs the failing case.

git bisect start
git bisect bad HEAD
git bisect good v1.3.0
git bisect run ./reproduce.sh
git bisect reset

#!/usr/bin/env bash
# reproduce.sh: exit 0 when the bug is absent, 1 when present, 125 to skip
cd "$(git rev-parse --show-toplevel)" || exit 125
out=$(./total.sh 2>&1) || exit 125
[[ $out == "total: 5" ]]

The status 125 tells bisect to skip a commit that cannot be tested, for instance one that does not build. When the change that broke things is in the environment, not the code, bisect the environment: compare package versions with dpkg -l before and after, and downgrade one at a time. The same idea works by hand on a script without history: delete half the script, see whether the failure persists, and repeat.

Debugging in the middle of a run

Sometimes you want to look around inside a script at the point of failure. A line that starts an interactive shell where the script stands lets you inspect variables and try commands, then continue by exiting it. Because the script's variables are not visible to a child shell unless exported, use a function that dumps them or source your debugging aids.

debug_shell() {
  echo "debug shell at ${BASH_SOURCE[1]}:${BASH_LINENO[0]}; type exit to continue" >&2
  declare -p >&2
  bash --norc -i </dev/tty >/dev/tty 2>&1
}
trap 'debug_shell' ERR

The dump of declare -p prints every variable and function, so you can see the state at that moment. On failure the ERR trap opens the shell attached to the terminal, so it works even when standard input is redirected. Add a check that standard input is a terminal, and never leave this in production code: a script waiting on a shell nobody can reach is a hang. For scripts that run unattended, dump the same state to a file in the trap and read it afterwards.

Environment differences, systematically

A frequent cause of "works here, fails there" is a difference in tools and versions. Capture a fingerprint of the environment where it works and where it fails, and compare. The fingerprint should cover the shell and its version, the versions of the tools the script calls, the locale, the time zone, the user, the umask, and the relevant parts of the environment.

{
  echo "== shell";   bash --version | head -1
  echo "== user";    id; umask
  echo "== locale";  locale | sort
  echo "== tools"
  for t in awk sed grep find tar curl jq; do printf '%s: ' "$t"; "$t" --version 2>&1 | head -1; done
  echo "== path";    echo "$PATH" | tr ':' '\n'
  echo "== os";      uname -a; cat /etc/os-release 2>/dev/null | head -3
} > env-report.txt 2>&1
diff env-report.local.txt env-report.remote.txt

Run the same block in both places, through the same mechanism that runs the failing script, such as cron or ssh. The diff usually has a short list of differences, and one of them is the cause. BusyBox tools print their version differently and may not accept --version; the report notes that and moves on. Keep a script like this in your toolbox; it turns an argument about whose machine is different into a table.

Debugging shell code produced by other tools

Many scripts are not written by hand: they come from configuration management templates, CI definitions, Dockerfile RUN lines, Makefile recipes and package maintainer scripts. Debugging them has two extra steps, getting the rendered text and seeing how the outer tool wraps it. A Dockerfile RUN line runs under /bin/sh -c, so Bash features fail on images where sh is dash or ash. A Makefile recipe line runs in its own shell, so a cd on one line does not affect the next, and dollar signs must be doubled. A CI step may run with -e and -o pipefail on or off depending on the platform's default.

docker build --progress=plain --no-cache . 2>&1 | tail -40
make -n deploy
make SHELL='bash -x' deploy
ansible-playbook site.yml -vvv --check --diff
rendered=$(envsubst < template.sh.in); printf '%s\n' "$rendered" | bash -n

Print the rendered script and run bash -n and ShellCheck on it, since problems introduced by templating, such as a variable that rendered as empty, are visible there. In CI, add a first step that prints the shell and its options, echo "$0"; set -o | grep -E 'errexit|pipefail', to learn how steps are run before you rely on it. Whichever tool wraps the shell, reproduce the problem outside it first, with the same text and the same shell, and the earlier techniques in this guide apply unchanged.

Knowing when to stop debugging and rewrite

Some scripts are not worth saving. If you spend an hour on a bug in a script whose logic you cannot follow, which has no tests and which three people have patched without understanding, the efficient fix may be a rewrite with tests that capture its observable behavior. Signs that the time has come: nested quoting through three layers, state held in dozens of global variables, parsing of structured data with regular expressions, copy-pasted blocks that differ in small ways, and a growing list of platform special cases. The method that works is to characterize first: run the old script on real inputs, save the outputs, write tests that assert them, then write the replacement against the tests, in the language that suits the job.

for input in samples/*.in; do
  ./old.sh "$input" > "golden/$(basename "$input" .in).out" 2>&1 || true
done
for input in samples/*.in; do
  diff -u "golden/$(basename "$input" .in).out" <(./new "$input" 2>&1) || echo "differs: $input"
done

The golden outputs are the specification you did not have. Differences are either bugs in the new version or bugs the old one hid, and each deserves a decision. Python or Go is often the better home once the script needs structures and tests. Debugging skills still apply after a rewrite, since the failure modes of processes, files and signals belong to the system and not to the language.

Where this does not apply

Large programs. Past a few hundred lines of logic, a script is better debugged as a program in a language with a real debugger, breakpoints and unit tests. Rewriting is often cheaper than extending the tooling around shell; the Linux command line guide lists what the shell is good at, and the rest should move.

Heisenbugs in concurrency. Tracing changes timing, so a race in background jobs may disappear under set -x. Use logging to a file with timestamps and process IDs, and design for idempotence and locking instead of chasing the exact interleaving.

Production systems. strace slows its target, sometimes by an order of magnitude, and can alter timing. Attach to production processes briefly and with a narrow filter, or prefer lower-overhead tools such as perf and eBPF-based tracers.

Restricted environments. Containers without ptrace, hardened hosts and managed platforms may not allow strace at all. Fall back to logging, set -x to a file and reproducing the case somewhere you control.

Non-Bash shells. Features like BASH_XTRACEFD, caller, PIPESTATUS and the ERR trap are Bash-specific. dash has set -x and PS4 but little else, and zsh has its own equivalents.

What to read next

Once a bug is fixed, keep it fixed with a test: the guide to testing shell scripts with Bats and ShellCheck turns each reproduction into a regression test. If the cause was unquoted input, the shell script security guide shows the wider class. For the underlying language, the Bash scripting tutorial covers the constructs that went wrong here.

Primary sources used here: the Bash manual on the set builtin, the Bash manual on shell variables, the Bash manual on builtins including trap, the Bash builtins reference including caller, strace(1), proc(5), signal(7), execve(2), the ShellCheck SC2086 page and the ShellCheck SC2155 page.