grep sed and awk With Real Examples You Can Copy and Paste

Three programs from the 1970s still do most of the text processing on Linux servers: grep finds lines, sed edits them and awk calculates over them. They all read lines from files or standard input, write lines to standard output, and compose with pipes. Every example below ran on Debian 12 with GNU grep 3.8, GNU sed 4.9 and GNU Awk 5.2.1, against small sample files shown in full so you can reproduce the results. Where mawk, the default awk on Debian, or BSD and BusyBox versions behave differently, the section says so.

The guide is arranged by verb. It sets up the sample data, then covers grep, sed and awk in turn, then how to combine them, three complete worked jobs, and the points at which another tool is the better choice. For the pattern language shared by all three, the regular expressions guide goes deeper than the summaries here.

The sample data

Three small files carry every example. An access log in the common format, an INI-style config, and a CSV of sales.

$ cat access.log
203.0.113.9 - - [05/Nov/2026:10:01:02 +0000] "GET /index.html HTTP/1.1" 200 5120
198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] "GET /api/users HTTP/1.1" 500 87
203.0.113.9 - - [05/Nov/2026:10:01:09 +0000] "POST /api/login HTTP/1.1" 401 45
192.0.2.77 - - [05/Nov/2026:10:02:11 +0000] "GET /missing HTTP/1.1" 404 312
198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] "GET /api/users HTTP/1.1" 502 0
203.0.113.9 - - [05/Nov/2026:10:03:41 +0000] "GET /img/logo.png HTTP/1.1" 200 20480

$ cat app.conf
# main settings
host = db01.internal
port = 5432
; legacy
timeout=30
user = deploy

$ cat sales.csv
region,product,units,price
north,widget,10,2.50
south,gadget,4,10.00
north,gadget,7,10.00
east,widget,12,2.50
south,widget,3,2.50

The addresses come from the documentation ranges reserved in RFC 5737, so they are safe to publish and never refer to a real host. The log has the columns you meet in real servers: client, two unused fields, a bracketed time, the quoted request, the status and the size.

How the three tools differ

Choose by the verb in your task. If the task is "show me the lines that", use grep. If it is "change this text to that", use sed. If it is "for each line, take these columns and total, count or reformat them", use awk. The boundaries blur because each can do some of the others' jobs, but the tool that matches the verb is almost always shorter and clearer.

All three process the input as a stream, one line at a time, without loading the file into memory, which is why they handle multi-gigabyte logs on a small machine. The record separator is a newline by default for all of them, and each lets you change it.

Quote patterns with single quotes

The shell sees the command line first. Characters such as *, $, (, | and \ are meaningful to both the shell and the regular expression, so an unquoted pattern is interpreted by the shell before grep or sed ever receives it. Single quotes stop all expansion.

grep -E ' (404|50[0-9]) ' access.log
grep "$var" file
sed 's/\$HOME/~/' notes.txt

Use single quotes for any literal pattern or script. Use double quotes only when you deliberately want the shell to insert a variable, and remember that the variable's contents then become part of the pattern, so a value containing a dot or a bracket changes what matches. For a variable that must match literally, use grep -F -- "$var", which treats it as a fixed string. A single quote inside a single-quoted string cannot be escaped; close the quote, add an escaped one, and reopen: 'it'\''s'.

Basic and extended regular expressions

All three tools use regular expressions, in two syntaxes. Basic regular expressions (BRE), the default for grep and sed, treat +, ?, |, ( and ) as ordinary characters, and require a backslash to give them special meaning in GNU versions. Extended regular expressions (ERE), selected with -E for grep and sed, and the default for awk, give those characters their special meaning directly. The POSIX specification defines both, and GNU adds extensions.

grep    'ab\+c\|xy' file      # BRE with GNU escapes
grep -E 'ab+c|xy' file        # ERE, same meaning
sed -E 's/(foo|bar)+/X/' file
awk '/ab+c|xy/' file

Prefer -E every time. It reads the way most people expect, it matches what awk and most other languages use, and it removes the backslash soup. The character classes [[:digit:]], [[:space:]] and [[:alpha:]] work in both and are locale-aware. Shorthand such as \d and \w is not POSIX; GNU grep supports \w and \s as extensions but not \d outside -P. The POSIX regular expression chapter is the authoritative reference.

grep: select lines

The simplest use prints every line that contains the pattern. The options you use most are -n for line numbers, -i to ignore case, -v to invert, -c to count, -w to match whole words, -x to match whole lines and -q for quiet, which only sets the exit status.

grep -n ' 50[0-9] ' access.log
grep -c GET access.log
grep -E ' (404|50[0-9]) ' access.log | wc -l

2:198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] "GET /api/users HTTP/1.1" 500 87
5:198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] "GET /api/users HTTP/1.1" 502 0
5
3

Matching ` 50[0-9] ` with surrounding spaces avoids matching a size that happens to contain 500, which is the common mistake of grepping for a bare number. The exit status is 0 if any line matched, 1 if none matched and 2 on error, so if grep -q 'pattern' file; then is the standard test. Note that the quiet flag stops reading at the first match, which makes it fast on big files.

grep: only the matching part

-o prints each match instead of the whole line, one per line. It turns grep into an extractor, and combined with sort | uniq -c it gives frequency tables.

grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' access.log | sort | uniq -c | sort -rn

      3 203.0.113.9
      2 198.51.100.4
      1 192.0.2.77

The pattern is deliberately loose: it matches any four dotted numbers, including values like 999.1.1.1, which is fine for extraction from trusted logs and not fine for validation. Validate with a stricter pattern or a purpose-built tool. When a line contains several matches, -o prints all of them, in order, each on its own line, so a count of lines is a count of matches.

grep: context, files and recursion

Context options show neighbors of a match: -A n after, -B n before, -C n both. File options choose what is searched: -r recurses, --include='*.conf' and --exclude-dir=.git filter, -l prints names of files with matches and -L names of files without.

grep -n -B1 -A1 404 access.log
grep -rl widget .
grep -rn --include='*.sh' 'set -e' /usr/local/bin

3-203.0.113.9 - - [05/Nov/2026:10:01:09 +0000] "POST /api/login HTTP/1.1" 401 45
4:192.0.2.77 - - [05/Nov/2026:10:02:11 +0000] "GET /missing HTTP/1.1" 404 312
5-198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] "GET /api/users HTTP/1.1" 502 0
./sales.csv

Context lines are marked with a dash after the line number and matching lines with a colon, which helps scripts and eyes tell them apart. Without a file argument, grep -r searches the current directory. With one file, grep omits the file name prefix, and -H forces it, which matters when a script feeds grep a variable number of files.

grep: whole words, fixed strings and PCRE

-w requires the match to sit between non-word characters, so it finds port and not ports or sport. -F treats the pattern as plain text with no special characters, which is both safer for user-supplied strings and faster. -P enables Perl-compatible expressions in GNU grep, giving lookahead, lookbehind, non-greedy quantifiers and \K to drop the part matched so far.

echo 'port ports sport' | grep -ow port
grep -oP '"\K[A-Z]+(?= )' access.log | sort | uniq -c

port
      5 GET
      1 POST

The second command extracts the HTTP method: \K discards the opening quote from the match and the lookahead (?= ) requires a space after without consuming it. -P is a GNU extension, absent from BSD grep and BusyBox, and flagged as experimental in the manual. If a script must run on macOS, use -E with a capture group and sed or awk to pull out the group.

grep: remove comments and blank lines

A very common job is to see only the active lines of a config file. Invert the match on comment and empty lines.

grep -vE '^\s*([#;]|$)' app.conf

host = db01.internal
port = 5432
timeout=30
user = deploy

The pattern reads: from the start of the line, optional whitespace, then either a comment character or the end of the line. The \s shorthand works in GNU grep and sed but is not POSIX; for portability, write [[:space:]]*. The same filter is the first step in most config-parsing scripts, because comments and blank lines are noise for every later stage.

grep: speed

grep is already fast, and the choices that matter are about avoiding work. A fixed string with -F skips the regular expression engine, and LC_ALL=C avoids multibyte decoding, which helps most with case-insensitive or Unicode patterns. -m 1 stops after the first match when only existence matters. Use -l or -q so grep can stop early. In a repository, git grep searches tracked files using the index and skips ignored directories, which is usually faster than grep -r.

time grep -c 'error' big.log
time LC_ALL=C grep -cF 'error' big.log

On a 54 MB file of three million short ASCII lines in the Debian 12 container, plain grep -c error took 0.115 seconds and LC_ALL=C grep -cF error took 0.110, so for a simple literal the gain was within noise. An awk program doing the same count took 0.252 seconds. Measure on your data before assuming; the locale and fixed-string options pay off on patterns that are case-insensitive or on multibyte text, and on small files startup dominates. Avoid cat file | grep: it adds a process and a copy, and grep can open the file itself. With several patterns, use one -E 'a|b|c' or a pattern file with -f patterns.txt, not several greps in sequence.

sed: how it works

sed reads input a line at a time into a buffer called the pattern space, runs the script's commands against it, and prints the result. The default command set is small. The commands to know are s to substitute, d to delete, p to print, a, i and c to append, insert and change lines, y to transliterate characters, and q to quit. A command can be limited by an address: a line number, a range, a regular expression or the last line $.

sed 's/ = /=/' app.conf
sed -n '2,4p' app.conf

# main settings
host=db01.internal
port=5432
; legacy
timeout=30
user=deploy
host = db01.internal
port = 5432
; legacy

The first command substituted the first occurrence on each line. The second used -n, which suppresses automatic printing, with the p command on lines 2 to 4. Forgetting -n when you use p prints each selected line twice, a rite of passage. The full command set is in the GNU sed manual.

sed: the substitute command

The form is s/pattern/replacement/flags. The most useful flags are g to replace every match on the line, a number to replace only the Nth, i to ignore case, and p to print when used with -n. The replacement can use & for the whole match and \1 to \9 for captured groups. Any character can replace the slash as the delimiter, which is the neat way to handle paths.

sed -E 's/^([a-z]+) *= *(.*)$/\2 <- \1/' app.conf | tail -4
echo '/usr/local/bin' | sed 's|/usr/local|/opt|'
echo 'a-b-c' | sed 's/-/_/2'

5432 <- port
; legacy
30 <- timeout
deploy <- user

The first command swaps key and value on each assignment line, leaving comment lines alone because they do not match. Using | as the delimiter in the second avoids escaping every slash in the paths. In the replacement, a literal ampersand must be written \& and a newline \n in GNU sed. When the replacement text comes from a shell variable, escape the delimiter, ampersand and backslash in it first, or you will inject sed syntax.

sed: addresses and ranges

An address selects the lines a command applies to. A number picks one line, $ the last, /regex/ every matching line, and first,last a range. A step 1~2 picks every second line in GNU sed. Add ! after the address to invert it.

sed -n '/host/,/timeout/p' app.conf
sed '/^[#;]/d;/^$/d' app.conf
sed '1d;$d' file
sed '5,$!d' file

host = db01.internal
port = 5432
; legacy
timeout=30
host = db01.internal
port = 5432
timeout=30
user = deploy

A regex range starts on the first matching line and ends on the next line that matches the second pattern, and if the end pattern never appears the range runs to the end of the file. That is a handy way to extract a section of a config file, and a dangerous way to delete one if the end marker is missing. Preview with -n ... p before you change a file.

sed: append, insert, change and delete

The one-line commands a, i and c add text after, before or in place of the selected line. In GNU sed they can be written on one line, a text. BSD sed requires the text on the next line, so portable scripts put it there or use a different approach.

sed '/^user/a\role = admin' app.conf | tail -2
sed '/^port/c\port = 6432' app.conf | sed -n 3p
sed '/legacy/d' app.conf | wc -l

user = deploy
role = admin

For multi-line insertions, the portable pattern is a backslash at the end of each line of text. When the text is long, a here document and a second tool are clearer than a crowded sed script. Keep the edit commands simple, and when a script needs more than a few lines, write it to a file and call sed -f script.sed.

sed: edit files in place

sed -i rewrites the file instead of printing. It creates a temporary file, writes the result, and renames it over the original, which means the file's inode changes and hard links to it are broken. Give it a suffix to keep a backup of the original.

cp app.conf t.conf
sed -i.bak 's/^port.*/port = 6432/' t.conf
diff t.conf.bak t.conf

3c3
< port = 5432
---
> port = 6432

On Linux with GNU sed, -i with no suffix works. On macOS and the BSDs, it requires a suffix argument, possibly an empty string written as a separate word (sed -i '' 's/a/b/'), which GNU sed then misreads as the script. The one portable form is attached suffix: -i.bak. After a successful edit, delete the backups if you do not want them. Always test an in-place edit without -i first, then use diff to verify.

sed: the hold space

sed has a second buffer, the hold space, that persists between lines. The commands h and H copy or append the pattern space to it, g and G copy or append it back, and x swaps them. They let sed remember a line and are the reason a short sed script can do things like reversing a file. They also make scripts hard to read, so use them sparingly.

printf '1\n2\n3\n' | sed -n '1!G;h;$p'

3
2
1

Read the script as: on every line except the first, append the hold space to the pattern space; copy the pattern space to the hold space; on the last line, print. Each line is placed before everything seen so far, which reverses the file. The equivalent tac is clearer, and so is awk with an array. The lesson is not to write such scripts but to recognise the tool's limit: when you reach for the hold space, ask if awk would read better.

sed: multiline work and special cases

sed processes one line at a time, so patterns cannot span a newline unless you collect lines. The N command appends the next line to the pattern space, and -z in GNU sed treats the whole file as one line by using NUL as the separator. Both let a regex match across lines.

sed -z 's/\n\n*/\n/g' file
sed ':a;N;$!ba;s/\n/,/g' file
paste -sd, file

The second command joins all lines with commas: label a, append the next line, and branch back to a until the last line. The third command does the same with paste, which is shorter and easier to read; this is a recurring pattern, where sed can do a job but a dedicated tool is simpler. Case conversion is another: GNU sed has \U, \L and \u in replacements, which BSD sed lacks, so for portable work use tr or awk's toupper.

awk: the model

An awk program is a list of pattern { action } pairs. For each input line, awk splits the line into fields, then runs the action of every pattern that matches. A missing pattern matches every line; a missing action prints the line. Special patterns BEGIN and END run before the first line and after the last. Fields are $1, $2, and so on, $0 is the whole line, and built-in variables include NR (line number), NF (field count) and FS (field separator).

awk '{print $1, $9}' access.log | head -3
awk '$9 >= 500 {print $1, $7}' access.log
awk 'NF {n++} END {print n " non-empty"}' app.conf

203.0.113.9 200
198.51.100.4 500
203.0.113.9 401
198.51.100.4 /api/users
198.51.100.4 /api/users
6 non-empty

The second program is the whole idea in one line: a condition on a field, then an action using other fields. Awk converts a field to a number when a numeric comparison needs it, so $9 >= 500 compares as numbers, whereas two fields that look like numbers compared with each other use string comparison unless they are clearly numeric. The GNU Awk User's Guide is the definitive reference and unusually readable.

awk: field separators

By default, awk splits on runs of blanks and ignores leading ones. Set a different separator with -F or by assigning FS. A single character is taken literally, except that a space is special; a longer string is a regular expression. The output separator is OFS, used by print a, b, and it only takes effect when you print fields as a list or reassign a field to force awk to rebuild $0.

awk -F' *= *' '/^[a-z]/ {print $1 "=" $2}' app.conf
awk -F, 'NR==1 {print toupper($0); next} {printf "%s:%d\n", $2, $3}' sales.csv | head -3
awk 'BEGIN {FS=":"; OFS="\t"} {print $1, $7}' /etc/passwd | head -2

host=db01.internal
port=5432
timeout=30
user=deploy
REGION,PRODUCT,UNITS,PRICE
widget:10
gadget:4

The first command splits on an equals sign with optional spaces around it, which normalises the two spacing styles in the sample config. The second skips the header row with next. Using FS in a BEGIN block is equivalent to -F and is better inside a script file. Use -F'\t' or -F'\\t' for tab-separated data, and note that mawk, gawk and BSD awk differ in how they treat escapes in -F.

awk: variables and arrays

Variables are created on first use, hold strings or numbers, and are initialised to empty or zero. Arrays are associative: keys are strings, so count[$1]++ builds a frequency table without declaring anything. The for (key in array) loop visits keys in no guaranteed order, so pipe through sort when order matters.

awk '{b[$1]+=$10} END {for (i in b) print i, b[i]}' access.log | sort
awk -F, 'NR>1 {t[$1]+=$3*$4} END {for (r in t) printf "%-6s %7.2f\n", r, t[r]}' sales.csv | sort
awk '!seen[$1]++ {print $1}' access.log

192.0.2.77 312
198.51.100.4 87
203.0.113.9 25645
east     30.00
north    95.00
south    47.50
203.0.113.9
198.51.100.4
192.0.2.77

The first sums bytes per client. The second computes revenue per region from units and price, formatting with printf. The third is the famous de-duplicator: seen[$1]++ evaluates to the old count (0 on first sight, which is false) and increments it, so !seen[$1]++ is true exactly once per key, and unlike sort -u it preserves the original order. These three one-liners cover a large share of real log and CSV questions.

awk: passing values in

Do not splice shell variables into the program text; pass them with -v, which handles quoting and keeps the program in single quotes. Variables set with -v are available in BEGIN. Assignments written as arguments between files take effect when awk reaches them.

awk -v min=400 '$9 >= min {c++} END {print c+0}' access.log
threshold=500
awk -v t="$threshold" '$9 >= t' access.log
awk '{print FILENAME ": " $1}' access.log app.conf | head -2

4
198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] "GET /api/users HTTP/1.1" 500 87
198.51.100.4 - - [05/Nov/2026:10:02:30 +0000] "GET /api/users HTTP/1.1" 502 0
access.log: 203.0.113.9
access.log: 198.51.100.4

Adding +0 to c in the first program prints 0 instead of an empty string when nothing matched, since an uninitialised variable is empty until used as a number. A -v value goes through escape processing, so a backslash in it is interpreted; for arbitrary text, pass it through the environment and read ENVIRON["name"], which is not processed.

awk: printf, functions and control flow

awk has C-like control flow: if/else, while, for, break, continue. It has string functions (length, substr, index, split, sub, gsub, match, sprintf, tolower, toupper) and numeric ones (int, sqrt, exp, log, rand). You can define your own functions. printf works like the C and shell version and does not add a newline.

awk 'function human(n) {
       split("B KB MB GB", u); i = 1
       while (n >= 1024 && i < 4) { n /= 1024; i++ }
       return sprintf("%.1f %s", n, u[i])
     }
     {total += $10} END {print human(total)}' access.log

25.4 KB

The function turns a byte count into a readable size, using split to build a unit array. Array indexes in awk start at 1 by convention for functions like split. Local variables in a function are declared as extra parameters, which is an awk quirk to know about when a function seems to leak values.

awk: ranges, getline and joining files

A range pattern /start/,/end/ matches from the first line that matches the start to the next one that matches the end, like sed's. Two-file idioms use NR==FNR: NR is the total line number and FNR the line number within the current file, so they are equal only while reading the first file. That makes a lookup join possible.

awk '/10:02:/,/10:03:/' access.log | wc -l
awk -F, 'NR==FNR {p[$1]=$2; next} {print $1, p[$1]}' regions.csv sales.csv

3
north N
south S

The join loads the first file into an array, then for each line of the second looks up the key. getline reads input explicitly, from a file or from a command, but its many forms are a source of subtle bugs; prefer the two-file idiom, and read the manual before using getline in a loop. For joining by a sorted key without loading memory, the standard join command is the alternative.

awk: gawk, mawk and BSD awk

The word awk names several programs. On Debian and Ubuntu the default is mawk, a small and fast implementation of POSIX awk; Fedora and Arch use gawk, the GNU version with many extensions; macOS ships a BSD-derived one-true-awk. The POSIX subset works in all of them, and gawk adds gensub, true multidimensional arrays, asort, time functions strftime and systime, FPAT for CSV-like parsing, BEGINFILE and the --csv mode introduced in 5.3.

awk --version | head -1
mawk -W version 2>&1 | head -1
gawk '{print gensub(/\[([^:]*):.*/, "\\1", 1, $4)}' access.log | head -2

GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.0, GNU MP 6.2.1)
mawk 1.3.4 20200120
05/Nov/2026
05/Nov/2026

The first line is what awk reported after installing the gawk package on Debian, which replaces mawk as the awk alternative. Write for POSIX awk unless the script declares gawk in its shebang or documentation, and test with mawk and BusyBox awk, which are the most restrictive. The portable-shell guide at POSIX sh versus Bash shows the testing setup that works for these as well.

Combine the three in a pipeline

Each tool does its own part and passes plain text on. grep narrows, sed normalises, awk computes. Order the stages so the cheapest, most selective one runs first and the data shrinks early.

grep -E '"(GET|POST) /api/' access.log \
  | sed -E 's/\?[^ ]*//' \
  | awk '{c[$7]++; if ($9 >= 500) e[$7]++}
         END {for (p in c) printf "%-12s %d requests %d errors\n", p, c[p], e[p]+0}' \
  | sort

/api/login   1 requests 0 errors
/api/users   2 requests 2 errors

The grep keeps only API calls, the sed removes any query string so that /api/users?page=2 and /api/users count as one path, and awk counts requests and errors per path. Splitting jobs this way keeps each stage testable on its own: run the pipeline up to a stage and look at the output. As a pipeline grows past four stages, ask whether one awk program would be clearer; awk can do the select and normalise steps itself, and reading the file once is faster.

Worked job: a log report

Turn the access log into a one-screen report: total requests, error rate, the busiest client and the transferred size. One awk program does it in one pass.

awk '
  { total++; bytes += $10; client[$1]++ }
  $9 >= 500 { errors++ }
  END {
    top = ""; max = 0
    for (c in client) if (client[c] > max) { max = client[c]; top = c }
    printf "requests:    %d\n", total
    printf "5xx errors:  %d (%.1f%%)\n", errors, 100 * errors / total
    printf "busiest:     %s (%d)\n", top, max
    printf "bytes:       %d\n", bytes
  }' access.log

requests:    6
5xx errors:  2 (33.3%)
busiest:     203.0.113.9 (3)
bytes:       26044

The program has three kinds of rules. The first runs on every line and accumulates. The second has a condition, so it only counts server errors. The END block finds the maximum and prints. If the file has no 5xx lines, errors is empty and prints as 0 with %d, but dividing by an uninitialised value works because awk treats it as zero; guard the division by total in case of an empty file, or the program reports a division by zero error.

Worked job: change a config value safely

Change port in a config to a new value, only if the key exists, and keep a backup. Check, edit, verify.

#!/usr/bin/env bash
set -euo pipefail
file="$1"; new="$2"
grep -qE '^port[[:space:]]*=' "$file" || { echo "no port key in $file" >&2; exit 1; }
sed -i.bak -E "s/^(port[[:space:]]*=[[:space:]]*).*/\\1${new}/" "$file"
grep -E '^port' "$file"
diff "$file.bak" "$file" || true

$ ./setport.sh t.conf 6432
port = 6432
3c3
< port = 5432
---
> port = 6432

The grep guard stops the script from silently doing nothing on a file without the key. The substitution keeps the key and its spacing with the capture group and replaces only the value. Because ${new} is inserted into the sed script, the script should validate it first: a value containing a slash or an ampersand would break or alter the command, so check [[ $new =~ ^[0-9]+$ ]] for a port. Quoting rules for this kind of input are the topic of the shell script security guide.

Worked job: summarize a CSV

Revenue by region and product from the sales file, as a sorted table with a total line. The awk uses the multidimensional-style key trick to group on two columns at once.

awk -F, 'NR > 1 { r = $3 * $4; sum[$1 "," $2] += r; all += r }
  END { for (k in sum) printf "%-14s %8.2f\n", k, sum[k]
        printf "%-14s %8.2f\n", "TOTAL", all }' sales.csv | sort -k2 -nr

TOTAL            172.50
north,gadget      70.00
south,gadget      40.00
east,widget       30.00
north,widget      25.00
south,widget       7.50

Because the separator is a comma, the composite key is built by concatenating fields with a comma, and the sort at the end places the total first. This works on this file because no value contains a comma or a quote. The moment a field like "Smith, Jane" appears, the split is wrong, which is the subject of the CSV note under the limits below.

A regex cheat sheet for the three dialects

The same idea is often spelled three ways, and a short table saves a trip to the manual. The table lists what GNU grep, GNU sed and gawk accept on Debian 12; POSIX-only systems accept the first column of each row.

Two behaviors are shared across all of them and surprise newcomers. Matches are greedy and leftmost: the engine finds the match that starts earliest and, among those, takes the longest. And without an anchor a pattern matches anywhere in the line, so /error/ also matches terrors. Add \b, -w or explicit anchors when whole words matter. The POSIX rule for leftmost-longest differs from Perl, where alternation takes the first alternative that works, so a pattern like a|ab matches ab in grep -E and only a in Perl.

Use dynamic input safely

When a pattern or replacement comes from a variable, the characters in it become syntax. A search term containing ., *, [ or / changes what grep and sed do. For grep, -F makes the variable literal. For sed, escape the three characters that matter in a replacement, and for the pattern the set of regex metacharacters, or avoid the problem by moving the logic into awk, where a string comparison with == or index() is literal.

needle='price (USD)'
grep -F -- "$needle" prices.txt
awk -v n="$needle" 'index($0, n) {print}' prices.txt
escape_sed() { printf '%s' "$1" | sed -e 's/[\/&]/\\&/g'; }
sed "s/old/$(escape_sed "$replacement")/" file

The -- before the pattern stops a needle starting with a dash being read as an option. Treat any value that came from a user, a web request or a file name as untrusted data: a replacement containing a newline can add a second sed command in some versions, and an unescaped delimiter can end the substitution early. The injection cases are collected in the shell script security guide.

Bulk edits across many files

To change text across a project, combine grep to find the files and sed to edit them, instead of running sed on everything. This touches only files that match, so unchanged files keep their timestamps and do not show up in version control as modified.

grep -rlZ --include='*.conf' 'old.example.org' . \
  | xargs -0 sed -i.bak 's/old\.example\.org/new.example.org/g'
git diff --stat
find . -name '*.bak' -delete

The -Z flag prints names separated by NUL bytes and xargs -0 reads them, so file names with spaces survive. The dots in the pattern are escaped so that they match literal dots, not any character. Review the result with git diff before committing, and delete the backups only after you have looked. In a Git repository, git grep -lz is a faster source of the file list that skips ignored files.

Small helpers: tr, cut, paste, sort and uniq

Several single-purpose tools sit next to the big three and are better for their one job. cut -d, -f1,3 picks columns from a clean delimited file. tr -d '\r' removes Windows carriage returns, tr -s ' ' squeezes repeated spaces and tr a-z A-Z changes case. paste -sd, joins lines with a comma, as shown above. sort -u, sort -k2,2n and uniq -c order and count. comm, join and diff compare. Using the dedicated tool makes the intent obvious, and the line reads like a sentence.

tr -d '\r' < windows.txt > unix.txt
cut -d, -f1 sales.csv | tail -n +2 | sort | uniq -c
paste -sd, <(cut -d, -f2 sales.csv | tail -n +2 | sort -u)

      1 east
      2 north
      2 south
gadget,widget

When a task needs two of these joined by an intermediate file, or a computation, move up to awk. A good rule is that the first time you reach for a temporary file in a pipeline, awk or a script probably fits better.

Debugging a sed or awk script

Run the pipeline one stage at a time and read the output of each. Replace -i with printing while testing. For awk, add print statements to standard error, such as print "x=" x > "/dev/stderr", and use NR to limit the run to a few lines. gawk offers --lint to warn about suspicious constructs and --debug for an interactive debugger. For sed, GNU sed has --debug, which prints the canonical form of the script and traces each command, and sed --posix disables GNU extensions to check portability.

awk --lint '{ total += $10 } END { print totl }' access.log
sed --debug 's/GET/FETCH/' access.log | head -6

The lint run flags that totl is used but never set, which is how a typo in a variable name shows up: awk otherwise prints an empty string without complaint. When a regex seems not to match, print the field it should be matching with printf '[%s]\n', $3; square brackets reveal trailing spaces and carriage returns. The broader techniques are in the guide to debugging shell scripts.

Locales, Unicode and line endings

Text tools follow the locale for character classes, sorting and case. In a UTF-8 locale, . matches a whole character and [[:alpha:]] includes accented letters; in the C locale each byte is a character. A script that behaves on a developer's laptop and differently under cron is often a locale difference, because cron runs with a minimal environment. Set LC_ALL=C when you want byte semantics and stable sort order, and LC_ALL=C.UTF-8 or en_US.UTF-8 when you need Unicode-aware matching and the locale exists.

Windows line endings are the other invisible variable. A file with CRLF ends each line with a carriage return that is part of the last field, so $NF == "ok" fails and a pattern ending in $ never matches. Strip it first with tr -d '\r', or in awk by setting RS="\r?\n" in gawk. Use file to confirm the line endings of a file when output looks right but comparisons fail.

Exit statuses inside scripts

All three tools have exit statuses that a script must treat deliberately. grep's 1 for no match is not an error but will stop a script running with set -e or pipefail. sed returns 0 even when its substitution matched nothing, so a script cannot tell success from a no-op without checking the result. awk returns 0 unless the program calls exit n or hits a runtime error. Make the intent explicit.

set -euo pipefail
count=$(grep -c ERROR app.log || true)
grep -q '^port' app.conf || { echo "port missing" >&2; exit 1; }
awk 'END { exit (NR == 0) }' app.log || echo "log was empty"

The last line uses awk's exit status to signal an empty file, a handy way to carry a condition out of an awk program. Remember that under pipefail a pipeline such as grep ... | head -1 can fail with status 141 when head closes the pipe early; test for the condition you care about instead of relying on the pipeline's final status.

Pick a tool in ten seconds

Ask three questions in order. Does the task only need to know whether, or which, lines match? Use grep, and stop there. Does it need to alter the text and pass the rest through untouched? Use sed. Does it need the columns, a running total, a lookup or a decision per line? Use awk. If the answer to the first question is yes and you also need part of the line, take grep with -o and a capture-free pattern first, and move to awk only when that cannot express it. The choice is rarely permanent: a one-liner that starts as grep often grows into awk after the third request for a new column, and that is the right time to switch, since the work you did on the pattern carries over unchanged.

Common mistakes and their fixes

• Unquoted patterns. grep ^a b* file is expanded by the shell. Use single quotes.
• Forgetting -n with sed p. Every selected line prints twice.
• Greedy matches. .* runs to the last occurrence. Use [^"]* or a more specific class.
• Using grep where a fixed string is meant. A dot in grep 1.5 matches any character. Use -F.
• sed -i on macOS. Needs a suffix. Use -i.bak.
• Counting with grep | wc -l when grep -c will do. And note -c counts lines, not matches.
• Reading a variable unset in awk as zero. Fine for arithmetic, but prints empty in string context. Add +0.
• Whitespace fields containing spaces. Default splitting breaks on them. Pick a real separator.
• Changing a file you are reading in the same pipeline. sed ... file > file truncates it first. Write to a new file or use -i.

The last one destroys data silently: the shell truncates the output file before sed starts, so sed reads an empty file. Always write to a temporary name and rename, or let sed -i do it.

Keep awk and sed programs in files

A one-liner is perfect until it needs a comment. Once a program passes three or four lines, move it into a file: awk takes -f program.awk and sed takes -f script.sed. In a file you can indent, add comments, use blank lines and version the program with the rest of the project. The shebang form lets the file run as a command. The only changes are that quoting is no longer needed, since the shell does not see the program, and that variables are passed with -v or as arguments as before.

cat > report.awk <<'EOF'
# Summarize an access log: requests, errors and bytes per client.
# usage: awk -f report.awk -v min_status=500 access.log
BEGIN { if (min_status == "") min_status = 500 }
{
    requests[$1]++
    bytes[$1] += $10
    if ($9 >= min_status) errors[$1]++
}
END {
    for (client in requests)
        printf "%-16s %4d requests %3d errors %8d bytes\n",
               client, requests[client], errors[client] + 0, bytes[client]
}
EOF
awk -f report.awk access.log | sort

192.0.2.77         1 requests   0 errors      312 bytes
198.51.100.4       2 requests   2 errors       87 bytes
203.0.113.9        3 requests   0 errors    25645 bytes

The comments explain intent, the default for the threshold is explicit, and a long printf is split across lines with the arguments aligned. The same file works as a library of functions when you add several -f options. Testing a program in a file is also easier: feed it a three-line input and compare the output, as the testing guide shows.

Word counts, rankings and percentiles

Three reports come up so often that they deserve a recipe each. Word or token frequency is the sort and count chain. A ranking by one column is a sort with a key. A percentile needs the data sorted and an index, which awk does in two passes or with a sort feeding it.

# top words in a text, case-insensitive
tr -cs '[:alpha:]' '\n' < essay.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head

# 95th percentile of response times in column 11
awk '{print $11}' access.log | sort -n | awk '{a[NR]=$1} END {print a[int(NR*0.95+0.5)]}'

# average and maximum of a column
awk '{s+=$10; if ($10>m) m=$10} END {printf "avg %.1f max %d\n", s/NR, m}' access.log

The first command turns every run of non-letters into a newline, so each word has its own line. The second sorts numerically and picks the element at the 95th percent position, rounded. The third keeps a running sum and maximum. Each processes the file in a stream, apart from the percentile, which stores the values and so needs memory proportional to the data. On very large inputs, sort to a temporary file and pick the line by number with sed -n instead.

Templating text with envsubst, sed and awk

Generating a config file from a template is a common job. The standard tool is envsubst from gettext, which replaces ${NAME} in its input with environment values, and you can pass a list of names to limit which variables it touches. When it is not installed, sed or awk do the same with a loop over the variables.

export HOST=db01.internal PORT=5432
printf 'host = ${HOST}\nport = ${PORT}\nuser = $USER\n' | envsubst '${HOST} ${PORT}'

render() {
  sed -e "s|@HOST@|${HOST//|/\\|}|g" -e "s|@PORT@|${PORT//|/\\|}|g" "$1"
}
render app.conf.in > app.conf

host = db01.internal
port = 5432
user = $USER

The list of names given to envsubst matters: without it, every dollar sign in the template is treated as a variable, which breaks nginx configs, shell snippets and anything else that uses dollars for itself. The sed version uses markers like @HOST@ that cannot collide with shell syntax, and escapes the delimiter in the value. If the values are not trusted, validate them first, because a value containing the delimiter or an ampersand changes what sed does.

Extract a section from a structured text file

Many text files have sections: INI files with bracketed headings, changelogs with a version heading, Markdown with titles. A range in sed or awk extracts one by matching the heading and stopping at the next. The awk form is clearer because it can set a flag on the heading and clear it on the next.

cat > app.ini <<'EOF'
[database]
host = db01
port = 5432

[cache]
host = cache01
ttl = 300
EOF
awk '/^\[/ {on = ($0 == "[cache]"); next} on && NF' app.ini
sed -n '/^\[cache\]/,/^\[/{/^\[/!p}' app.ini

host = cache01
ttl = 300
host = cache01
ttl = 300

Both print the contents of the cache section only. In awk, the flag on is true between the target heading and the next heading, and the pattern on && NF prints non-empty lines while it is. In sed, the range runs from the section heading to the next heading, and the inner command prints every line that does not start with a bracket. The awk version reads better and copes with the last section, where the sed range just runs to the end of the file.

More than one file at a time

awk reads its file arguments in sequence and exposes FILENAME, FNR (line within the current file) and NR (line overall), which supports a family of tasks: prefix output with the file name, process the first line of each file, and compare two files. The idiom FNR==1 detects the start of each file, and NR==FNR detects the first file.

awk 'FNR == 1 {print "== " FILENAME} {print FNR ": " $0}' a.txt b.txt
awk 'NR == FNR {seen[$1]; next} !($1 in seen)' old.txt new.txt
awk -F, 'NR == FNR {price[$1] = $2; next} {print $0, price[$1]}' prices.csv orders.csv

The second command prints the lines of the second file whose first field does not occur in the first, a set difference in one line. The third joins orders to prices by key. For big files this loads the first file into memory, so put the smaller file first. When both are large, sort both and use join, which streams. Both approaches assume keys are unique in the lookup file; if they are not, the last value wins in awk and join produces one output line per pair.

Grep alternatives for large trees

GNU grep is everywhere and fast for single files, and tools built for searching code trees are faster and more convenient there. ripgrep (rg) respects .gitignore, skips binary files and hidden directories, searches in parallel and prints with color and line numbers by default. The Silver Searcher (ag) works similarly. git grep searches only tracked files, uses the index and needs no installation beyond Git. For one-off use on a server you do not control, plain grep is the dependable choice.

rg -n 'TODO|FIXME' --type sh
git grep -n -E 'TODO|FIXME' -- '*.sh'
grep -rn --include='*.sh' -E 'TODO|FIXME' .
rg -l 'set -e' | xargs rg -L 'pipefail'

The last pipeline finds files that use set -e but never mention pipefail, which is a code review question answered in one line. ripgrep's regex dialect is the Rust one, which is close to ERE with \d and \w shorthands and no backreferences; -P switches to PCRE2 when built with it. Know the dialect of the tool in your hand, as the regular expressions guide explains.

sed one-liners worth memorizing

A short list covers most daily editing. Each is documented in the sed manual, and each is safer to run first without -i on a copy.

sed -n '$='            file    # count lines
sed 's/[ \t]*$//'      file    # strip trailing blanks
sed '/^$/d'            file    # delete empty lines
sed -n '10,20p'        file    # print lines 10 to 20
sed '1!G;h;$!d'        file    # reverse the file (tac)
sed -e :a -e '$!N;s/\n=//;ta' -e 'P;D' file   # join lines that begin with =
sed '0,/foo/s//bar/'   file    # replace only the first match in the file (GNU)
sed 's/^/    /'         file    # indent every line by four spaces

The first-match-only trick, using the address 0,/foo/, is a GNU extension; the portable form needs a flag variable in awk. The strip-trailing-blanks pattern uses \t inside a bracket, which GNU sed reads as a tab and POSIX sed reads as a backslash and a t; use a literal tab or [[:blank:]] to be portable. Each of these is easy to read once explained and cryptic without, so keep them in a personal notes file with the explanation beside them.

Reading fixed-width and messy input

Not all tables are delimited. Output from older tools and mainframe exports often uses fixed column positions, and awk's substr or gawk's FIELDWIDTHS read them. For messy input with varying spaces, the default whitespace splitting is usually the right answer, and for output of other commands, check whether the tool offers a machine-readable mode first: ps -o, df --output, lsblk -J, ip -j and systemctl show all do, and parsing their stable formats is more reliable than scraping the pretty ones.

df --output=source,pcent,target | awk 'NR > 1 {gsub("%", "", $2); if ($2 > 80) print $3, $2 "%"}'
ps -eo pid=,comm=,%mem= --sort=-%mem | head -3
ip -j addr show lo | jq -r '.[0].addr_info[0].local'
gawk 'BEGIN {FIELDWIDTHS = "10 6 8"} {print $2}' report.txt

The first command lists mounted filesystems that are more than 80 percent full, using the percent column with the symbol stripped so awk can compare numbers. The equals signs in the ps format suppress the header. When a program has a JSON output mode, take it and use jq; the shape of the data survives changes in the pretty format, which scraping does not.

Where this does not apply

JSON and XML. Regexes cannot reliably parse nested structures, and a pipeline of sed commands on JSON breaks the first time a value contains a quote or a newline. Use jq for JSON and xmllint or xmlstarlet for XML.

Quoted CSV. Fields with embedded commas, newlines or doubled quotes defeat -F,. Use gawk with FPAT, gawk 5.3's --csv, Miller, csvkit or Python's csv module.

Binary files and mixed encodings. These tools assume text. grep reports binary file matches on standard error and awk and sed may misbehave on invalid multibyte sequences; set LC_ALL=C or convert with iconv first.

Large transformations with logic. When an awk program passes about fifty lines or needs functions, data structures and tests, write it in Python or Go. The line is not about capability, since awk is Turing complete, but about maintainability.

Very large files with simple needs. For a single count or filter over tens of gigabytes, a purpose-built tool such as ripgrep can be much faster than GNU grep with a complex pattern.

What to read next

The next step in text processing is the pattern language itself, covered in the regular expressions in shell tools guide, which compares the dialects in grep, sed, awk and Bash. To turn these one-liners into reliable scripts, read the Bash scripting tutorial, and to find out why a pipeline gives the wrong answer, the debugging guide. The wider toolbox these three sit in is the Linux command line guide.

Primary sources used here: the GNU grep manual, the GNU sed manual, the GNU Awk User's Guide, the POSIX awk specification, the POSIX sed specification, the POSIX grep specification, the POSIX regular expressions chapter, RFC 5737 for the documentation address blocks and the mawk manual.