Regular Expressions in Shell Tools From Basics to Practice

Regular expressions are the pattern language of the command line, and the confusion around them comes from there being several of them. The pattern that works in grep -P fails in grep -E, the one for sed needs backslashes that awk forbids, Bash has its own rules for quoting, and the globs used for file names look similar and mean something else. This guide sorts that out. It explains the three dialects, the operators they share, how each tool applies them, and the traps at each boundary, with every example run on Debian 12 using GNU grep 3.8, GNU sed 4.9, gawk 5.2.1, mawk 1.3.4, BusyBox 1.35 and Bash 5.2.15. The tools themselves are covered in the grep, sed and awk guide, and this guide goes deeper on the patterns.

The guide goes from the building blocks to the dialect differences, then each tool, then globs, locale and performance, and ends with a method for building and testing a pattern and a list of cases where a regex is the wrong tool.

What a regular expression is

A regular expression describes a set of strings. A tool reads text and asks, for each line or each position, whether some part of it belongs to that set. The expression cat describes the single string cat. The expression c.t describes all three-character strings that start with c and end with t. The expression ^[0-9]+$ describes every line made only of digits. The power comes from a small set of operators: pick one of several, repeat, group, and anchor.

What the tool does with a match depends on the tool. grep prints the line, or just the match with -o. sed replaces the match. awk uses the match to select records, split fields or edit them. Bash uses it in a condition and exposes the groups. In all of them the same two rules apply: a match can start anywhere in the line unless you anchor it, and when several matches are possible at a position the engine takes the longest. Hold on to those two rules, since they explain most surprises.

Three dialects

POSIX defines two syntaxes, and a third family comes from Perl. They differ in which characters are operators.

The syntax is specified in the POSIX regular expressions chapter. The practical advice follows from the table: write ERE, because it is the one most people can read, and use -E with grep and sed so that you never have to remember which characters need a backslash. Reach for PCRE only when you need what ERE cannot say, and only where you know it is installed.

BRE and ERE side by side

The same pattern means different things depending on the flag. In basic syntax, an unescaped plus sign is a literal plus. In extended syntax it is a repetition operator.

echo "aab" | grep 'a\+b'
echo "aab" | grep -E 'a+b'
echo "a+b" | grep 'a+b'
echo "a+b" | grep -E 'a\+b'

aab
aab
a+b
a+b

All four lines matched, for different reasons. The first uses the GNU backslash form in BRE to mean one or more. The second uses ERE. The third matches a literal plus in BRE, since the plus has no special meaning there. The fourth escapes the plus in ERE to make it literal. The backslash flips the meaning between the dialects: in BRE it adds a meaning to a character, in ERE it removes one. The backslash forms \+, \? and \| in BRE are GNU extensions, not POSIX, so a script that must run on BSD or BusyBox should use -E.

Literal characters and escaping

Most characters match themselves. A handful have special meanings, and to match them literally you escape them with a backslash or place them in a bracket expression. In ERE the special characters are . [ ] ( ) * + ? { } | ^ $ \. The dot is the most frequent mistake: it matches any character, so 1.5 also matches 125.

echo 'a.b axb' | grep -o 'a.b'
echo 'a.b axb' | grep -o 'a\.b'
echo 'a]b' | grep -o '[]]'

a.b
axb
a.b
]

The first pattern matched both the real dot and the x; the escaped version matched only the real dot. Inside a bracket expression, most special characters lose their meaning, which is a tidy way to write a literal: [.] is a dot. The closing bracket must come first in the list to be literal, as in []]. Two layers of escaping are involved when the pattern passes through the shell, so keep patterns in single quotes. When the text to find is not a pattern at all, use grep -F and skip the problem.

Bracket expressions and character classes

A bracket expression matches one character from a set. [abc] matches a, b or c; [a-z] is a range; [^0-9], with a caret first, is anything except a digit. POSIX also defines named classes written inside brackets: [[:digit:]], [[:alpha:]], [[:alnum:]], [[:space:]], [[:upper:]], [[:lower:]] and [[:punct:]]. The double bracket is correct: the class name is inside a bracket expression.

echo "ab12 _x" | grep -o '[[:digit:]]*'
echo "ab12 _x" | grep -oE '[[:alpha:]]+'
echo "ab12 _x" | grep -oE '\w+'
echo "a1" | grep -E '\d' || echo "no \d in ERE"
echo "a1" | grep -oP '\d'

12
ab
x
ab12
_x
no \d in ERE
1

The named classes work everywhere and follow the locale. The shorthand \w works in GNU grep and sed as an extension, matching letters, digits and underscore, which is why it matched ab12 and _x as words. The Perl shorthand \d is not defined in ERE: GNU grep treated the backslash-d as a plain d, so the pattern found nothing, and it works only with -P. Ranges such as [a-z] depend on the locale's collation order in some implementations; the named classes are the portable way to say the same thing.

Anchors and boundaries

The caret matches the start of a line and the dollar sign matches the end. Without them a pattern can match anywhere in the line, which is the usual reason for a pattern that matches too much. grep offers shortcuts: -x requires the whole line to match, and -w requires the match to be a whole word, bounded by non-word characters.

printf 'foo\nfoobar\nbar foo\n' | grep -c '^foo'
printf 'foo\nfoobar\nbar foo\n' | grep -x foo
printf 'foo\nfoobar\n' | grep -w foo
echo "foo-bar" | grep -ow 'foo'

2
foo
foo
foo

Two lines start with foo, so the count was 2. -x kept only the line that is exactly foo. -w rejected foobar but accepted foo-bar, because a hyphen is not a word character. GNU tools also accept \b for a word boundary, \< and \> for the start and end of a word, and gawk uses \y for the boundary; these are extensions, so check the target tool. In sed, the caret and dollar anchor each line of the pattern space, and in multi-line mode (the next sections) they can be made to anchor the whole buffer.

Quantifiers

A quantifier follows an item and says how many times it may repeat. * is zero or more, + one or more, ? zero or one, {n} exactly n, {n,} at least n and {n,m} between n and m. An item is a single character, a bracket expression, or a group.

echo "aaa" | grep -oE 'a{2}'
echo "aaa" | mawk '/a{2}/ {print "mawk match"}'
echo "aaa" | mawk '!/a{2}/ {print "mawk no match"}'
echo "aaa" | gawk '/a{2}/ {print "gawk match"}'

aa
mawk no match
gawk match

The same interval gave different results in two awks. On Debian, awk is mawk 1.3.4 (version 20200120), which treated the braces as literal characters, so a{2} meant the text a{2}; gawk supports intervals. A script that relies on {n,m} in awk must either require gawk or repeat the item by hand, as in aa. It is a good example of why the tool's dialect matters as much as the pattern's, and the portable-shell guide, POSIX sh versus Bash, lists the testing approach for such differences.

Groups and alternation

Parentheses group items so a quantifier applies to the whole group and capture the matched text for later use. The vertical bar chooses between alternatives. In BRE with GNU extensions both need backslashes.

echo "cat dog" | grep -o 'cat\|dog'
echo "cat dog" | grep -oE 'cat|dog'
echo "2026-11-05" | sed -E 's/([0-9]+)-([0-9]+)-([0-9]+)/\3.\2.\1/'
echo "2026-11-05" | sed 's/\([0-9]*\)-\([0-9]*\)-\([0-9]*\)/\3.\2.\1/'

cat
dog
cat
dog
05.11.2026
05.11.2026

The two sed commands do the same date rearrangement, and the extended form is a good deal easier to read. The captured groups are numbered by their opening parenthesis from left to right, and in the replacement \1 to \9 refer to them. Group with (?:...) to avoid capturing is a PCRE feature; in ERE every group captures, and the practical cost is only that group numbers shift when you add one. Alternation tries alternatives left to right in Perl but takes the longest overall match in POSIX, so a|ab against ab matches the whole of it in grep and sed and only the first letter in Perl-style engines.

Backreferences

A backreference matches the same text that an earlier group matched. It makes it possible to find repeated words and matching quote characters, which is beyond the reach of a regular language in the strict sense and is supported by all three dialects.

echo "abcabc xyzxyz abab" | grep -oE '([a-z]+)\1'
echo "hello wood" | grep -E '(.)\1'
echo 'say "hi" and '"'"'bye'"'"'' | grep -oE "(['\"])[^'\"]*\1"

abcabc
xyzxyz
abab
hello wood

The first pattern finds a run of letters followed by a repeat of itself. The second finds any doubled character, which matched both words. The third finds text inside matching single or double quotes. Backreferences carry a cost: patterns that use them cannot be run by the fast finite-automaton engines, so grep switches to a slower matcher for them. In POSIX ERE the backreference is an extension of the standard that GNU and BSD both implement, so it works on them and not in every awk.

Greedy matching and negated classes

Quantifiers are greedy: they take as much as possible while still allowing the whole pattern to match. This is the cause of the classic mistake of trying to match a delimited section with .*, which runs to the last closing delimiter and swallows everything in between.

echo '<b>x</b><b>y</b>' | grep -o '<b>.*</b>'
echo '<b>x</b><b>y</b>' | grep -oE '<b>[^<]*</b>'
echo '<b>x</b><b>y</b>' | grep -oP '<b>.*?</b>'

<b>x</b><b>y</b>
<b>x</b>
<b>y</b>
<b>x</b>
<b>y</b>

The first pattern matched the whole string as one. The second uses a negated class, [^<]*, meaning any run of characters that are not a less-than sign, which cannot cross the next tag. The third uses the lazy quantifier .*? from PCRE, which takes as little as possible. POSIX has no lazy quantifier, and the negated class is the portable substitute, and a clearer one: it says exactly what the content may contain. The same trick pulls the first quoted string out of key="a" other="b" with "[^"]*".

grep in practice

grep's three engines are chosen by flag: none for BRE, -E for ERE, -P for PCRE, and -F for fixed strings with no pattern at all. Add -i to ignore case, -o to print only matches, -w and -x for whole words and lines, -v to invert, and -c to count lines. Multiple -e options combine patterns.

echo "Error ERROR error" | grep -oi 'error' | wc -l
echo "price: 100USD 200EUR" | grep -oP '\d+(?=USD)'
echo "foo.bar" | grep -oP '(?<=foo\.)\w+'
printf 'a\nb\nc\n' | grep -Pzo 'a\nb' | tr '\0' '\n'

3
100
bar
a
b

The lookahead (?=USD) asserts that USD follows without consuming it, so only the number is printed; the lookbehind (?<=foo\.) asserts what comes before. The -z option treats the whole input as one line by using NUL as the separator, so a pattern can span newlines, and the output is NUL-terminated, hence the final tr. -P is documented as experimental in the GNU grep manual, is missing from BusyBox (grep: invalid option -- 'P') and BSD grep, and is not available in every build of GNU grep, so scripts should not assume it. The GNU grep manual documents each option.

sed in practice

sed uses BRE by default, ERE with -E (GNU also accepts -r), and applies the pattern in the s command and in address expressions. The special characters in the replacement are the ampersand, which stands for the whole match, and the backslash digits. Any character can serve as the delimiter, which avoids escaping slashes in paths.

echo "a/b" | sed 's/\//|/'
echo "a/b" | sed 's#/#|#'
echo "a.b" | sed 's/./X/g'
echo "a.b" | sed 's/\./X/g'
echo "hello" | sed 's/l*/X/'
echo "hello" | sed 's/l*/X/g'
echo "x" | sed -E 's/(x)/[&]/'

a|b
a|b
XXX
aXb
Xhello
XhXeXoX
[x]

The third result shows the dot at work: every character became X. The fifth and sixth results show a trap with patterns that can match nothing: l* matches zero l characters at the very start, so the first substitution inserted an X before hello, and with the g flag it inserted one at every position and replaced the real ll with a single X. Use l\+ or ll* when you mean at least one. To make a match case-insensitive use the I flag in GNU sed, and to match across lines use N to join lines first, or -z.

awk in practice

awk uses ERE. A regex between slashes is a pattern that matches against the whole record, and ~ and !~ match a regex against any string. The functions match, sub, gsub, split and gensub (gawk) take regexes. match sets RSTART and RLENGTH, and gawk's three-argument match fills an array of captures.

echo "id=42 name=bob" | awk 'match($0, /[0-9]+/) {print substr($0, RSTART, RLENGTH), RSTART, RLENGTH}'
echo "id=42 id=7" | gawk '{match($0, /id=([0-9]+)/, m); print m[1]}'
echo "a.b.c" | awk '{n=gsub(/\./,"-"); print $0, n}'
echo "a.b.c" | awk -F'[.]' '{print $2}'

42 4 2
42
a-b-c 2
b

POSIX awk has no way to extract a capture group, only the whole match with RSTART and RLENGTH; gawk's extension fills the array m. gsub returns the number of replacements, which is a handy counter. A field separator longer than one character is treated as a regex, and -F'[.]' splits on a dot by putting it in a bracket expression. A pattern given as a string, such as $0 ~ p with p holding text, is a dynamic regex, where escapes are processed twice, once by awk's string parser and once by the regex engine, so "\\." is needed to get a regex that means an escaped dot.

Dynamic patterns in awk

Passing a pattern into awk with -v works, and it brings the double-escaping issue plus a more basic one: the value is a regex, not a literal string, so special characters in it keep their meaning.

echo "a.b" | awk -v p='a.b' '$0 ~ p {print "re match"}'
echo "axb" | awk -v p='a.b' '$0 ~ p {print "dot matches x"}'
echo "axb" | awk -v p='a.b' 'index($0,p) {print "index"} END {print "done"}'

re match
dot matches x
done

The second line shows the danger: a value meant as the literal text a.b also matched axb. The third command uses index($0, p), which does a plain substring search, and printed only done, correctly finding nothing. Use index or == when the variable holds text, and ~ only when it is deliberately a pattern. This matters for security as well as correctness, since a pattern built from outside input is an injection point.

Bash and the =~ operator

Inside [[ ]], =~ matches the left string against an ERE on the right and fills the array BASH_REMATCH with the match and its groups. The rule that catches everyone is quoting: since Bash 3.2, any part of the right-hand side that is quoted is matched literally. Store the regex in a variable and leave the variable unquoted.

re='^([a-z]+)-([0-9]+)$'
s='app-42'
[[ $s =~ $re ]] && echo "${BASH_REMATCH[1]} ${BASH_REMATCH[2]}"
[[ $s =~ "^app" ]] && echo quoted-match || echo "quoted: no match"
[[ "file.txt" == *.txt ]] && echo glob-ok
[[ "file.txt" =~ *.txt ]] 2>&1; echo "rc=$?"

app 42
quoted: no match
glob-ok
rc=2

The quoted version of ^app failed to match app-42, since it looked for the literal text ^app. The last line shows what happens when a glob is written where a regex belongs: a leading asterisk is an invalid regex, so the test returned status 2, which is an error and not a plain false. Remember that status 1 means no match and 2 means a bad pattern. Bash regexes use the platform's regex library and so inherit its limits and locale. Bash offers no lazy quantifier and no lookaround.

Globs are not regular expressions

Globs are the wildcard patterns that the shell expands in file names and uses in case, in [[ == ]] and in parameter expansions such as ${var#prefix*}. They look similar and are a different language. In a glob, * means any string, ? one character and [...] a set, and the whole pattern must match the whole string. In a regex, * repeats the previous item, . is any character, and matching is unanchored.

case "img_01.png" in *_[0-9][0-9].png) echo glob-class-ok ;; esac
mkdir -p ft; touch ft/a1.log ft/b22.log ft/c.txt
find ft -name '*.log'
find ft -regex '.*/[a-z][0-9]+\.log'

The pattern in find -name is a glob, quoted so the shell passes it along; find -regex matches the whole path against a regex and defaults to Emacs syntax, selectable with -regextype posix-extended. A symptom of mixing them up is ls *.txt working while grep *.txt file fails: the shell expanded the glob before grep saw it, which is a reason to quote regex arguments.

Case-insensitive matching

Each tool has its own switch. grep uses -i. GNU sed uses the I flag on the substitution or address, s/error/X/I. gawk has the IGNORECASE variable, but POSIX awk does not, so the portable way is to lower-case the text before matching with tolower($0) ~ /error/. Bash has shopt -s nocasematch to change [[ ]] and case globally, so set and unset it around the use.

echo "Error" | sed 's/error/X/I'
echo "Error" | awk 'tolower($0) ~ /error/ {print "awk ok"}'
echo "Error" | gawk 'BEGIN {IGNORECASE=1} /error/ {print "IGNORECASE ok"}'
shopt -s nocasematch; [[ Error == error ]] && echo bash-ok; shopt -u nocasematch

X
awk ok
IGNORECASE ok

Case-insensitive matching is locale-dependent for non-ASCII letters, so it is reliable for English identifiers and logs and less so for text in other languages. Where behavior matters, state the locale in the script.

Locale and Unicode

Character classes, ranges, case folding and the meaning of a dot depend on the locale. In a UTF-8 locale a dot matches a whole character and [[:alpha:]] includes accented letters. In the C locale, each byte is its own character, so a two-byte letter is two bytes of nothing in particular.

echo "été" | LC_ALL=C.UTF-8 grep -o '[[:alpha:]]*'
echo "été" | LC_ALL=C grep -o '[[:alpha:]]*' | od -c | head -2

été
0000000   t  \n
0000002

In the C locale only the ASCII letter t matched, and both accented letters were treated as non-alphabetic. Cron jobs and containers often run with the C or POSIX locale because no LANG is set, so a pattern that handles Unicode on your laptop quietly stops at an accent in production. For text that must be treated as bytes, set LC_ALL=C on purpose, and for text that is Unicode, set a UTF-8 locale that exists in the image, C.UTF-8 being available on most Debian-based systems.

Multiline matching

Line-oriented tools match within a line. To match across lines you have to change how the input is split or collect lines first. grep's -z reads NUL-separated records, so a text file with no NULs is one record; combined with -P, a pattern may contain \n. In sed, N appends the next line to the pattern space and -z works like grep's flag. In awk, setting RS to an empty string reads paragraphs, and gawk accepts a regex as the record separator.

sed -z 's/\n\n*/\n/g' file
awk 'BEGIN {RS=""; FS="\n"} {print $1}' paragraphs.txt
gawk 'BEGIN {RS="\n--\n"} {print NR ": " $1}' sections.txt

Remember that in a multiline buffer the caret and dollar may anchor the start and end of the whole buffer and not each line, depending on the tool and mode. For anything structured over many lines, such as an XML or JSON document or a config with nesting, a real parser is the answer, as the final section explains.

Performance and catastrophic backtracking

Regex engines fall in two families. Automaton-based engines, which GNU grep uses for BRE and ERE, scan the text once and take time proportional to its length however tricky the pattern. Backtracking engines, which Perl, PCRE, Python and many others use, try alternatives one at a time and go back when one fails; that is what makes lookaround and lazy quantifiers possible, and it also makes some patterns exponentially slow. The classic case is nested quantifiers such as (a+)+$ against a long run of a followed by a character that stops the match.

time (printf 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!\n' | grep -E '^(a+)+$' > /dev/null)
printf 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!\n' | grep -P '^(a+)+$'

real	0m0.001s
grep: (standard input): exceeded PCRE's backtracking limit

GNU grep's ERE engine finished in about a millisecond on forty characters, while the PCRE engine gave up with a backtracking limit error on thirty. Both outcomes are real behavior of GNU grep 3.8 in the container. The lesson has two sides. With grep, sed and awk in their default modes you are protected from this class of problem; with -P, and with regexes in Python, JavaScript and other languages, an attacker who controls the input can freeze a program with a crafted string, the failure known as ReDoS. Avoid nested quantifiers over overlapping alternatives, anchor patterns, and cap input length when the text is not yours.

Validation versus extraction

Two different jobs use regexes, and they want different patterns. Extraction pulls something out of text you trust, so a loose pattern is fine and often better. Validation decides whether input is acceptable, so it must be strict and anchored, and regexes are good at shape but poor at meaning. An IPv4 pattern shows the gap.

for ip in 10.0.0.1 999.1.1.1 1.2.3; do
  echo "$ip" | grep -qE '^([0-9]{1,3}\.){3}[0-9]{1,3}$' && echo "$ip loose-match" || echo "$ip no"
done

10.0.0.1 loose-match
999.1.1.1 loose-match
1.2.3 no

The pattern checks the shape of four dotted numbers and accepts 999.1.1.1, which is not an address. A regex that also limits each octet to 0 to 255 exists, and is long and hard to read; a better design checks the shape with the regex and the range in code, or uses a tool that knows addresses, such as ipcalc, or a library. The same holds for email addresses, dates and URLs: a regex can catch a missing at-sign, and only sending a message proves an address works. When the input is untrusted, validate with an allow-list pattern anchored at both ends, as in the shell script security guide.

Build a regex step by step

Write a pattern in stages, testing each one against real lines, rather than all at once. Take a log line and extract the status code and path of failing requests.

198.51.100.4 - - [05/Nov/2026:10:01:03 +0000] "GET /api/users HTTP/1.1" 500 87

grep -E '" [0-9]{3} ' access.log
grep -E '"[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} ' access.log
grep -E '"[A-Z]+ [^ ]+ HTTP/[0-9.]+" 5[0-9]{2} ' access.log
sed -nE 's/.*"[A-Z]+ ([^ ]+) HTTP\/[0-9.]+" (5[0-9]{2}) .*/\2 \1/p' access.log

500 /api/users
502 /api/users

The first version finds any line with a closing quote, a space, a three-digit number and a space, which is loose but a good start. The second describes the request more completely, so it will not match text that happens to look like a status. The third restricts the status to the 5xx range. The sed command wraps the request in two groups and prints them in the order you want, using .* only on the outer edges, where its greediness is harmless because the inner parts are tightly defined. At each step run the pattern, look at what it matched and what it missed, and tighten or loosen. Print with grep -o or sed -n ... p to see exactly the matched text.

Test a pattern before trusting it

Keep a small file of lines that must match and lines that must not, and run every change to the pattern against both. The negative cases are the ones people skip and the ones that find the bug. A pair of loops makes it a ten-line test.

re='^[a-z][a-z0-9_-]{2,31}$'
for good in app web-1 db_main; do
  [[ $good =~ $re ]] || echo "SHOULD MATCH: $good"
done
for bad in A1 a '-x' 'a b' '' "$(printf 'ab\ncd')"; do
  [[ $bad =~ $re ]] && echo "SHOULD NOT MATCH: $bad"
done
echo done

The last negative case contains a newline, which matters: in some tools a dollar sign matches before a trailing newline, so a pattern meant to be strict lets ab followed by a newline through. In Bash the whole string is matched, but in grep each line is tested separately, so a multi-line value passes if any single line matches; validate a single line, or check for newlines first. For patterns that will live in a script, put the cases in a Bats test, as described in the guide to testing shell scripts.

Other tools that take regexes

Several other programs in a normal system accept patterns and each has its own flavor. find -regex matches the full path, with Emacs syntax by default. expr STRING : REGEX uses BRE anchored at the start and prints the first group, an old POSIX way of extracting a part. less and vim search with their own variants, and vim uses a distinct syntax with its own escapes. rename, pgrep, pkill, journalctl --grep, git grep and tmux each have a regex option whose dialect is in their manual page. When the dialect is unclear, test it with a small known pattern such as a+b against aab.

expr "abc123" : '[a-z]*\([0-9]*\)'
pgrep -f 'nginx: (master|worker)'
journalctl -u ssh --grep 'Failed|Invalid' --no-pager | tail -3
git grep -nE 'TODO|FIXME' -- '*.sh'

123

That expr call anchors at the start implicitly, which differs from every other tool here; it is the commonest reason people give up on it. The cost of learning each flavor is small if you keep the three families in mind and check the manual for the exceptions.

A compact reference

Keep the table for the moments when you know what you want and not the spelling. The deeper reasoning is in the sections above, and the manual pages hold the exceptions: regex(7) for POSIX and the PCRE2 pattern reference for Perl syntax.

Common mistakes and their fixes

• Unquoted patterns. The shell expands *, $ and ? first. Use single quotes.
• Using \d in ERE. It is a plain d. Use [0-9] or [[:digit:]].
• Forgetting anchors. An unanchored pattern matches anywhere, including inside longer words.
• Greedy dot-star. Use a negated class between delimiters.
• Unescaped dots. 1.5 matches 125. Escape or use -F.
• Quoting the regex in [[ =~ ]]. A quoted pattern is literal. Use a variable.
• Using a glob where a regex is needed, or the reverse. Check which language the tool speaks.
• Patterns that can match nothing. x* with the g flag inserts at every position.
• Assuming -P exists. BusyBox and BSD grep lack it.
• Relying on the locale. Cron and containers often run in C.

Nearly all of these are one error: writing a pattern in one dialect and running it in another. Naming the dialect in your head, glob, BRE, ERE or PCRE, before you type the first character prevents most of them.

A gallery of everyday patterns

Most real use of regexes is a short list of recurring jobs. Each of these ran against sample input in the Debian 12 container, and each shows a pattern written in the dialect that suits it.

printf '  padded line  \n' | sed -E 's/^[[:space:]]+|[[:space:]]+$//g' | od -c | head -1
printf 'a    b \t c\n' | sed -E 's/[[:space:]]+/ /g'
printf 'v1.12.3 and v2.0.0-rc1\n' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+(-[a-z0-9.]+)?'
printf 'mail [email protected] or [email protected] now\n' | grep -oE '[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}'
printf 'see https://example.org/a?b=1 and http://x.test\n' | grep -oE 'https?://[^[:space:]"<>]+'
printf '\033[31mred\033[0m text\n' | sed -E 's/\x1b\[[0-9;]*m//g'
printf 'line one \\\nline two\nline three\n' | sed -e ':a' -e '/\\$/N; s/\\\n//; ta'
printf 'IMG_001.JPG\nIMG_002.JPG\n' | sed -E 's/^IMG_([0-9]+)\.JPG$/photo-\1.jpg/'

The first removes leading and trailing whitespace with one alternation and the g flag. The second squeezes any run of blanks and tabs to one space. The third extracts version strings with an optional pre-release suffix. The fourth is a pragmatic email extractor that accepts the common shapes and ignores the complexity of the full address grammar. The fifth pulls URLs by taking everything up to whitespace or a quote. The sixth strips terminal color codes by matching the escape character, a bracket, digits and semicolons, and an m. The seventh joins lines that end with a backslash continuation, using sed's N command in a loop. The last renames by restructuring a file name with a capture group, a pattern that feeds a mv in a loop.

for f in IMG_*.JPG; do
  new=$(printf '%s\n' "$f" | sed -E 's/^IMG_([0-9]+)\.JPG$/photo-\1.jpg/')
  printf 'mv -- %q %q\n' "$f" "$new"
done

Printing the mv commands first is a dry run: read them, and pipe to sh only when they look right. Bash can do this particular rename with parameter expansion and no sed at all, ${f#IMG_} and ${f%.JPG}, which is faster and safer for simple prefix and suffix changes. Regexes pay off when the structure is more complicated than a prefix and a suffix.

Escape text for use inside a pattern

When a literal string must be embedded in a pattern, escape every character that is special in the dialect. The safest way is to avoid the situation: use grep -F or awk's index for a literal search. For sed, where there is no fixed-string mode, escape the pattern side and the replacement side separately, because they have different special characters.

escape_re()  { printf '%s' "$1" | sed -e 's/[][\.*^$/]/\\&/g'; }
escape_rep() { printf '%s' "$1" | sed -e 's/[\/&]/\\&/g'; }
old='a.b[1]'
new='x/y&z'
echo 'see a.b[1] here' | sed "s/$(escape_re "$old")/$(escape_rep "$new")/"

see x/y&z here

On the pattern side the helper escapes brackets, dot, star, caret, dollar and the delimiter; on the replacement side only the delimiter, the ampersand and the backslash matter. Both are for BRE. If the input may contain a newline, neither is enough, and the right answer is to stop using sed for that substitution. The general advice from the security guide applies: treat the value as data, and prefer tools that take it as data, over escaping it into code.

Read and document a regex

A pattern is write-once code unless it is explained. When you meet an unfamiliar one, read it left to right and name each piece: an anchor, a group, a class, a quantifier. Break it onto separate lines mentally at every group and alternation. The pattern ^([a-z][a-z0-9_-]{2,31})@([a-z0-9.-]+)$ reads as: start of line, a name that begins with a letter and continues with letters, digits, underscores or hyphens up to 32 characters in all, an at-sign, a host made of letters, digits, dots and hyphens, end of line. Written out in words it is clear; as a bare string it is a puzzle, and a script full of them is hard to change safely.

So document patterns where they are defined. Put each in a variable with a name that says what it matches, and give it a comment that includes one example that matches and one that does not. Build a long pattern from named parts joined together, which shortens each piece and lets you test them alone. Bash has no extended mode that ignores whitespace and comments, which Perl and PCRE do with the x flag, but string concatenation gets most of the benefit.

# matches 10.0.0.1 and 192.168.1.20; the range of each octet is checked separately
octet='[0-9]{1,3}'
ipv4="^(${octet}\\.){3}${octet}$"

# matches app-42 and web-7; does not match App-42 or app_42
service='^[a-z]+-[0-9]+$'

[[ $1 =~ $ipv4 ]] && echo "address"
[[ $1 =~ $service ]] && echo "service name"

The comment with a matching and a non-matching example is a tiny test case, and copying those lines into a Bats test costs a minute. In code review, ask for them. A regular expression with no example attached is a claim nobody can check, and the sibling habit of testing against negative cases, described earlier, is what catches the pattern that matches too much.

Regexes in editors and pagers

The patterns you type into an editor are another dialect to keep apart from the shell tools. Vim's default search treats most operators as literals until escaped, much like BRE, and its "very magic" mode, entered with \\v at the start of the pattern, makes it behave like ERE. Less uses the system regex library for its slash search. Many graphical editors use the Perl-style engine of their language. When you design a pattern interactively in an editor and then paste it into a script, retest it in the script's tool, since the dialect rules differ and the first symptom is a pattern that finds nothing or too much. Sed's own substitute command is borrowed by vim as the colon-s command, which is a reason the two feel alike and still differ in the escapes they need.

Regexes in other languages, briefly

The skills here carry over to scripting languages with adjustments. Python's re, JavaScript, Java, PHP's preg_* and Ruby are all Perl-style backtracking engines, so they accept \d, lazy quantifiers and lookaround, and they share the risk of catastrophic backtracking. Go's regexp and Rust's regex crate use linear-time automata like grep, and so they omit backreferences and lookaround by design. PowerShell uses .NET syntax. The ripgrep tool uses the Rust engine with optional PCRE2 through -P.

That split gives a rule for choosing: when you need guaranteed running time on untrusted input, pick an automaton engine and live without lookaround; when you need lookaround and backreferences, pick a backtracking engine and bound the input. Within shell, GNU grep gives you both, selected by -E and -P, and knowing which you are using is the safeguard.

Parsing structured lines with a stack of small patterns

Real log lines carry several kinds of data, and one giant regex for the whole line is hard to write and harder to read. A stack of small patterns, each extracting one field, is easier to test and to maintain. Take a line from an application log and pull the pieces separately, then combine them.

line='2026-11-05T10:01:03Z level=error user=alice req=8f3a latency=231ms msg="upstream timeout"'
printf '%s\n' "$line" | grep -oE 'level=[a-z]+' | cut -d= -f2
printf '%s\n' "$line" | grep -oE 'latency=[0-9]+' | cut -d= -f2
printf '%s\n' "$line" | sed -nE 's/.*msg="([^"]*)".*/\1/p'
printf '%s\n' "$line" | awk '{for (i = 1; i <= NF; i++) if ($i ~ /^user=/) {sub(/^user=/, "", $i); print $i}}'

error
231
upstream timeout
alice

Each pattern is short enough to read at a glance, and each can be tested alone. The key-value form makes this tolerant of fields appearing in any order, which a single positional pattern would not survive. For the quoted message, a negated class between the quotes is the right tool. When logs are in a structured format such as JSON lines, skip the patterns altogether and use jq, as the end of this guide recommends. The point is to prefer several simple extractions over one clever one.

Anchoring strategies and the cost of unanchored patterns

An unanchored pattern may match in many places in a line, and the tools handle that differently. With grep you only learn that the line matches. With sed and awk substitution, the leftmost match is replaced. With grep -o, every non-overlapping match is printed. Anchors and boundaries decide what you get, and a few habits keep results predictable. Anchor to the line start and end when the whole line has a known shape. Use word boundaries or -w when you mean a token and not a substring. Use explicit delimiters, as in =[^ ]*, when matching a value in a key and value line.

printf 'port=80 sport=8080 port=8081\n' | grep -oE 'port=[0-9]+'
printf 'port=80 sport=8080 port=8081\n' | grep -oE '(^| )port=[0-9]+' | tr -d ' '
printf 'port=80 sport=8080 port=8081\n' | grep -owE 'port=[0-9]+'

port=80
port=8080
port=8081
port=80
port=8081
port=80
port=8081

The first pattern matched inside sport=8080, a result that looks right until you count. The second requires the key to start the line or follow a space. The third uses -w to require word boundaries around the match, which also excludes sport, because the letter before port is a word character. Every pattern that fails by matching too much has this shape, and the fix is a boundary.

Replacing with function-like logic in awk

The substitution in sed can only insert text taken from the match. When the replacement needs a computation, such as converting units, formatting a number, or looking up a value, awk can match with one regex and build the output with code. The match function finds the text, and substr and arithmetic transform it. gawk goes further with gensub and with a function as a replacement in some forms.

printf 'sent 1048576 bytes, received 2048 bytes\n' |
awk '{
  out = ""; rest = $0
  while (match(rest, /[0-9]+ bytes/)) {
    n = substr(rest, RSTART, RLENGTH - 6) + 0
    out = out substr(rest, 1, RSTART - 1) sprintf("%.1f KiB", n / 1024)
    rest = substr(rest, RSTART + RLENGTH)
  }
  print out rest
}'

sent 1024.0 KiB, received 2.0 KiB

The loop finds each number followed by the word bytes, divides, formats and keeps scanning the rest of the line. The same job in sed would need a separate arithmetic step. It is a good example of a rule of thumb: a regex finds and delimits, and code computes. When the regex part starts to include logic, split it, and the result is both easier to debug and easier to extend.

Regexes for dates, times and numbers

Dates and numbers are the most common things to extract, and the patterns are worth keeping at hand with their limits clear. A pattern for an ISO date checks the shape, not the calendar. Numbers have more shapes than they appear to: signs, decimals, thousands separators and exponents.

For anything that must be correct, check the value with the tool that understands it after the regex has found it: date -d "$value" +%F rejects an impossible date, and an arithmetic test rejects an out-of-range number. A regex followed by a semantic check is the combination that works. Putting the check in the regex alone grows patterns that nobody can read and that still miss cases.

Regular expressions in everyday administration

Regexes earn their keep in routine tasks beyond log reading. Finding processes by name with pgrep -f, selecting services with systemctl list-units --type=service | grep -E 'nginx|php', filtering the journal with journalctl --grep, matching hosts in a config with a pattern, and editing config values in place all use them. The danger in administration is that a pattern that matches too much acts on the wrong object. pkill -f python kills every process whose command line mentions python, including your editor's language server and other people's jobs.

pgrep -af 'python worker\.py'
pkill -f '^python worker\.py --queue=billing$'
systemctl list-units --type=service --no-legend | awk '$1 ~ /^(nginx|php[0-9.]+-fpm)\.service$/ {print $1}'
sudo sed -i.bak -E 's/^(PermitRootLogin)[[:space:]]+.*/\1 no/' /etc/ssh/sshd_config

The habit that prevents accidents is to run the selecting form first, with pgrep -af or a plain grep, look at what it matches, and only then run the destructive form with the same pattern. The anchored pattern in the second command matches one exact command line. The last command rewrites a setting in a config file and keeps a backup, with the pattern anchored at the start of the line so a commented example is untouched; check the result with sshd -t before reloading, as the SSH guide advises in the hardening section.

Learning and practicing

Regex fluency comes from practice with feedback, and the cheapest feedback loop is a terminal and a file of sample lines. Collect real lines from your own logs and configs, save them in a file, and try patterns against them with grep -n and grep -o, which show exactly what matched. Make the patterns fail on purpose: add a line that should not match and see whether it does. Read the regex section of the tool's manual once, which takes twenty minutes and answers most future questions. For visual feedback, online testers highlight matches as you type, but they use a particular dialect, usually PCRE or JavaScript, so confirm the final pattern in the tool that will run it.

cat > samples.txt <<'EOF'
web-01 up 12d
web-02 down
db_01 up 3d
cache-1 up 0d
EOF
grep -nE '^[a-z]+-[0-9]+ up' samples.txt
grep -noE '[0-9]+d$' samples.txt
grep -vnE '^[a-z]+[-_][0-9]+ ' samples.txt

1:web-01 up 12d
4:cache-1 up 0d
1:12d
3:3d
4:0d

The third command printed nothing, meaning every line matched the looser pattern, an observation that tells you something about the data. Keep your sample file next to the script that uses the pattern and turn it into a test, as the testing guide describes. Patterns built from real samples, with the negative cases included, are the ones that survive contact with production.

Where this does not apply

Nested and structured formats. HTML, XML, JSON, YAML and source code have nesting that regular expressions cannot match in general. Use jq, xmllint, a YAML parser or a language-aware tool. A regex is acceptable for a quick look at a trusted, flat piece of such a file, and a hazard in a program that must be right.

Quoted CSV. Fields with embedded delimiters need a CSV parser, as covered in the grep, sed and awk guide.

Meaningful validation. Dates, addresses and numeric ranges need code. A regex can check shape and nothing more.

Untrusted input in a backtracking engine. Anything that takes patterns or text from outside and runs them in PCRE, Python or JavaScript needs a time limit or a linear-time engine such as RE2.

Very large searches. For gigabytes of text, a tool like ripgrep, or an index, will beat a regex scan with grep.

Windows and PowerShell. .NET regular expressions are another dialect with named groups and lookbehind; the shell-tool rules here do not carry over directly.

What to read next

To put these patterns to work, read the grep, sed and awk guide, where each tool is used on logs, configs and CSV. For the Bash side, including [[ =~ ]] inside scripts, see the Bash scripting tutorial. When a pattern runs on input you do not control, the shell script security guide covers validation. And for a wider tour of the commands that take these patterns, the Linux command line guide is the place to continue.

Primary sources used here: the POSIX regular expressions chapter, regex(7), the GNU grep manual, the GNU sed manual, the GNU Awk User's Guide, the Bash manual on conditional constructs, the GNU findutils manual, the PCRE2 pattern documentation, the mawk manual and the BusyBox documentation.