How to Test Regex Patterns and Debug Them the Safe Way

A regex that "almost works" is more dangerous than one that fails, because it ships. Debugging one has a method, and it is mostly about shrinking the problem. Pick a small sample, find the part that goes wrong and fix that, and then keep the examples as tests. It uses the ideas from greedy versus lazy matching and anchors and word boundaries. The examples assume GNU grep 3.11 and Python 3.13.

See what actually matched

echo "id=42 name=ada id=7" | grep -oE 'id=[0-9]+'
echo "id=42 name=ada id=7" | grep -oE 'id=.*'
echo "id=42 name=ada id=7" | grep -oE 'id=.*?'

id=42
id=7
id=42 name=ada id=7
id=42 name=ada id=7

-o prints only the matched part, one per line, so you see where each match begins and ends. The second command shows greedy matching swallowing the line, and the third shows that ERE has no lazy quantifier: .*? is accepted but does not behave as in PCRE. Switch to grep -oP for lazy matching, then check the target engine still supports what you used.

Build up in steps

pat='^[A-Z]'
for step in '^[A-Z]' '^[A-Z][a-z]+' '^[A-Z][a-z]+ [A-Z][a-z]+$'; do
  printf '%-32s' "$step"; printf 'Ada Lovelace\nada\nAda L\n' | grep -cE "$step"
done

^[A-Z]                          2
^[A-Z][a-z]+                    2
^[A-Z][a-z]+ [A-Z][a-z]+$       1

Starting with the simplest part and adding one piece at a time shows where matches drop out. Count matches with -c on a fixed set of lines. When the number jumps unexpectedly, the last addition is the cause.

Keep a table of cases

import re

pattern = re.compile(r"^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])$")

should_match = ["2026-11-03", "1999-01-31"]
should_not_match = ["2026-13-01", "2026-11-3", " 2026-11-03", "2026-11-03\n"]

for s in should_match:
    assert pattern.match(s), s
for s in should_not_match:
    assert not pattern.match(s), s
print("all cases pass")

all cases pass

Negative examples catch over-matching, and a pattern tested only on text it should accept can be .*. The last negative case is a lesson: in Python, $ matches before a trailing newline, so a validator should use re.fullmatch or \Z. In PHP, $ behaves the same way unless you use the D modifier. Paste these cases into the repository as a test, as in testing Bash functions.

Use a tool that explains the match

perl -Mre=debug -e '"abc" =~ /a.c/' 2>&1 | head -12
python3 -c "import re; re.compile(r'(\d+)-(\w+)', re.DEBUG)"

Interactive testers such as regex101 show each group, an explanation of every token and the number of steps the engine took, with a choice of flavour. They are third-party websites, so do not paste secrets or production data into them. Perl's debug mode and Python's re.DEBUG print the compiled program, useful when you suspect the engine reads the pattern differently from how you wrote it.

Common causes

Missing anchors cause partial matches (\d+ matches inside abc123def). Greedy quantifiers eat too much. An unescaped dot or a forgotten flag, like case-insensitive matching, hides cases. Mixing flavours, such as \d in grep -E (not supported in POSIX ERE), fails silently. And the shell may have altered the pattern before the tool saw it; print it with printf '%s\n' "$pat" to check, as covered in escaping special characters.

Debugging is a search problem

A failing regex has one known bad input and a pattern of twenty to eighty characters, so the cause lives in a small space. The efficient approach is bisection. Cut the pattern in half at a group boundary, test each half on its own against the part of the input it should cover, and the half that misbehaves holds the bug. Repeat until a single token is left. This beats re-reading the whole pattern, because people tend to read what they meant, not what they wrote, and a split forces a literal reading of each piece.

Shrink the input as well. Remove characters from the failing line until the behaviour changes, and the last removed character is a clue. If the pattern stops matching when a trailing space is removed, the pattern depends on that space. If a failure disappears with the line shortened to ten characters, the problem is length-related: anchoring, a repetition count or a backtracking limit.

Know which engine you are talking to

Most "the regex works in the tester but not in the script" reports are flavour mismatches. POSIX basic expressions in plain grep and sed treat +, ?, | and parentheses as literal unless escaped, extended mode in grep -E and sed -E treats them as operators, and Perl-compatible patterns add \d, lazy quantifiers and lookaround. JavaScript has its own gaps, including lookbehind support that arrived late and named groups that use a different syntax from Python's. Before debugging the pattern, confirm the engine, and test with that engine's own command rather than a website that defaults to another flavour.

The same text can also reach the engine in a different form from what you typed. A pattern inside double quotes in the shell has its backslashes and dollar signs interpreted first; a pattern in a Python string without the raw prefix has \b turned into a backspace character; a pattern in JSON needs each backslash doubled. When behaviour makes no sense, print the pattern as the engine received it, character by character, before suspecting the logic.

Flags and anchors, the quiet failures

Three settings change the meaning of a pattern without changing its text. Case-insensitive mode alters which characters a class matches. Multiline mode makes ^ and $ match at every line break instead of the ends of the string, and dot-all mode lets . cross newlines. A pattern checked against one line of input and then run on a whole file hits all three. If the result differs between a single line and the full text, compare the flag settings first.

Unicode is the other silent difference. \w matches only ASCII letters in some engines and all letters in others, and a name with an accent or a digit from another script passes in one tool and fails in another. If the data is not purely ASCII, include a non-ASCII sample in the test cases and state in a comment which behaviour is intended.

Turning the fix into a test

When the bug is found, write the failing input into the table of cases before changing the pattern. Watch it fail, apply the fix, watch it pass, and keep the row. The table then documents every mistake the pattern has made, which is the most useful comment a regex can have. Add a line above the pattern explaining what it is meant to match in words, since the next maintainer will otherwise have to reverse-engineer the intent from the symbols, and a pattern whose intent is unknown is one nobody dares to touch.

One more habit saves time: when a pattern works on the sample and fails on the real data, extract ten real lines that fail and ten that pass, and diff their characters. The difference is often a character class the sample never contained, such as a plus sign in an address, an apostrophe in a name or a double space after a comma. Real data is the best source of cases, and the first ten failures usually explain the other thousand.

Where this does not apply

When a pattern has grown past a screen, stop debugging it and rewrite it in pieces with comments, as in verbose mode, or replace it with a parser. For slow patterns, the cause is backtracking, a separate problem. See the Python regex HOWTO and the GNU grep manual.