How to Use Regex in Python re Module for Scripts Easily

When a pattern outgrows grep or sed, Python is the usual next step: the engine is a full backtracking one with groups, lookarounds and named captures, and the surrounding language handles the logic. The module has a handful of functions that cover almost everything. This post goes through them with a log-parsing example, as the Python entry in regex in Python scripts. Examples use Python 3.13.

Search, match, fullmatch

import re

line = "2026-11-03 09:01:10 ERROR upstream timeout after 30s"

print(re.search(r"ERROR", line))
print(re.match(r"ERROR", line))
print(re.fullmatch(r"\d{4}-\d{2}-\d{2}", "2026-11-03") is not None)

<re.Match object; span=(20, 25), match='ERROR'>
None
True

search finds the first match anywhere. match only looks at the start of the string, which surprises people coming from other languages; the second line returns None because the line starts with a date. fullmatch demands that the entire string match, which is the right tool for validating a value and saves writing ^ and $. Always test a result against None before using it.

Groups and named groups

pattern = re.compile(r"""
    (?P<date>\d{4}-\d{2}-\d{2})\s+
    (?P<time>[\d:]+)\s+
    (?P<level>[A-Z]+)\s+
    (?P<msg>.*)
""", re.X)

m = pattern.match(line)
if m:
    print(m["level"], m["msg"])
    print(m.groupdict())

ERROR upstream timeout after 30s
{'date': '2026-11-03', 'time': '09:01:10', 'level': 'ERROR', 'msg': 'upstream timeout after 30s'}

Named groups document the pattern and survive later edits that would renumber positional ones. re.X (verbose) lets you spread the pattern over lines with whitespace and comments, which is how a long pattern stays readable. Compile once with re.compile when the pattern is used in a loop; the module caches recent patterns, but an explicit object is clearer.

All matches

text = "retry in 5s, then 30s, then 120s"
print(re.findall(r"\d+(?=s)", text))
for m in re.finditer(r"(\d+)s", text):
    print(m.start(), m.group(1))

['5', '30', '120']
11 30

findall returns a list of strings, or of tuples when the pattern has several groups. finditer yields match objects with positions, which is what you need to edit the text in place. The second output is abbreviated for space; each match prints its start and group.

Replace, with a function

print(re.sub(r"(\d+)s", lambda m: f"{int(m[1]) * 1000}ms", text))
print(re.sub(r"\s+", " ", "a   b \n c"))

retry in 5000ms, then 30000ms, then 120000ms
a b c

When the replacement is a function, it receives the match object and returns the string, so computed replacements are straightforward. With a string replacement, \1 and \g<name> refer to groups. Use re.escape(user_text) to build a pattern from literal text.

Flags and traps

re.M makes ^ and $ match at each line; without it they match only at the ends of the string. re.S lets the dot match newlines. re.I ignores case. A pattern such as (a+)+$ can take exponential time on input like aaaa...!; the engine has no built-in limit, so never run an untrusted pattern or apply a complicated one to untrusted input without a timeout. Python 3.11 added possessive quantifiers and atomic groups, which prevent some backtracking.

Where this does not apply

For a quick line filter, grep is shorter; for structured data (JSON, HTML, CSV) use a parser, not a regex. Lookbehind needs a fixed width in re; the third-party regex module lifts that and adds more, at the cost of a dependency. Related flavors are compared in regex across tools, and testing is covered in debugging regex.

The three matching functions and what they anchor

The module offers three entry points that differ in where the match may start. re.match anchors at the beginning of the string, so it succeeds only when the pattern matches from position zero. re.search scans for the first position where the pattern matches anywhere. re.fullmatch requires the whole string to match. Mixing them up is the most common bug: a validation that uses match accepts "123abc" as a number because the leading digits match, while fullmatch rejects it. The dollar sign adds a subtlety, since it matches at the end of the string and also just before a trailing newline, so $ alone lets "abc\n" through. Use fullmatch for validation, or \Z to mean the true end. All of them return a match object or None, so test the result before using it, or the next line raises AttributeError on the unlucky input.

Write patterns as raw strings, with the r prefix, so that backslashes reach the engine unchanged. Without it, "\b" is a backspace character, and the mistake produces patterns that match nothing.

Groups, findall and the surprise in its return value

The result of re.findall depends on the number of groups in the pattern. With none, it returns a list of the matched strings. With one group, it returns a list of what that group captured, not the whole match. With several groups, it returns a list of tuples. That switch is a classic trap: adding parentheses for grouping a repetition changes what the function returns. When you want the whole match and also use groups for structure, use non-capturing groups, written (?:...), or use finditer and read the match objects, which give you everything. Named groups, (?P<name>...), make code readable, and m.groupdict() returns them as a dictionary. For a pattern used many times, compile it once with re.compile, which also gives you a place to attach a descriptive name and a comment.

Use the verbose flag, re.VERBOSE, for long patterns. It ignores whitespace and allows comments inside the pattern, which turns an unreadable line into a documented one.

Replacement, Unicode and the cost of backtracking

In re.sub, the replacement may be a string, in which \1 and \g<name> refer to groups, or a function that receives the match object and returns the text, which allows any computation, such as looking up a value or converting a number. The shorthand classes follow Unicode by default for text strings, so \d matches digits from other scripts and \w matches letters with accents, which may or may not be what you want. Add re.ASCII to restrict them. Nested quantifiers on overlapping alternatives, such as (a+)+b, can take exponential time on a string that almost matches, and an unlucky input can freeze a service, which is why regular expressions applied to untrusted input deserve a review. Limit the input length, avoid nested repetition, and consider the third-party regex module, which supports timeouts, or a linear-time engine for hostile data.

Habits that keep patterns maintainable

Give each non-trivial pattern a name and a comment, and put it near the code that uses it. Test it with a table of examples, including cases that must not match, using a few lines of pytest with parametrize, so a later edit cannot quietly widen it. Prefer small patterns combined in code over one enormous pattern, since splitting a line with str.split and checking each piece is often clearer than a pattern that does everything. Use the standard library's parsers for formats that have them: json, csv, urllib.parse, email and html.parser. A regular expression that parses a structured format works on the examples you tried and fails on the one you did not, and the fix is always the parser you should have used first.

See the re module documentation and the Regular Expression HOWTO.