How to Use Regex in Vim for Search and Replace the Right Way

Vim has had a regular expression engine for decades, with its own dialect that predates most of what you know from other tools. That is why a pattern that works in grep fails here, and why a few options are worth learning first: the substitute command and its flags, very magic mode and the global command. This post is the working set for editing scripts and config files, part of setting up Vim for shell scripting. Examples use Vim 9.1.

The substitute command

:s/foo/bar/          first foo on the current line
:s/foo/bar/g         every foo on the current line
:%s/foo/bar/g        every foo in the file
:%s/foo/bar/gc       ask before each replacement
:5,20s/foo/bar/g     lines 5 to 20
:'<,'>s/foo/bar/g   the visual selection

The range goes before s; % means all lines. With c, each match prompts for y (yes), n, a (all), q (quit) or l (last). The flag n reports the count of matches without changing anything, which is a good first check: :%s/foo//gn. If you use / in the pattern, choose another delimiter: :%s#/usr/local#/opt#g.

Magic and very magic

:%s/\(ab\)\+/X/g          default (magic): groups and + need backslashes
:%s/\v(ab)+/X/g           very magic: same meaning, ERE-like
:%s/\v(\d{4})-(\d{2})-(\d{2})/\3.\2.\1/

In the default mode, only a few characters (., *, [, ^, $) are special without a backslash. Adding \v at the start of the pattern makes nearly every punctuation character an operator, as in extended regular expressions, so groups, +, ?, |, {n} and < and > are written without the backslash. You can set the habit by mapping / to /\v in your vimrc. Character classes such as \d, \w and \s work in all modes.

Boundaries, groups and the replacement

:%s/\<cat\>/dog/g           whole word only (default mode)
:%s/\vlast, (\w+)/\1 last/    swap using a group
:%s/foo/&bar/                 & inserts the matched text
:%s/\s\+$//e                  strip trailing spaces; e hides the error if none

\< and \> match the start and end of a word (in very magic mode, plain < and >). & and \0 mean the whole match, \1 to \9 the groups. The e flag suppresses the "pattern not found" error, which matters in mappings and scripts. A tilde ~ in the replacement inserts the previous replacement string, so write \~ for a literal tilde.

Newlines

:%s/,/\r/g        break the line at every comma
:%s/\n\n\+/\r\r/  collapse runs of blank lines
:%s/\v(\w+)\n\1/\1/   remove a repeated word on the next line

On the pattern side, \n matches a line break. On the replacement side, \r inserts one; \n there inserts a null byte (shown as ^@), which is a common surprise. Backreferences inside the pattern, as in the third line, work because Vim's engine backtracks.

The global command

:g/^\s*#/d            delete all comment lines
:g!/ERROR/d           keep only lines with ERROR (also :v/ERROR/d)
:g/TODO/normal A !    append text to every line with TODO
:g/pattern/m0         reverse matching lines to the top

:g/pattern/command runs an Ex command on every matching line; :v (or :g!) inverts. Together with :s it covers most bulk edits. Undo works as one step for each command, with u.

Case and search

Put \c anywhere in a pattern to ignore case and \C to force case. The options ignorecase and smartcase together ignore case unless the pattern has an uppercase letter. Typing :%s//new/g with an empty pattern reuses the last search. Highlight and walk through matches with hlsearch, n and N.

Where this does not apply

Neovim uses the same dialect by default. Vim's engine is slower on huge files and long lines, and a regexp that needs lookahead has its own syntax (\@=, \@!). For edits across many files, sed or :argdo %s/.../.../ge | update is better; see sed regex replacement for the shell route and regex differences across tools for the comparison.

Magic modes and why patterns look different in Vim

Vim's default pattern syntax, called magic, is neither the basic nor the extended form from other tools. A few characters are special without a backslash, namely the dot, the star, the square brackets, the caret and the dollar sign. Many others need a backslash to become operators: groups are \( and \), alternation is \|, and the plus and question mark are \+ and \= or \?. That explains why patterns pasted from an extended-regex tool do nothing useful in the editor. Starting the pattern with \v switches to very magic mode, in which nearly every punctuation character is an operator, and the pattern reads much closer to extended syntax: \v(foo|bar)+ works as expected. Starting with \V switches to very nomagic, where only the backslash is special, which suits searching for literal text with odd characters in it. The choice can be made per pattern, and the setting magic itself is best left alone.

Word boundaries are written \< and \> for the start and end of a word, and character classes such as \d, \s and \w work in the editor, as does \zs to set where the reported match starts, which gives a cheap form of lookbehind.

Replacement details and newlines

In the replacement of :s, an ampersand stands for the whole match, and \1 to \9 for groups. A newline in the replacement is written \r, which inserts a line break, while \n inserts a NUL byte, displayed as ^@, which surprises people who expect it to mean a newline. In the pattern, \n matches a line break, and it can be used to join lines, as in :%s/\n\n\+/\r\r/. The flags are written after the last slash. g replaces every match on the line, c asks for confirmation on each, i ignores case and n only counts the matches without changing anything, which is a useful preview. An empty pattern reuses the last search, so a search with / followed by :%s//new/g replaces what you just looked at, with no retyping. The range before the command picks the lines: % for the whole file, numbers for a span, and '<,'> for the visual selection.

Because a mistake in a range-wide substitution is easy to make, preview with the n flag, and remember that u undoes the entire command as one step.

Working on matching lines and case rules

The :g command runs another command on every line that matches a pattern, and :v runs it on lines that do not. :g/^$/d deletes empty lines, :v/error/d keeps only lines with the word, and :g/pattern/normal Atext appends text to each matching line using normal-mode keys. The pattern can use the same magic modes as a search. Case handling is controlled by the options ignorecase and smartcase, which together make a search case-insensitive unless it contains an uppercase letter, and by the pattern items \c and \C that force the choice for a single pattern. Be aware that those options apply to :s as well, so a substitution can match more than expected if ignorecase is set in your configuration, and a script run on another machine behaves differently unless it sets the options it relies on.

Making edits repeatable and safe

A regular expression edit in the editor is easy to repeat with a macro or a command history, and easy to get wrong across a large file. Use the confirm flag on the first run, and the preview with n before a large replacement. For changes across many files, the argument list and :argdo %s/old/new/ge | update apply a substitution to each file, where the e flag suppresses the error for files with no match, and update saves only the files that changed. Check the result with version control, since a diff is the best review of a bulk edit. Put patterns you reuse in a mapping or a small function in your configuration. For edits that must be reproducible outside the editor, write the same change as a sed or script command, and keep it in the repository.

Beware of the special behaviors of the gdefault and edcompatible options, which change what the g flag does. A script that depends on defaults should set the options it needs.

See Vim's pattern documentation and the substitute command.