How to Handle Spaces and Special Characters in Filenames

Scripts that pass with report.txt break on Q3 report.txt and on a download named -rf. The cause is the shell's word splitting and option parsing, not the filename. The patterns below use Bash 5.2 and GNU findutils, and build on quoting variables and word splitting.

Quote and loop over globs

for f in ./*.txt; do
  [[ -e $f ]] || continue
  printf 'processing %s\n' "$f"
  wc -l -- "$f"
done

The shell expands the glob into one word per file before the loop runs, so spaces inside names are safe, provided "$f" is quoted in every use. The ./ prefix keeps a file named -n from being parsed as an option. The -e test skips the literal pattern that remains when nothing matches, unless shopt -s nullglob is set.

Never parse ls

for f in $(ls); do   # wrong
  wc -l "$f"
done

Command substitution output is split on spaces, so Q3 report.txt becomes two words, and a glob character in a name is expanded a second time. ls also changes how it prints names depending on the terminal. A glob, or find, hands over the names without that damage; the reasoning is documented in the ParsingLs page of the Wooledge wiki.

Recursive lists with NUL separators

find . -type f -name '*.log' -print0 | xargs -0 gzip --

while IFS= read -r -d '' f; do
  printf '%s\n' "$f"
done < <(find . -type f -name '*.log' -print0)

NUL is the one byte a filename cannot contain, so it is the safe separator. xargs -0 reads it, and IFS= with -r stops read from trimming whitespace or reading backslashes. The process substitution keeps the loop in the current shell so variables set inside it survive; see xargs and find -print0 and looping over files.

Names that start with a dash

touch -- -rf
rm -- -rf

-- ends option parsing, so -rf is treated as a file. When the command does not support it, or the name comes from a glob, prefix with ./.

What the shell does to an unquoted variable

Three separate steps happen to an unquoted expansion, and each one breaks filenames in its own way. First comes word splitting, using the characters in IFS, which by default are space, tab and newline. A name with a space becomes two arguments. Second comes pathname expansion, so a name that contains * or ? is replaced by whatever matches in the current directory, which can turn one argument into hundreds or into none. Third, if the result starts with a dash, the command may read it as an option. Quoting prevents the first two, and the double dash or a ./ prefix prevents the third. All three are needed, because each one covers a failure the others do not.

The same rules apply on the left of a test. In [ $f = x ] an empty or space-containing value changes the number of words the bracket sees and produces a syntax error or a wrong answer. Inside [[ ]] the shell does not split or expand the variable, which is one reason to prefer it in Bash. The right side of == inside [[ ]] is a pattern unless quoted, and that detail causes its own bugs when a name contains a bracket or an asterisk.

Arrays instead of strings

When a script collects several names to pass on, a string joined with spaces loses the boundaries. A Bash array keeps each name as its own element, and expanding it with "${files[@]}" produces one argument per element with the original spacing intact. The pattern is the correct way to build a command line from parts, such as a list of flags and a list of paths.

files=()
while IFS= read -r -d '' f; do
  files+=("$f")
done < <(find . -type f -name '*.csv' -print0)

printf 'found %d files\n' "${#files[@]}"
tar -czf csv.tar.gz -- "${files[@]}"

The array can be inspected with declare -p files, which prints it with quoting so that a hidden newline or trailing space is visible. That one command settles many arguments about what a variable actually holds. Arrays are a Bash and Ksh feature, so a script that must run under POSIX sh falls back to find -exec or to the positional parameters set with set --.

Unusual names worth testing against

A script that handles filenames should be tried against a directory of awkward ones before it runs on real data. Create names with a leading space, a trailing space, a tab, a newline, a leading dash, a glob character, a quote, a dollar sign and a non-ASCII letter. The test costs a minute and finds the quoting mistake that a normal name never will. Run the script on a copy, never on the only copy of anything, and compare the results with the expected list.

Some mistakes come from tools, not the shell. xargs without -0 splits on whitespace and treats quotes specially, so it fails on exactly the names above. cp and mv treat a name starting with a dash as an option. basename and dirname take their argument like any other command, and need -- too. Command substitution strips trailing newlines, so a name that ends in a newline loses it when captured with $(...). Each tool has a safe invocation, and the habit is to look for it each time a name moves through a new one.

Why not just rename everything

Cleaning names up is a legitimate fix when the files belong to you: replace spaces with hyphens, drop control characters, and keep the original in a mapping file. It does not help with files from users, backups or other systems, where the names are data and must be preserved byte for byte. Write the script to be correct for any name, and treat renaming as a convenience.

Filenames in other tools and languages

The same care continues after the shell. A name passed to ssh host "rm $file" is parsed twice, once locally and once on the remote side, so a space in the name splits it on the far end even though the local quoting looked right. Use printf '%q' to build a version that survives the second parse, or send the names on standard input to a remote xargs -0. Inside awk, a name with spaces is one field only if the field separator is changed, and sed expressions that embed a name need its delimiter characters escaped. When a script grows to this many special cases, moving the file handling into Python or Go, where a path is a value and not a string to be re-parsed, is a sensible step.

Where this does not apply

Printing names for display with echo can still mangle them, because echo interprets backslashes in some shells and drops a name like -n; use printf '%s\n'. A filename containing a newline will break any line-oriented tool downstream, such as sort and wc -l, unless it supports NUL input, for example sort -z. Check which tools in your pipeline do. POSIX sh has no read -d and no process substitution, so use find -exec cmd {} + there, described in the POSIX find specification. Names are also bytes, not text, so a script that cares about encoding should set the locale explicitly.