How to Debug Encoding and Locale Problems the Right Way

An "é" that appears as "é", a sort that puts lower-case before upper-case on one machine and not another, a ${#var} that counts bytes instead of characters: all are the same family of bug, a mismatch between the encoding of the data and the locale of the program reading it. The diagnosis is mechanical once you look at bytes. This post follows the order that finds the cause fastest, as part of the shell debugging guide. Examples use Debian 13 with glibc 2.41.

Look at the bytes

printf 'café\n' | hexdump -C
file -i names.txt

00000000  63 61 66 c3 a9 0a                                 |caf..|
names.txt: text/plain; charset=utf-8

In UTF-8, "é" is two bytes, c3 a9; in Latin-1 it is the single byte e9. If a file shows c3 83 c2 a9 for it, the text was converted from Latin-1 to UTF-8 twice, and each byte was treated as a character: that is the "é" pattern. file -i reports a guess based on the content, which is reliable for UTF-8 and only a hint for 8-bit encodings.

Look at the locale

locale
locale -a | head
echo "${#var}"
LC_ALL=C bash -c 'v="café"; echo ${#v}'

LANG=en_US.UTF-8
LC_ALL=
5

The locale decides how bytes are read as characters: string length, case conversion, sorting, character classes in regular expressions. With LC_ALL=C the string "café" has length 5 (bytes), under a UTF-8 locale 4 (characters). If locale prints warnings such as Cannot set LC_ALL to default locale, the named locale is not generated on the system; locale -a lists what exists, and sudo dpkg-reconfigure locales or locale-gen adds one.

Where the locale goes missing

LANG=C.UTF-8
LC_ALL=C.UTF-8

Cron jobs and systemd services run without your login environment, so they use the POSIX C locale unless told otherwise; Docker images usually have none set either. A Python script that prints non-ASCII text then fails with UnicodeEncodeError under cron but works in your terminal. Set LANG=C.UTF-8 in the crontab or the unit (Environment=LANG=C.UTF-8) or the Dockerfile (ENV LANG=C.UTF-8). C.UTF-8 exists on current Debian, Ubuntu and Fedora without generating anything. Python 3.7 and later also coerces the C locale to UTF-8 for its own text handling.

Convert

iconv -f LATIN1 -t UTF-8 old.txt > new.txt
iconv -f UTF-8 -t ASCII//TRANSLIT notes.txt
iconv -f UTF-8 -t UTF-8 -c dirty.txt > clean.txt

The source encoding must be right. Converting a UTF-8 file as if it were Latin-1 produces the doubled-up mojibake. //TRANSLIT replaces characters that do not exist in the target with look-alikes. -c drops invalid sequences. For Windows files, the source is often CP1252, not Latin-1: the euro sign and curly quotes live in the range where they differ.

Byte order marks

head -c 3 data.csv | hexdump -C
sed -i '1s/^\xEF\xBB\xBF//' data.csv

A UTF-8 file saved by some Windows tools starts with the three bytes ef bb bf. Most programs show nothing, and the first column name of a CSV then carries an invisible prefix, so a lookup of id fails while the file looks fine. Remove it as shown, or read with Python's utf-8-sig encoding.

Sorting and matching

printf 'b\nA\na\nB\n' | sort
printf 'b\nA\na\nB\n' | LC_ALL=C sort

In an English UTF-8 locale the first prints a A b B, in the C locale A B a b. Scripts that compare sorted output, use comm or join, or depend on a fixed order should set LC_ALL=C for those commands, which is also faster. A regular expression like [a-z] may match accented letters in some locales; use classes such as [[:lower:]] deliberately.

Where this does not apply

Binary files are not text, and conversions corrupt them. Filenames on Linux are bytes without an encoding, so a name in an old encoding looks broken in a UTF-8 terminal (convmv can rename them). For speed with large files, see LC_ALL=C for speed, and for the line-ending relative of this problem, line endings.

What bytes and characters are

A file contains bytes, and encoding is the rule that turns bytes into characters. UTF-8 is the common one: ASCII characters take one byte, accented letters two, many other scripts three, and emoji four. Older files may use Latin-1 or another single-byte encoding, where every byte is one character, and Windows files may use UTF-16, where nearly every character takes two bytes with a zero byte in between for ASCII text. When a program assumes the wrong encoding, the symptoms are specific. UTF-8 read as Latin-1 turns an accented letter into two odd characters, such as an A with a tilde followed by a symbol, a pattern called mojibake. Latin-1 read as UTF-8 produces replacement characters or errors about invalid sequences. UTF-16 read as anything else shows letters with spaces between them and a program that treats the file as binary. Recognizing these patterns tells you which conversion is needed before you try one.

Tools help: file guesses an encoding and the line ending style, and od -c or hexdump -C show the real bytes, which settles arguments about what a file contains in a way that the terminal's rendering cannot.

What the locale controls

The locale is a set of environment variables, LANG as the default and LC_ALL as an override, plus category variables such as LC_CTYPE and LC_COLLATE. They decide how the C library classifies characters, sorts text and formats numbers. In a UTF-8 locale, ${#s} counts characters, [[:alpha:]] matches letters from many alphabets and sort follows the language's rules. In the C or POSIX locale, text is treated as bytes, a length counts bytes, and sorting is by byte value, so uppercase sorts before lowercase. The same script gives different results under each, which is the root of many failures that appear on one machine only. locale prints the current settings and locale -a lists the ones installed. A locale that is named but not installed falls back to C with a warning from some programs, and a minimal container often has only C and C.UTF-8. Set LC_ALL=C.UTF-8 in such an image for predictable Unicode behavior.

Where the setting disappears

An interactive terminal usually has a UTF-8 locale set by the desktop or the login. Cron jobs, systemd services, Docker containers and non-interactive SSH commands often do not, and fall back to C. A script that runs fine by hand then mangles accented names under cron: counting bytes instead of characters, failing a case-insensitive match, or writing a file whose name is garbled. Set the locale explicitly where the script starts, either in the crontab, in the unit with Environment=LANG=C.UTF-8, or in the script with export LC_ALL=C.UTF-8. Conversely, set LC_ALL=C on purpose when you want byte semantics and speed, for example for sorting large ASCII logs, as the speed-up post describes. The point is to choose a locale in the script and not inherit whatever the caller had, since inheritance is the way identical code gives different answers on two machines, and neither is wrong, just different.

Be careful with file names. On macOS, the file system stores names in a decomposed Unicode form, so a name typed with a precomposed accented letter may not compare equal to the name that ls returns, and a script that builds names by hand can miss the file.

See locale(7) and iconv(1).