How to Use Golden Files for Command Output Tests Safely

A command that prints a table, a report or a generated config file is awkward to test: asserting twenty lines one by one is noisy, and a change in one place breaks tests in ten. A golden file turns the whole output into a single expected value. When it changes, the test shows the diff and you decide whether the change is intended. This post builds the pattern in shell, with the normalization and update steps that make it workable. It complements Bats tests and asserting output and status. Examples use Bash 5.2 and Bats 1.11.

The basic test

golden() {   # name, actual-output-file
  local expected="tests/golden/$1.txt"
  if [[ ${UPDATE_GOLDEN:-0} == 1 ]]; then
    mkdir -p tests/golden
    cp "$2" "$expected"
    return 0
  fi
  diff -u "$expected" "$2"
}

@test "list prints the inventory table" {
  run ./inventory list --fixtures tests/fixtures
  [ "$status" -eq 0 ]
  printf '%s\n' "$output" > "$BATS_TEST_TMPDIR/out.txt"
  golden list "$BATS_TEST_TMPDIR/out.txt"
}

The helper compares the output with the stored file and, when UPDATE_GOLDEN=1 is set, overwrites the file instead. diff -u exits non-zero on a difference, which fails the test, and its output appears in the Bats failure message. The first run needs the update mode to create the file.

Normalize what changes

normalize() {
  sed -E \
    -e 's#'"$PWD"'#/PROJECT#g' \
    -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9:.]+Z?/TIMESTAMP/g' \
    -e 's/pid=[0-9]+/pid=PID/g' \
    -e 's/[0-9a-f]{40}/SHA/g'
}
run ./inventory list --fixtures tests/fixtures
printf '%s\n' "$output" | normalize > "$BATS_TEST_TMPDIR/out.txt"

Replace the parts that legitimately differ on every run with fixed placeholders, so the golden file does not break at midnight or on another machine. The project path is the most common: a developer's checkout and CI's are different. Be specific: a broad pattern that masks too much hides real changes. Also make the environment deterministic with LC_ALL=C, TZ=UTC and NO_COLOR=1, and sort where the command's order is not guaranteed.

Updating and reviewing

UPDATE_GOLDEN=1 bats tests/golden.bats
git diff tests/golden/

When you change the output on purpose, regenerate and read the diff before committing. The diff is the review: reviewers see exactly how the output changed in the pull request. The discipline that makes golden tests valuable is reading that diff. The habit that ruins them is running the update whenever a test fails, which records whatever the code does as the truth.

What they suit

• Help text and usage output, so documentation and behavior cannot drift.
• Reports and tables with many columns.
• Generated files such as configs, manifests and code.
• JSON output, compared after jq -S . to fix key order.

For exact values that matter, such as an exit code or one specific field, a direct assertion is clearer than a golden file; use both.

Where this does not apply

Large golden files (thousands of lines) make diffs unreadable and encourage blind updates; split them or assert on the parts that matter. Output that is nondeterministic by nature, such as timing or random identifiers, cannot be normalized well and needs property-style checks instead. Binary output needs a tool-specific comparison. The same technique in other languages is shown in pytest for CLI tools and Go table-driven tests.

What the technique is good at

A golden file test stores the expected output of a command in a file and compares the real output with it, failing on any difference. It suits output that is large, structured and hard to assert piece by piece: help text, formatted reports, generated configuration, compiler diagnostics, table layouts. Writing twenty assertions for a report would be tedious and miss most of it, and one comparison covers every character. The cost is brittleness: any change to the output, intended or not, fails the test, so the files must be easy to review and update. The technique also gives you a readable record of what the program prints, kept next to the code, which doubles as documentation and shows up in code review as a diff when behavior changes. That diff is the main benefit: a reviewer sees exactly what users will see differently, and an accidental change in spacing or ordering cannot slip through unnoticed.

Use it where the exact output is the requirement. For logic, such as whether a number is computed correctly, a targeted assertion states the intent better than a pile of text that happens to contain the answer.

Removing the noise that changes between runs

A golden test is only useful if the output is the same every run, so find and fix every source of variation. Timestamps and durations change, so replace them with a placeholder using a filter or pass a fixed clock into the program. Temporary directory names, process IDs, usernames and absolute paths differ between machines, so normalize them with sed before comparing. Map and set ordering in some languages is random, so sort output or make the program's order deterministic, which is good practice anyway. Terminal colors add escape codes, so run with colors disabled, with the NO_COLOR convention or a flag. Pin the locale and the terminal width, since both change formatting and wrapping. Line endings and the final newline matter on Windows. Do the normalization in one helper function used by every test, so the rules are visible in one place and applied the same way, and be careful that the filters themselves do not hide real differences.

Updating files without losing the point

When the output changes on purpose, regenerating the golden files should be one command, usually an environment variable or flag such as UPDATE=1 or -update, which writes the new output over the old files. The danger is that updating becomes a reflex: a failing test is "fixed" by regenerating the file with nobody reading the diff. Counter it with process. Treat golden file changes as part of the change under review, look at the diff of the files as carefully as the code, and have the pull request description say why the output changed. Keep each file small and focused on one case, so a diff is short enough to read. Name them for what they show. Run the tests in CI without the update flag, so a stale file fails the build, and make sure the update mode does not run there. The goal is a test that fails for real regressions and that people trust enough to read when it does.

Tooling and where to keep the files

The convention in most languages is a testdata or fixtures directory beside the tests, with one file per case named for what it shows, and a pair of input and expected output files when the input is large. In Go, files under testdata are ignored by the build, and tests read them by relative path. In Python, libraries such as pytest-regressions and syrupy manage snapshot files and the update command, and for shell tests, diff against the stored file inside a Bats test gives the same result. Whatever the tool, keep the files in version control, in plain text, with a stable newline convention, so the diffs are clean. Avoid snapshots of huge outputs. A thousand-line golden file that changes with every feature is too big to review, and splitting the output into smaller, focused cases keeps each file meaningful and each failure easy to read.

See the unified diff format and the Bats special variables.