How to Run Bats Tests in CI and Avoid the Pitfalls

Bats tests that pass on a laptop and fail in CI almost always depend on something the runner does not have: the helper libraries, a tool version, a locale or a leftover file. The fix is a small, explicit CI job. This post builds one and lists the pitfalls, as the pipeline side of testing shell scripts with Bats. Examples use bats-core 1.11 and GitHub Actions; the commands apply to any CI.

A workflow

name: test
on: [push, pull_request]

jobs:
  bats:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true
      - name: Install bats and parallel
        run: |
          sudo apt-get update
          sudo apt-get install -y bats parallel shellcheck
          bats --version
      - name: Run tests
        run: |
          mkdir -p reports
          bats --jobs 4 --print-output-on-failure \
               --report-formatter junit --output reports tests/
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: bats-report
          path: reports/

Ubuntu 24.04 ships Bats in its package archive, which is convenient and a fixed version; if you need a newer one, install from a Git tag instead: git clone --depth 1 --branch v1.11.0 https://github.com/bats-core/bats-core.git && sudo ./bats-core/install.sh /usr/local. Pinning the version keeps a new release from changing results. if: always() uploads the report even when tests fail, which is when you want it.

Helper libraries

load "$BATS_TEST_DIRNAME/../test/bats-support/load.bash"
load "$BATS_TEST_DIRNAME/../test/bats-assert/load.bash"

If bats-support and bats-assert are Git submodules, the default checkout does not fetch them, and the tests fail with load: file not found. Hence submodules: true. In GitLab the equivalent is GIT_SUBMODULE_STRATEGY: recursive. See bats-assert and bats-support for the assertions.

Output formats

bats --tap tests/
bats --formatter pretty tests/
bats --report-formatter junit --output reports tests/

TAP is the portable plain format; pretty is the human default; the JUnit report goes to a file that most CI systems display as a list of tests with durations and failure messages. The output directory must exist.

Parallel runs

With --jobs 4, files run in parallel, and with --no-parallelize-within-files tests inside one file stay serial. It needs GNU parallel installed, or Bats stops with an error. Tests that write to a shared path or a fixed port will collide, so give each test its own directory with BATS_TEST_TMPDIR, which Bats creates and removes. Flaky failures that appear only with jobs above 1 usually come from shared state.

Environment traps

• macOS runners have BSD tools and Bash 3.2; a test using sed -i or readlink -f passes on Linux and fails there. Run the matrix you ship for, as discussed in scripts for macOS and Linux.
• Locale and time zone differ: set LC_ALL=C and TZ=UTC in the job to make sorting and dates stable.
• CI has no TTY, so tests that check terminal behavior or colors need script or an explicit override.
• Tests that run the script under test should call it by path, not rely on PATH, which differs.

Where this does not apply

Bats gives you pass or fail for command behavior, not coverage; see coverage with kcov for that. Run shellcheck in the same pipeline, as in ShellCheck in CI, since it catches what tests do not. Slow integration tests belong in a separate job so that quick feedback stays quick.

What run captures and what it hides

The run helper in Bats executes a command, captures its exit status in $status and its combined output in $output, and splits the output into the array $lines. It runs the command in a subshell, so anything the command does to variables, the working directory or shell options is gone when it returns. A test that calls a shell function through run and then checks a variable the function set will find it unchanged, and the failure looks like the function did nothing. To test side effects on the shell, call the function directly, outside run, and check the variables. Standard error is merged into $output by default, which hides which stream a message used. Newer versions accept run --separate-stderr, which fills $stderr separately, and that is the right choice when a script promises clean output on standard output. A failing test prints the output that run captured, so a message that appears only in a failure is a good reason to run a command even when you do not assert on its output.

Each test also runs with set -e in effect inside the test body, so a failing command that is not wrapped in run or in a conditional ends the test as failed. That is helpful for setup steps and surprising in an assertion like [ "$a" = "$b" ] placed before the end of the test, where Bash does not stop on a failed test in some positions. Use the assert helpers, which fail explicitly.

Setup, teardown and isolation

Every test should start from a known state and leave nothing behind. Define setup to create a temporary directory with BATS_TEST_TMPDIR, which Bats creates for each test and removes afterwards, copy fixtures into it and change into it, so a test can create, rename and delete files freely. Use setup_file for expensive work that is shared, such as starting a local server or building a binary, and teardown_file to clean it up. Keep fixtures small and checked into the repository next to the tests, and never point a test at your real home directory or a production path. Set HOME and the PATH inside setup so the script under test cannot read your configuration or call tools you did not intend, and put a directory of stub commands at the front of the PATH to fake curl or docker for tests that should not touch the network.

Name tests as sentences describing behavior, such as "rejects a missing argument" or "keeps the original file when the copy fails". When one fails in CI, the name is the first thing a person reads.

Running them in CI, in parallel and on other platforms

Install Bats from a package, from the npm package, or by checking out the repository at a pinned tag into the workflow, and avoid latest so a new release cannot change the result. bats --formatter junit or -F junit writes a JUnit report that most CI services can display, and --tap prints the TAP format for tools that read it. Parallel runs with --jobs N require GNU parallel and tests that do not share state, which the temporary directory per test makes possible, though tests that bind a fixed port or touch a shared file will collide. macOS ships Bash 3.2, which lacks features that newer Bats versions use, so install a current Bash through the package manager and run Bats with it. Add a matrix with at least one Linux and one macOS runner if the script claims to support both, since the differences in sed, date and readlink show up in tests first.

Testing scripts that call other programs

Most real scripts call curl, git, docker or ssh, and tests should not depend on those working. Create a directory of small stub scripts that record how they were called and print a canned answer, put it first on the PATH in setup, and assert on the recorded arguments afterwards. This tests your script's logic and quoting without touching the network, and it makes failure cases easy: a stub that exits with an error shows how the script handles a failed download. Keep real integration tests few, in their own file, and run them where the dependencies exist.

See the bats-core usage guide and writing tests.