How to Remove Duplicates From a Bash Array and Keep Order

Deduplicating a list of hostnames or paths usually has to keep the first occurrence where it was, because the order means something, such as a search path. Bash 5.2 does not have a builtin for it, so the choice is between an associative array that remembers what it has seen and an external tool. The array syntax used below is explained in the Bash arrays guide.

Keep the order with an associative array

items=(b a b c a d)
declare -A seen
unique=()
for x in "${items[@]}"; do
  if [[ -z ${seen[$x]+x} ]]; then
    seen[$x]=1
    unique+=("$x")
  fi
done
echo "${unique[@]}"

b a c d

${seen[$x]+x} expands to x only if the key exists, so the test is true the first time a value shows up. Testing with +x rather than the value matters because an element can be an empty string. Key checks on dictionaries are covered in checking a key in an associative array.

Sort and drop duplicates in one step

items=(b a b c a d)
mapfile -t unique < <(printf '%s\n' "${items[@]}" | sort -u)
echo "${unique[@]}"

a b c d

This is the shortest answer and the right one when order does not matter. The result is sorted, so first-seen order is lost. The sort keys and locale rules are described in sorting a Bash array.

Keep the order with awk

items=(b a b c a d)
mapfile -t unique < <(printf '%s\n' "${items[@]}" | awk '!seen[$0]++')
echo "${unique[@]}"

The awk idiom !seen[$0]++ is true only the first time a line appears, because the post-increment returns the old value, which is zero for a new key. It runs on any system with awk, including the Bash 3.2 era, since the associative array lives in awk rather than in the shell. The gawk manual on arrays explains the subscript behaviour.

Wrap it in a function

uniq_keep_order() {
  local -n _in=$1 _out=$2
  local -A _seen=()
  local x
  _out=()
  for x in "${_in[@]}"; do
    [[ -n ${_seen[$x]+x} ]] || { _seen[$x]=1; _out+=("$x"); }
  done
}
uniq_keep_order items clean

local -n creates a name reference, available from Bash 4.3, so the function can fill an array by name without printing and re-reading it. Name references are the subject of Bash namerefs.

Choosing a method by what you need

The methods differ in four ways: whether they preserve order, whether they handle awkward elements, how fast they are, and how portable they are. The associative array loop preserves order and handles any element that is a valid key, but it needs Bash 4 and a loop in the shell. The sort -u pipeline is one line and handles any text but loses order and breaks on newlines. The awk idiom preserves order and is fast, and it also breaks on newlines unless NUL is used. For a list of a few dozen items, any of them is instant. For hundreds of thousands, the pipelines are much faster than a Bash loop because the work happens in compiled code, and the Bash loop is better only when the data is already in an array that other code uses.

Case is a decision too. Two names that differ only in case are different keys in an array and the same line to sort -f -u. If the data is host names or e-mail domains, convert to lower case first with ${x,,}, a Bash 4 expansion, so that duplicates are detected as intended. For file paths on a case-insensitive filesystem, normalise with the filesystem's rules rather than guessing.

Removing duplicates from a PATH-like variable

The most common real use is cleaning a colon-separated search path that grows each time a profile is sourced. Split on the colon into an array, deduplicate with the ordered method, and join again. The join uses a subshell to set IFS for the expansion: (IFS=:; echo "${unique[*]}"). The star form joins the elements with the first character of IFS, and doing it in a subshell keeps the change local. Assign the result with PATH=$(...) and skip empty elements, which mean the current directory and are a security risk in a search path. This tidy-up runs on every login, so keep it fast and free of external commands.

Elements that contain special characters

The pipelines that use newline as the separator break when an element contains a newline, and the associative array approach has a subtler problem with the empty string, which is a valid element but, in older Bash versions, could not be used as a key. Test the function with an empty string, a string with spaces, one with a glob character like an asterisk, and one with a leading dash. If the data comes from file names, prefer NUL-separated processing end to end: printf '%s\0' "${items[@]}" | sort -zu | mapfile -d '' -t unique is wrong because the last stage runs in a subshell in a pipe, so use process substitution to feed mapfile instead. That detail catches many people, and the test with awkward elements is what reveals it.

Deduplicating by a key instead of the whole element

Real data is often records, such as lines of the form host port in which only the host should be unique. Extract the key and test that, while keeping the entire element: key=${rec%% *}, then check seen[$key]. The awk form is awk '!seen[$1]++', which keys on the first field and keeps the first complete line. Decide which duplicate wins, the first or the last, and say so in a comment. Keeping the last needs a reverse pass or a second loop, as in tac | awk '!seen[$1]++' | tac, and the choice matters for configuration where later entries override earlier ones.

Verifying the result

A deduplication step should be checked by invariant: the output count should not exceed the input count, each output element should appear in the input, and no element should appear twice. A small test with the awkward elements named above, run whenever the function changes, catches regressions. Printing the number of removed duplicates in a debug message helps confirm that the data was what you thought.

Where this does not apply

If the array holds newlines inside elements, the printf '%s\n' pipelines split them; use printf '%s\0', awk 'BEGIN{RS="\0"}' and mapfile -d '' together, or stay with the pure-Bash loop. Case-insensitive uniqueness needs the key lowercased first, for example seen[${x,,}]. On macOS, the stock Bash 3.2 rejects declare -A with "invalid option", so use the awk form or install a newer Bash. For a few dozen elements any of these is instant; the pure-Bash loop is the slowest on huge arrays because each iteration runs in the interpreter.

For a quick check of what a deduplication produced, print the array with declare -p unique and compare the count with ${#unique[@]}. The POSIX specification for uniq is a useful contrast: it removes only adjacent duplicates, which is why sorted input is the usual precondition and why the associative array approach above needs no sorting at all.