How to Use awk With Multiple Files and NR FNR Step by Step

Looking up values from one file in another is something people do with a loop of grep calls, slowly. Awk does it in one pass by remembering the first file in memory and checking the second against it. The only trick is knowing which file you are in, and awk gives you two counters for it. This continues associative arrays. Examples use GNU awk 5.2.

Two counters

awk '{ print FILENAME, NR, FNR }' a.txt b.txt

a.txt 1 1
a.txt 2 2
b.txt 3 1
b.txt 4 2

While reading the first file, the two counters are equal. When awk opens the second, FNR goes back to 1 and NR keeps counting, so the counters differ. The condition NR == FNR therefore selects the first file only.

Annotate lines from a lookup table

web1 10.0.0.11
web2 10.0.0.12
db1 10.0.0.21

web1 up
db1 down
cache1 up

awk 'NR == FNR { ip[$1] = $2; next } { print $1, $2, ($1 in ip ? ip[$1] : "unknown") }' hosts.txt status.txt

web1 up 10.0.0.11
db1 down 10.0.0.21
cache1 up unknown

The first rule fills ip from hosts.txt and moves on with next, so the second rule runs only for status.txt. The membership test with in avoids creating empty entries, which would later look like matches. The argument order matters: the lookup file comes first.

Filter by a list

awk 'NR == FNR { keep[$1]; next } $1 in keep' wanted.txt data.txt
awk 'NR == FNR { drop[$1]; next } !($1 in drop)' blocked.txt data.txt

The first prints lines of data.txt whose first field is listed in wanted.txt, like a keyed grep -f that matches only that field, and not substrings anywhere in the line, which grep -f would. The second drops them. With millions of keys the array uses memory, but the speed is far above that of repeated greps.

Lines only in the first file

awk 'NR == FNR { b[$0]; next } !($0 in b)' second.txt first.txt

This is the set difference of whole lines, without requiring sorted input, in the way comm would demand it. Order is preserved from the file read second.

The empty first file

awk 'FILENAME == ARGV[1] { ip[$1] = $2; next } { print $1, ($1 in ip ? ip[$1] : "unknown") }' hosts.txt status.txt

If hosts.txt is empty, no line is ever read from it, so for the first line of status.txt both counters are 1 and NR == FNR is true; the second file then gets loaded as the lookup table. Comparing FILENAME against ARGV[1] avoids this. The same check fails if both arguments name the same file, and in GNU awk the BEGINFILE and ENDFILE hooks give an alternative. Check that the lookup file is not empty in the calling shell script before relying on the shorter form.

Where this does not apply

If the lookup file does not fit in memory, sort both and use join or load into SQLite. Whitespace-separated defaults assume no spaces in keys; set -F, or the right separator for both files. For a full join with several output columns, see joining files on a key. For fields, awk fields covers the basics.

Why the first-file idiom works and when it breaks

Awk keeps two counters. NR counts all records read so far, across all files, and FNR counts records within the current file and resets to 1 when a new file starts. While the first file is read, they are equal. After the second file starts, FNR restarts and NR does not, so they differ. The condition NR==FNR is therefore true only while reading the first file, and a rule guarded by it can load a lookup table, followed by next, so later rules apply only to the second file. The weakness is the empty first file. If it has no lines, NR and FNR are equal throughout the second file, because no records were read before it, and the lookup rule runs on the data file, which is the opposite of what you meant. A script that handles optional or generated lookup files should not depend on this.

Avoid the failure by testing which file you are in, with FILENAME compared to the first argument, or with a counter that rises when FNR==1, as in FNR==1 { fileno++ } and rules guarded by fileno==1. GNU awk offers ARGIND for the same purpose.

Keys, types and whitespace in lookups

Lookups compare strings, and data from two sources is rarely identical. A key with trailing spaces, a different case or Windows line endings fails to match, and the failure is silent, since a missing key just returns an empty string. Normalize both sides the same way, with tolower, gsub(/^[ \t]+|[ \t]+$/, "") and a carriage return strip. Numbers also bite: an array index is a string, and awk converts a number to a string using CONVFMT, so 1.0 and 1 can end up as different keys if one comes from a computation and the other from text. Read both as text from the fields, and avoid computing keys. When an unmatched line matters, print it to standard error with a count in END, so a quiet loss of data does not go unnoticed.

Use a different field separator per file when the formats differ. Setting FS inside a rule on FNR==1 affects the next line only, and passing -F applies to every file, so a mixed case is easier to handle by splitting each line with the split function and a named separator.

Size and alternatives

The lookup file is held in memory, as an array, for the whole run, so it must fit. A table of a few million small entries is fine on a modern machine, while a table the size of the data is not. Put the smaller file first, since only the first is stored. For two large files, sort both on the key and use join, which streams, or load into SQLite and join in SQL. If you find yourself writing nested loops over arrays to match records, stop: that is a sign the work belongs to a database, which has indexes. Keep the awk version for the cases where a table of keys and a stream of lines is the whole problem, and it will stay short, fast and easy to read.

A short recipe to compare two lists

To print the lines of the second file whose first field is missing from the first, load the keys, then print the non-matching lines. To print those present in both, reverse the test. To print the lines that appear only in the first file, you need a second pass or a flag array: read the second file into the array, then loop over the first file's stored lines in END. When the lists are small and sorted, comm is shorter. When they are large and unsorted, awk avoids the sort. Check the answer on a three-line example before trusting it on thousands, and report the sizes of both inputs and the result, so a result of zero lines is obviously either right or a sign that the key did not match.

See the gawk built-in variables and POSIX awk variables.