How to Write a Go Replacement for xargs or grep Step by Step

Rewriting grep in Go is not about beating it, since GNU grep is very fast. It is a compact way to learn how Unix tools behave: input from files or stdin, output to stdout, errors to stderr, and an exit status that scripts depend on. Those habits come from building a command line tool in Go. Code targets Go 1.24.

A grep clone

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
	"regexp"
)

func search(re *regexp.Regexp, name string, r io.Reader, show bool) (bool, error) {
	sc := bufio.NewScanner(r)
	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
	found := false
	for n := 1; sc.Scan(); n++ {
		if re.Match(sc.Bytes()) {
			found = true
			if show {
				fmt.Printf("%s:%d:%s\n", name, n, sc.Text())
			} else {
				fmt.Printf("%d:%s\n", n, sc.Text())
			}
		}
	}
	return found, sc.Err()
}

func main() {
	if len(os.Args) < 2 {
		fmt.Fprintln(os.Stderr, "usage: mygrep PATTERN [FILE...]")
		os.Exit(2)
	}
	re, err := regexp.Compile(os.Args[1])
	if err != nil {
		fmt.Fprintln(os.Stderr, "mygrep:", err)
		os.Exit(2)
	}
	files := os.Args[2:]
	status := 1
	if len(files) == 0 {
		if ok, err := search(re, "(stdin)", os.Stdin, false); err != nil {
			fmt.Fprintln(os.Stderr, "mygrep:", err)
			os.Exit(2)
		} else if ok {
			status = 0
		}
		os.Exit(status)
	}
	for _, name := range files {
		f, err := os.Open(name)
		if err != nil {
			fmt.Fprintln(os.Stderr, "mygrep:", err)
			status = 2
			continue
		}
		ok, err := search(re, name, f, len(files) > 1)
		f.Close()
		if err != nil {
			fmt.Fprintln(os.Stderr, "mygrep:", err)
			status = 2
		} else if ok && status != 2 {
			status = 0
		}
	}
	os.Exit(status)
}

go build -o mygrep mygrep.go
./mygrep 'ERROR' app.log | head -n 3
echo "status=$?"

The Scanner's default maximum line is 64 KB, and a longer line stops the scan with bufio.Scanner: token too long. The code raises the cap to 1 MB with Buffer; a minified JSON file may need more. Go's regexp implements RE2 syntax, which guarantees linear time but lacks backreferences and lookaheads; see regex in Go.

An xargs -0 clone

package main

import (
	"bufio"
	"bytes"
	"os"
	"os/exec"
)

func splitNUL(data []byte, atEOF bool) (int, []byte, error) {
	if i := bytes.IndexByte(data, 0); i >= 0 {
		return i + 1, data[:i], nil
	}
	if atEOF && len(data) > 0 {
		return len(data), data, nil
	}
	return 0, nil, nil
}

func run(base []string, batch []string) error {
	cmd := exec.Command(base[0], append(base[1:], batch...)...)
	cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
	return cmd.Run()
}

func main() {
	sc := bufio.NewScanner(os.Stdin)
	sc.Split(splitNUL)
	var batch []string
	status := 0
	flush := func() {
		if len(batch) == 0 {
			return
		}
		if err := run(os.Args[1:], batch); err != nil {
			status = 123
		}
		batch = batch[:0]
	}
	for sc.Scan() {
		batch = append(batch, sc.Text())
		if len(batch) == 50 {
			flush()
		}
	}
	flush()
	os.Exit(status)
}

find . -name '*.log' -print0 | ./myxargs gzip --

The items go straight into exec.Command as separate arguments, with no shell between, so names with spaces or quotes are safe. Real xargs also limits batches by total length against ARG_MAX, supports -P and handles exit codes in more detail (123 for any failed command is borrowed from it).

Where this does not apply

Use the originals for real work. GNU grep, ripgrep and xargs are tuned in ways a 60-line program is not. These clones are for learning the conventions, for a tool that must ship as a single static binary without dependencies, or for embedding a search in a larger program. For parallelism see concurrency for file processing, and for the shell versions, the xargs command.

How the grep clone reads lines safely

The tempting way to read lines in Go is bufio.Scanner, and it has a limit that surprises people: by default a token, here a line, can be at most 64 KiB, and a longer line stops the scan with bufio.ErrTooLong. Many files never reach that, and minified JavaScript, log files with embedded payloads and generated data do. Call scanner.Buffer with a larger maximum to raise the limit, and always check scanner.Err() after the loop, since a scanner that stopped early on an error looks exactly like one that finished. For input of unknown shape, bufio.Reader.ReadString or ReadLine loops handle any length, at the cost of a little more code. Also decide how binary files behave. Real grep detects NUL bytes and prints "binary file matches", and a clone should at least skip or flag them, or it will print garbage into a terminal.

The regular expression engine in Go's standard library is RE2, which guarantees linear time in the size of the input. That rules out catastrophic backtracking, and also rules out features that need it, such as backreferences and lookahead. A pattern that works in grep -P may not compile in Go, and the error from regexp.Compile says which construct is unsupported.

Why a clone needs a null-separated mode

File names on Unix may contain spaces, quotes and even newlines, and the only byte they cannot contain is NUL. The classic pipeline find ... -print0 | xargs -0 exists for that reason. A clone of xargs -0 reads standard input, splits on the NUL byte instead of whitespace, and runs the command with the names as separate arguments, never through a shell, so no quoting is applied or needed. In Go that means exec.Command(name, args...), which passes each argument exactly as given. Writing the splitter takes a few lines with a custom bufio.SplitFunc that cuts at zero bytes, and it also handles a trailing name with no final NUL.

The reason to batch is the kernel's limit on the total size of arguments, reported by getconf ARG_MAX. Real xargs fills each command line up to a limit and starts another invocation. A clone should cap the number of names per call, say a few hundred, and run the command once per batch. That is also what makes it faster than one process per file.

Exit status, signals and honest behavior

A tool that other scripts call must report its result in the exit status. Follow the convention of grep: 0 when something matched, 1 when nothing matched, and 2 on an error such as an unreadable file or a bad pattern. In Go, os.Exit skips deferred functions, so structure the program as a run() function that returns an error and a status, with main calling os.Exit(run()), and every deferred close inside run has completed first. For the xargs clone, return the failure of the command in a way scripts can tell apart: real xargs exits with 123 when any invocation exited with a status between 1 and 125, and 124 when a command exited with 255. Matching those codes makes the clone usable as a replacement in scripts that check them.

Be clear in the documentation about what the clone leaves out. It does not match every option of the original, and it should refuse unknown flags with a message and not ignore them, so a script that relies on a missing feature fails loudly.

Testing both tools against the originals

A clone earns trust by agreeing with the tool it copies. Write a few tests that run both the original and your version on the same inputs and compare output and exit status. For the grep clone, use a directory with plain text, a file with a very long line, an empty file, a file with Windows line endings, a binary file and a file with no trailing newline. For the xargs clone, feed names containing spaces, quotes, a leading dash and a newline, and make the command a small script that prints its arguments one per line, so any splitting error is visible. Differences are worth reading closely, since each one is either a bug or a documented departure. Keep the tests in the repository and run them in CI, so a later change to the splitter cannot reintroduce a quoting problem unnoticed.

Measure against the original too. GNU grep is highly tuned and will beat a simple clone on a large file by a wide margin, and that is expected. The point of the clone is to learn the mechanics and to add a feature the original lacks, such as a particular output format, and not to compete on speed.

Finally, handle the edges that real use finds: a path that is a directory, a file that disappears between listing and reading, permissions that deny a read, and a pipe closed early by the next command, which should end the program quietly without a stack trace. Ignore the broken pipe error from a closed standard output, since head closing the pipe is normal.

See the bufio.Scanner documentation and the os/exec package.