How to Use Go Concurrency to Speed Up File Processing

Go makes parallel work cheap to write, which makes it easy to write too much of it. A file-processing tool needs a walk, a limit on how many files are open at once, and a safe way to collect results. This post builds a checksum tool in that shape and discusses when it pays off. It continues walking directories in Go and addresses the performance angle of replacing shell scripts with Go. Targets Go 1.24 on Linux.

Walk, then fan out with errgroup

package main

import (
	"crypto/sha256"
	"fmt"
	"io"
	"io/fs"
	"os"
	"path/filepath"
	"runtime"

	"golang.org/x/sync/errgroup"
)

func hashFile(path string) (string, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", err
	}
	defer f.Close()
	h := sha256.New()
	if _, err := io.Copy(h, f); err != nil {
		return "", err
	}
	return fmt.Sprintf("%x", h.Sum(nil)), nil
}

func main() {
	root := os.Args[1]
	var g errgroup.Group
	g.SetLimit(runtime.NumCPU())
	results := make(chan string, 64)

	go func() {
		defer close(results)
		_ = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
			if err != nil || d.IsDir() {
				return err
			}
			g.Go(func() error {
				sum, err := hashFile(p)
				if err != nil {
					return err
				}
				results <- sum + "  " + p
				return nil
			})
			return nil
		})
		_ = g.Wait()
	}()

	for line := range results {
		fmt.Println(line)
	}
}

SetLimit makes g.Go block when all slots are busy, so the walker naturally slows down instead of queueing a million goroutines. Since Go 1.22 each loop iteration has its own variables, so the closure captures p safely. The walker runs in its own goroutine because g.Go can block, and the main goroutine drains the channel. The order of output is not the walk order, so sort afterward if you need stable output.

The classic worker pool

paths := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 8; i++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		for p := range paths {
			// process p
		}
	}()
}
// send paths, then close(paths) and wg.Wait()

This is the same idea with explicit goroutines. Use it when you need per-worker state, such as a reusable buffer, or custom retry behavior. Close the channel from the sender only, and wait for the group before reading results from an unbuffered channel to avoid deadlock.

Does it help?

It depends on the bottleneck. Hashing is CPU-bound once the data is in memory, so eight cores give close to eight times the throughput. Reading cold files from a spinning disk is limited by the seek rate, and more concurrent readers can make it slower. Measure both cases: run the tool twice, drop the cache with sync; echo 3 | sudo tee /proc/sys/vm/drop_caches for the cold run, and time with hyperfine or time. The caveats about warm caches are in why caches skew benchmarks.

Where this does not apply

For a one-off task over a few hundred files, a shell pipeline with find -print0 | xargs -0 -P8 sha256sum is quicker to write and about as fast. Concurrency adds ordering, error and memory questions; if the single-threaded version finishes in a second, keep it. Shared maps need a mutex or sync.Map; the race detector (go run -race) will tell you when you forgot.

What limits a file-processing job

Before adding goroutines, ask what the job waits on. Reading many small files is usually limited by the storage and by system call overhead, hashing or compressing is limited by the CPU, and parsing text is somewhere in between. On a laptop with an NVMe drive and a warm page cache, reading is fast and the work is CPU-bound, so a pool sized to the number of cores gives a large speedup. On a spinning disk or a network file system, many concurrent readers can make things worse, since the head moves between files or the server queues requests, and a small pool of two or four workers often beats a large one. The worker count is a parameter to measure, not a constant to guess, and it should be adjustable with a flag so you can test it on the machine that matters.

Go helps by making goroutines cheap, and cheap does not mean free. A goroutine per file for a million files allocates a million stacks and a million pending operations, and can exhaust file descriptors, because each open file counts against the process limit shown by ulimit -n. Bound the concurrency with a fixed pool or with errgroup.SetLimit, and close files as soon as you finish with them, using defer inside the function that handles one file and not in the loop that starts them.

Errors, cancellation and ordering

A pool needs a policy for failures. With errgroup.WithContext, the first error cancels the context, and workers that check it stop early, while the group's Wait returns that first error. That fits all-or-nothing jobs such as building an index. For jobs where one unreadable file should not stop the rest, collect errors in a slice protected by a mutex or sent over a channel, and report them at the end with the file names. Choose before you write the code, since the two behaviors lead to different structures. Respect cancellation in the walk too: filepath.WalkDir can return filepath.SkipAll or an error to stop, and the producer should select on the context so it does not block on a full channel after the workers have quit.

Results arrive in whatever order workers finish. If the output must follow the input order, attach an index to each job and write results into a slice at that index, or sort at the end. Printing from many goroutines also interleaves lines, so send results to one goroutine that owns standard output.

Measuring honestly

Benchmark with the real data and a cold cache when the real case is cold. The first run reads from disk and later runs read from memory, which flatters any version. Drop the page cache between runs on Linux with the drop_caches file as root, or use a data set larger than memory. Record the time, the file count, the worker count and the hardware, and run each setting several times to see the spread. A change from 4 to 8 workers that moves the total by two percent is noise. Look at CPU use while it runs: if the cores are not busy, more workers will not help, and the limit is the storage or the network. The standard pprof tool shows where the time goes if the CPU is busy. Compare against the simple sequential version and against a shell pipeline with find and xargs -P, since sometimes the shell version is as fast and has no code to maintain.

A structure that stays readable

Keep the pieces separate so each is easy to test. One function walks the tree and sends paths on a channel, one function processes a single file and returns a result or an error, and the pool wires them together. The per-file function takes a path and returns plain data, so a unit test can call it on a temporary directory without any goroutines. The pool code is then a dozen lines that you check once. Run the tests with the race detector, go test -race, which finds unsynchronized writes to shared slices and maps, the most common concurrency mistake in this kind of program. Close the jobs channel from the producer when the walk ends and let the workers finish on their own range loop, so shutdown needs no extra signalling.

See the errgroup package and Go Concurrency Patterns: Pipelines and cancellation.