How to Download Many Files in Parallel From a List

Fetching a thousand files one at a time wastes the network: each transfer spends much of its time on connection setup and waiting for the first byte. Running several at once fills the pipe. The examples use curl 8.9, xargs from GNU findutils 4.10 and aria2 1.37; the general pattern of parallel shell work is in running commands in parallel.

xargs and curl

xargs -n1 -P8 curl -fsSLO --retry 3 < urls.txt

-n1 passes one URL to each curl, and -P8 keeps eight running. -O saves each file under its remote name, so two URLs ending in the same file name overwrite each other; for such lists, build the output name yourself. -f prevents an HTTP error page from being saved as the file. The flags are explained in xargs.

Control the file names

while IFS=$'\t' read -r url name; do
  printf '%s\0%s\0' "$url" "$name"
done < list.tsv |
  xargs -0 -n2 -P6 sh -c 'curl -fsSL --retry 3 -o "$2" "$1" || echo "$1" >> failed.txt' _

A tab-separated list of URL and name is converted to NUL-separated pairs, and each worker gets two arguments. The failing URL is appended to failed.txt. Appending one short line from several processes is effectively atomic on local filesystems, but for large logs use a lock or per-worker files. Rerun the job later with just failed.txt as the list.

curl's own parallel mode

sed 's/^/url = /; s/$/\n-O/' urls.txt > curl.cfg
curl --parallel --parallel-max 8 --fail --silent --show-error --remote-name-all -K curl.cfg

Since curl 7.66, one process can run several transfers, reusing connections and HTTP/2 multiplexing to one host, which is more efficient than starting a process per file. The config file lists one url = line per transfer. --remote-name-all makes -O the default for each. This is the fastest approach for many small files from the same host.

aria2 for big files and resume

aria2c -i urls.txt -j 8 -x 4 -c --auto-file-renaming=false --dir downloads

-j is the number of parallel downloads, -x the connections per server, and -c resumes partial files, so an interrupted run continues rather than restarting. aria2 can split one large file across connections, which helps when a server limits per-connection speed. See rclone for downloads from cloud buckets.

Why parallel downloads help, and when they stop helping

A single transfer is limited by latency, by the slow start of TCP, and by whatever per-connection cap the server applies. For many small files, latency dominates: each file pays a DNS lookup, a connection setup, a TLS handshake and a round trip before data flows, so the pipe sits idle while the client waits. Running eight transfers at once hides most of that wait. For a few large files the picture reverses. One connection can fill a fast link, and extra connections gain little unless the server throttles each stream, which is the case aria2's multi-connection mode is built for.

There is a ceiling, and finding it is worth a quick experiment. Time the same list with 1, 4, 8 and 16 workers and plot total seconds. The curve flattens where the bandwidth, the server's rate limit or the local disk becomes the bottleneck. Past that point more workers raise the error rate, trigger 429 responses and can get an address blocked. Choose the smallest count that reaches the plateau.

Connection reuse and HTTP/2

Starting a new curl process per file prevents connection reuse, so every file pays for a full handshake. When the URLs point at the same host, curl's own parallel mode keeps connections open and can multiplex many requests over one HTTP/2 connection, which is both faster and kinder to the server than eight separate sockets. Passing several URLs to a single curl command also reuses the connection sequentially, and batching a hundred URLs per curl process with xargs -n100 is a good compromise that works on any version. Check what happened with -w '%{num_connects}\n', which reports how many new connections a transfer needed.

Failure handling and idempotence

Large batches always have failures: timeouts, transient 503 responses, a file removed on the server. Plan for them. Use --retry for transient errors, record permanent failures in a file as in the example, and make the whole job safe to run again. The simplest way is to skip files that already exist and are complete. Writing to a temporary name such as name.part and renaming after success prevents a half-written file from being mistaken for a finished one. A checksum list, if the publisher offers one, closes the loop: run sha256sum -c over the results and re-fetch only the files that fail.

Rate limits need respect. Many APIs publish limits in headers, and a fixed pause between starts, using xargs with a small wrapper that sleeps, keeps the run within them. If the server answers 429 with a Retry-After header, honour it instead of retrying at once.

Output and observability

Eight interleaved progress bars are unreadable. Use -s with -S to keep the output quiet but report errors, and print one line per finished file from the worker, such as the name and the HTTP status taken from -w. Parallel output from several processes can interleave within a line when it exceeds a pipe buffer, so keep each message short and written in one call. For a long job, a count of completed files against the total, derived from the output directory, gives a progress measure without parsing any logs.

Finally, set a sensible user agent and identify the job if the server is not yours, and avoid hammering a single small host. A polite default of two to four connections per host is a rule that keeps downloads fast and keeps access.

A tested pattern to adapt

For a list of URLs where every file must land under a predictable name, derive the name from the URL, skip files that exist, and download to a part file first. The worker below is small enough to read in one go, and the same body can be run by xargs or by GNU parallel when its job log and retry options are wanted.

fetch() {
  url=$1
  out=downloads/${url##*/}
  [ -s "$out" ] && return 0
  curl -fsSL --retry 3 --connect-timeout 10 -o "$out.part" "$url" \
    && mv "$out.part" "$out" \
    || { rm -f "$out.part"; echo "$url" >> failed.txt; }
}
export -f fetch
mkdir -p downloads
xargs -P6 -I{} bash -c 'fetch "$1"' _ {} < urls.txt

Rerunning the command fetches only what is missing, and failed.txt becomes the input for the next pass. The names collide if two URLs share a final path segment, so use a hash of the full URL as the name when that can happen.

Where this does not apply

More workers is not always faster: once the link or the server is saturated, extra connections only add errors, and many servers limit connections per client, so four to eight is a sensible start. Be considerate with other people's servers, and respect rate limits as in curl retries and rate limits. If the files are on one rsync or SFTP server, a single rsync is simpler. Check the downloads afterwards against checksums when the publisher gives them, as in the checksum guide. See the curl parallel documentation and the aria2c manual.