Download a List of URLs from a File with curl and xargs

A file holds one URL per line and you want each saved under its remote name.

printf '%s\n' http://localhost:8000/a.txt http://localhost:8000/missing http://localhost:8000/b.txt > urls.txt
xargs -n 1 curl -sS -f -O < urls.txt
echo "xargs exit: $?"
ls *.txt

curl: (22) The requested URL returned error: 404
xargs exit: 123
a.txt
b.txt
urls.txt

xargs -n 1 runs curl -O once per URL, and -O names the local file after the last part of the URL. Without -f, curl would save the body of an error page as missing. With -f it reports The requested URL returned error: 404 and exits 22, while the other two files were fetched. xargs then exits 123, which means at least one invocation failed. See --remote-name and --fail in the manual.

Each call opens a new connection, so a long list is slow. Passing many URLs to one curl with --remote-name-all reuses connections, and -Z fetches them in parallel. Two URLs with the same final name overwrite each other, and a URL that ends in a slash has no name for -O.