How to Upload and Download Large Files Through APIs

A small JSON request works with any method, and a 5 GB archive does not. The failure modes are new ones: the transfer takes longer than a timeout, a proxy rejects the body size, the connection drops at 90 percent, memory fills because the tool read the file into a variable. Each has a curl option or a design choice that solves it. This post lists them. It builds on calling REST APIs from the shell and rate limits and retries. Examples use curl 8.11.

Stream the upload

curl -T backup.tar.gz -H "Authorization: Bearer $TOKEN" https://files.example.org/upload/backup.tar.gz
curl --data-binary @backup.tar.gz -H 'Content-Type: application/gzip' https://api.example.org/v1/backups
curl -F '[email protected];type=application/gzip' -F 'name=nightly' https://api.example.org/v1/files

-T sends the file as the body of a PUT; --data-binary @file as the body of a POST, exactly as it is on disk; -F builds a multipart form with the file as a part, which is what a browser upload form does. All three read from disk as they send, so memory use stays flat. The mistake to avoid is -d @file (--data): it removes carriage returns and newlines, silently corrupting any file with them, and it reads the whole file into memory. Show progress with --progress-bar or -#, and add --limit-rate 10M if you must not saturate a link.

Timeouts and retries

curl --fail --connect-timeout 10 --retry 5 --retry-delay 3 --retry-all-errors \
     -T big.iso https://files.example.org/upload/big.iso

Do not set --max-time for a large transfer: it limits the total duration, and kills a healthy upload that happens to be slow. Use --connect-timeout for the connection and --speed-limit with --speed-time to abort only when the speed stays too low: --speed-limit 1000 --speed-time 60 gives up if the rate stays below 1 KB/s for a minute. --retry repeats on transient errors; --retry-all-errors (curl 7.71 and later) includes connection resets, but a retried upload restarts from the beginning unless the API supports resumption.

Resume a download

curl -L -C - -O --retry 5 https://downloads.example.org/big.iso
wget -c https://downloads.example.org/big.iso
curl -r 0-1048575 -o part1 https://downloads.example.org/big.iso

-C - tells curl to look at the size of the partial file and request the remainder with a Range header. It works when the server answers with 206 Partial Content; check with curl -sI URL | grep -i accept-ranges. -r fetches a byte range, useful for downloading a large file in pieces in parallel. Resuming a file that changed on the server produces a corrupt result, which is why you verify.

Cloud storage: pre-signed URLs and multipart

url=$(aws s3 presign s3://my-bucket/backups/db.dump --expires-in 3600 --http-method PUT 2>/dev/null || true)
curl -T db.dump "$url"
aws s3 cp db.dump s3://my-bucket/backups/db.dump

Object stores limit a single upload (5 GB on S3) and handle large objects as multipart uploads, where the client sends numbered parts and the server assembles them; aws s3 cp and similar tools do this automatically and can retry parts. A pre-signed URL lets a client without credentials upload one object for a limited time, which keeps long-lived keys out of scripts; the exact presign command differs by tool and version, so check the documentation. The part size and concurrency settings affect speed.

When the server says no

HTTP/1.1 413 Request Entity Too Large
HTTP/1.1 504 Gateway Timeout
HTTP/1.1 100 Continue

A 413 is a limit in a proxy or the application (for nginx, client_max_body_size); the fix is on the server, or switching to chunked or multipart uploads below the limit. A 504 means a proxy gave up waiting while the backend processed the upload: raise its timeouts, or accept the file and process asynchronously. curl sends Expect: 100-continue for large bodies and waits briefly for the server's permission, which some servers mishandle: -H 'Expect:' disables it.

Verify

sha256sum big.iso
curl -s https://downloads.example.org/big.iso.sha256
curl -s -H "Authorization: Bearer $TOKEN" "https://api.example.org/v1/files/42" | jq -r .sha256

A zero exit status means that the protocol finished, not that the bytes are right. Compare the checksum with the one the server publishes or returns, and for downloads from a source you do not control, a signature as well, as in verifying downloads.

Where this does not apply

For very large datasets, a purpose-built tool (rclone, rsync, aws s3 sync) handles retries, parallelism and verification better than a loop of curl commands. Mobile or unreliable networks benefit from resumable upload protocols such as tus. Never put tokens in the URL of a long transfer, as logs record them.

Why large transfers fail and what to look at first

A large transfer fails for reasons that have nothing to do with your code, and each reason points to a different fix. A timeout in the client, set too short for the size, aborts a transfer that was proceeding. A proxy or load balancer in the path has its own limits on request body size, idle time and total duration, and returns an error such as 413, 408 or 504 when they trip, often from a layer you did not know existed. The connection may drop in the middle, as mobile and long-distance links do. The server may run out of memory or disk when it buffers the whole body. Before changing anything, find which layer failed: read the status code and the headers of the failure, run the transfer with curl -v to see where it stopped, and test from another network. A 413 means the body is too big for some limit, and no retry will change that, while a reset mid-transfer is a candidate for resuming.

Measure throughput first. If the link carries two megabytes a second, a four gigabyte file takes over half an hour, and a timeout designed for a thirty second request will always fail for it.

Streaming, timeouts that fit and retries that help

Stream uploads from disk rather than loading them into memory. curl -T file URL sends a file with PUT and streams it, and --data-binary @file sends it as a POST body, which for very large files is less convenient. Distinguish the timeouts. --connect-timeout limits only the connection setup and is safe to keep short. --max-time limits the entire operation and is the wrong tool for a big file, since it ends a healthy transfer. A better guard against a stalled transfer is --speed-limit with --speed-time, which aborts if the rate stays below a floor for some seconds. For retries, --retry handles transient failures, --retry-delay and --retry-max-time bound them, and --retry-all-errors, available from curl 7.71, also retries on errors that are not classed as transient. Retrying an upload makes sense only if the server treats a repeated upload safely: the operation should be idempotent, so the second attempt replaces the first and does not create a duplicate.

Resuming downloads and cloud storage

A download can resume where it stopped if the server supports range requests, shown by an Accept-Ranges: bytes header. curl -C - -O URL checks the size of the partial file and asks the server for the rest, and a loop that repeats until the command succeeds finishes a transfer over an unreliable link. Confirm that the file has not changed on the server between attempts, using the ETag or Last-Modified value, since resuming a file that was replaced produces a corrupt result that looks complete. For uploads, cloud object stores avoid the problem with two mechanisms. A pre-signed URL lets a client upload directly to storage with a time-limited signature, so the file never passes through your API server and its limits. Multipart upload splits a large object into parts that are uploaded separately, in parallel and individually retried, then assembled, which turns one fragile transfer into many small ones. Tools such as rclone and aws s3 cp implement this, and are better than hand-built loops for large data.

Verifying the result

A transfer that ends without an error can still deliver wrong data. Compare a checksum of the file you have with one the other side publishes, using sha256sum against the value in the API response or a checksum file, and send a Content-MD5 or a newer checksum header on upload when the service supports it, so the server can reject a corrupted body. Compare sizes too, since a truncated file is the most common failure. Write the download to a temporary name and rename it only after verification, so a later step never sees a half-written file. Log the size, the time and the checksum of each transfer, which gives you a record for debugging and for audits. When the server says no for a reason you cannot change, such as a hard size limit, look for a different route: split the file, use a different endpoint designed for large objects, or ask for a limit increase, and do not keep retrying a request that is refused by design.

See the curl man page and HTTP range requests in RFC 9110.