How to Use Retries and Rate Limits With curl in Scripts

APIs fail briefly and often: a 503 during a deploy, a timeout under load, a 429 when the script is too fast. A script that retries sensibly survives these without hammering the service. curl has built-in retries and some gaps; the loop below fills them. The details are for curl 8.9 and Bash 5.2, and the basics of calling APIs are in curl in scripts.

Built-in retries

curl -fsS --retry 5 --retry-delay 2 --retry-max-time 60 \
  --connect-timeout 5 --max-time 30 \
  -o data.json https://api.example.com/v1/data

With --retry, curl retries on a timeout, and on HTTP 408, 429, 500, 502, 503 and 504, waiting --retry-delay seconds each time; without that option the delay doubles from one second up to ten minutes. --retry-max-time caps the total retry period. Connection refused is not retried unless you add --retry-connrefused, which suits a service that is still starting. --retry-all-errors also retries other HTTP errors such as 404, which is rarely what you want.

Honour Retry-After

fetch() {
  local url=$1 out=$2 hdr status wait n=0
  hdr=$(mktemp)
  while (( n < 5 )); do
    status=$(curl -sS --max-time 30 -D "$hdr" -o "$out" -w '%{http_code}' "$url") || status=000
    case $status in
      2??) rm -f "$hdr"; return 0 ;;
      429|503)
        wait=$(awk 'tolower($1)=="retry-after:"{gsub("\r",""); print $2}' "$hdr")
        [[ $wait =~ ^[0-9]+$ ]] || wait=$(( 2 ** n ))
        ;;
      000|5??) wait=$(( 2 ** n )) ;;
      *) echo "HTTP $status" >&2; rm -f "$hdr"; return 1 ;;
    esac
    sleep $(( wait + RANDOM % 3 ))
    n=$(( n + 1 ))
  done
  rm -f "$hdr"
  return 1
}

-D dumps the response headers to a file, so the loop can read Retry-After. The header may be a number of seconds or an HTTP date; this version handles seconds and falls back to exponential backoff otherwise. RANDOM % 3 adds jitter, so many clients that failed together do not all come back at the same second. Status 000 means curl itself failed. The header is defined in RFC 9110.

Slow down before the limit

while read -r id; do
  curl -fsS "https://api.example.com/items/$id" >> items.ndjson
  sleep 0.5
done < ids.txt

Two requests per second is under many published limits, and it is cheaper than retrying after a 429. --limit-rate 500k caps bandwidth for large downloads, which is separate from request rate. For parallel jobs use xargs -P with a small number and the same sleep, as discussed in xargs.

Retry only what is safe

A GET is safe to repeat, and so are PUT and DELETE when the API is idempotent. A POST that creates an order may succeed on the server and still time out on the client, so a blind retry creates a duplicate. Send an Idempotency-Key header if the API supports it, or check state before repeating, as covered in locking and idempotency in automation.

Backoff, jitter and why they matter

Retrying at fixed intervals makes many clients retry at the same moment, which turns a short outage into a repeated spike. Exponential backoff doubles the wait after each failure, and jitter adds randomness so that clients spread out. A common formula waits a random time between zero and the capped exponential value, known as full jitter. In a shell loop, sleep $(( RANDOM % (2 ** n) + 1 )) approximates it, with n the attempt number, and a cap of 30 to 60 seconds keeps waits reasonable. curl's own retry doubles the delay without jitter, which is acceptable for a single script and weaker for a fleet of jobs that all start from cron at midnight. Offsetting scheduled jobs by a random delay at the start, with sleep $((RANDOM % 300)), spreads that load as well.

Limits on total time are as important as limits on attempts. A job that retries five times with exponential waits can spend several minutes in sleep, longer than the schedule allows. Compute the budget up front and stop when it is spent, so the next scheduled run does not overlap with a stalled one.

Reading rate limit headers

Many APIs report their limits in headers on every response, commonly X-RateLimit-Remaining and X-RateLimit-Reset, or the standardised RateLimit fields in newer designs. Reading them lets a script slow down before it hits the wall, which is better than recovering after a 429. The names and units differ between providers, some report a reset time in epoch seconds and others a number of seconds from now, so read the documentation and handle the unit explicitly. Do not assume headers exist on error responses from a gateway in front of the API.

Idempotency keys

For requests that change state, retries are only safe if the server can recognise a repeat. Many payment and ordering APIs accept an Idempotency-Key header: the client generates a unique value per logical operation, sends it on every attempt, and the server returns the stored result for a key it has seen. In a script, create the key once before the loop, with uuidgen, and reuse it for each retry. Without server support, design the operation so a repeat is harmless, for example by using PUT with a known identifier in place of POST to create.

What to log about retries

A retry that succeeds on the third attempt is a warning sign, and nobody sees it if nothing is logged. Write one line per failed attempt with the status, the attempt number and the wait, and a final line with the outcome. A monitoring rule that counts retries over time catches a degrading dependency days before it fails outright. Keep response bodies out of the log unless they are small and free of personal data, since error pages sometimes echo the request. Finally, set an identifying user agent on automated clients; providers are more willing to raise limits for a script they can identify and contact than for anonymous traffic.

Testing the retry logic

Retry code runs only when something fails, so it breaks unnoticed. Test it against a local server that you control: a few lines of Python with http.server can return 503 twice and then 200, or send a 429 with a Retry-After header. Run the script against it and check the number of attempts, the waits and the final status. Also test the exhaustion case, where every attempt fails, and confirm that the script exits non-zero with a clear message instead of carrying on with a missing file. Those two tests, a few minutes of work, catch the usual errors: a loop that never ends, a wait that grows without a cap, and a success path that forgets to clean up the header file.

Where retries belong in the architecture

A script that retries inside every call hides trouble and multiplies load. Decide on one layer that owns retrying, normally the thin client function that wraps curl, and keep callers free of loops. Give that layer a clear contract: it returns success, or a final failure after the budget is spent, with the last status code. Callers then decide whether a failure stops the job or skips one item. Put the budget in configuration so operators can lower it during an incident, when extra retries would only add pressure to a struggling service.

Where this does not apply

Retrying cannot fix a permanent error such as a 401, a 403 for a wrong token or a malformed request, and retrying those wastes time and can trigger lockouts. When a provider documents specific limits, use their headers such as X-RateLimit-Remaining instead of guessing. For long jobs, track progress so a final failure can resume, as in resumable scripts. The retry options are in the curl manual.