API Error 429 Too Many Requests and Backoff in Scripts

A script that works on ten items and fails on ten thousand with 429 Too Many Requests is hitting the service's rate limit. The service is protecting itself, and the right response is to slow down, not to hammer harder. A good handler reads what the server says, waits sensibly and finishes. The wider treatment is in rate limits and retries in scripts, and this post is the focused recipe. The examples assume Bash 5.2, curl 8 and jq 1.7.

What the server tells you

curl -sS -D - -o /dev/null https://api.example.test/v1/items | grep -i -E '^(HTTP|retry-after|x-ratelimit|ratelimit)'

HTTP/2 429
retry-after: 30
x-ratelimit-limit: 100
x-ratelimit-remaining: 0
x-ratelimit-reset: 1793700060

The values are illustrative. Retry-After is standard: a number of seconds or an HTTP date. The X-RateLimit-* headers are common but not standardised (an IETF draft defines RateLimit-*), and the reset value may be a timestamp or seconds, so read the API's documentation. Use them to slow down before you hit the limit: when remaining is low, sleep until the reset.

Retry with backoff and jitter

api_get() {
  local url=$1 max=6 attempt=1 code wait
  while :; do
    code=$(curl -sS -D headers.txt -o body.json -w '%{http_code}' -H "Authorization: Bearer $API_TOKEN" "$url")
    case $code in
      2??) return 0 ;;
      429|502|503|504)
        (( attempt >= max )) && { echo "giving up after $attempt tries (HTTP $code)" >&2; return 1; }
        wait=$(awk 'tolower($1) == "retry-after:" {print $2}' headers.txt | tr -d '\r')
        if ! [[ $wait =~ ^[0-9]+$ ]]; then
          wait=$(( 2 ** attempt ))                       # 2, 4, 8, 16, 32
        fi
        wait=$(( wait + RANDOM % 3 ))                    # jitter
        echo "HTTP $code, waiting ${wait}s (attempt $attempt/$max)" >&2
        sleep "$wait"; (( attempt++ )) ;;
      *) echo "HTTP $code" >&2; return 1 ;;
    esac
  done
}

HTTP 429, waiting 31s (attempt 1/6)
HTTP 429, waiting 4s (attempt 2/6)

The function honours Retry-After when it is a number of seconds and falls back to exponential backoff otherwise, adding a little random jitter so that many clients do not retry at the same instant. It stops after six attempts. Only retry requests that are safe to repeat; a 429 normally means the request was not processed, but for a POST confirm with the API's documentation or an idempotency key. The sample lines show the messages the function prints.

Let curl retry

curl -fsS --retry 5 --retry-delay 2 --retry-max-time 120 --retry-all-errors https://api.example.test/v1/items

curl retries on transient errors (timeouts, 408, 429, 500, 502, 503, 504) with an increasing delay and honours Retry-After when it is present. --retry-all-errors (curl 7.71 and later) also retries other failures, which is rarely wise for POST. A script-level function gives more control over logging and limits; curl's flags are the quick option for single calls.

Stay under the limit

limit_per_min=60
interval=$(awk -v n="$limit_per_min" 'BEGIN { printf "%.2f", 60 / n }')
while read -r id; do
  api_get "$API/items/$id" && jq -r '.name' body.json
  sleep "$interval"
done < ids.txt

A fixed delay keeps you at or below the documented rate. Better, reduce the number of calls: fetch lists instead of single items, use batch endpoints, cache results between runs and send conditional requests, as in avoiding repeated API calls. Parallel workers share the limit, so a pool of ten workers needs a shared budget, not ten separate delays.

Respect the service

Persistent 429s mean the script is too aggressive for the plan. Identify yourself with a User-Agent, use an API key so the limit applies to you rather than to a shared IP, request a higher quota if needed, and run bulk jobs off-peak. Ignoring 429s can get an account throttled harder or blocked. Log the number of 429s per run and watch the trend. If the same tool is used by many people, build the backoff in once, as a shared function from a scripts library.

Why rate limits exist and what they protect

A rate limit is a promise the service makes to all its clients: no single one can use so much that the others suffer. It is also the cheapest defence a provider has against runaway scripts, scraping and denial of service. Limits are typically expressed per key or per address over a window, such as 100 requests per minute, and sometimes as several layers: a burst allowance, a sustained rate and a daily total. Reading the documentation to learn which layers apply is the first step, because the right behaviour for a per-minute limit, where waiting a few seconds helps, differs from a daily quota, where waiting a few seconds does not.

A 429 is not an error in your script. It is a message that tells you how fast you may go. A script that treats it as a failure and aborts wastes the work done so far, and one that retries immediately makes the situation worse. The right design treats pacing as a core part of the client and not as an exception path.

Adaptive pacing

A fixed delay is simple and wasteful when the limit is generous, and insufficient when other clients share the quota. A better client reads the remaining budget from the response headers after each call and adjusts. If remaining requests are plentiful, it goes at full speed; as the count approaches zero, it slows so that the reset time arrives just as the budget runs out. The result finishes sooner than a fixed slow rate and never hits the wall. Some APIs report the reset as a timestamp and others as seconds remaining, so parse the one the documentation names, and guard against clock differences by preferring relative values.

Concurrency complicates this. Several workers sharing a key need a shared view of the budget. A simple approach uses a single process that makes the calls and workers that queue their requests to it. Another uses a token bucket in a shared store. For a shell script, one process with a controlled loop is almost always sufficient and far easier to reason about than a pool of competing curl commands.

Making retries safe

Retries after a 429 are safe because the server rejected the request before acting on it, but retries after a timeout or 5xx may repeat work that already happened. Use idempotency keys where the API provides them. Generate one per logical operation and reuse it across attempts, so a repeat is recognised and the earlier result returned. Where there is no such facility, make the operation idempotent by design, for example by using PUT to a known identifier rather than POST to a collection, or by checking for the result before retrying.

Bound everything. A maximum number of attempts, a maximum total time and a cap on the delay between attempts keep a stuck retry loop from running for hours. When the budget is exhausted, exit with a distinct status and a message that says what was completed, so the next run resumes where this one stopped. Persist progress, for example the last processed identifier, in a small file written after each success.

Observing your own behaviour

Log each throttled response with the time waited, and print a summary at the end: calls made, throttled responses and total time spent waiting. If the waiting is a large share of the run, the script is too aggressive for the plan or the batch is too large for one session. Trends across runs show when a vendor tightens limits, or when your data volume has outgrown the plan, long before the nightly job starts missing its window.

Where this does not apply

Some services signal limits with 403 (GitHub's secondary limits, with a message) or 503, so read the body too. Daily quotas reset at a fixed time; waiting does not help before then, and the script should stop and report. Concurrency limits are different from rate limits and need a smaller worker pool. See Retry-After in RFC 9110 and the AWS builders library on retries, backoff and jitter.