How to Avoid Repeated API Calls in Scripts With Real Output

An API client script that works on ten items hits a rate limit or runs for hours on ten thousand, because it makes a request per item or calls the same endpoint again and again. The fix is usually structural: fetch once and reuse. It also makes scripts faster and kinder to the service. The caching mechanics are in caching command output, and the rate-limit handling is in retries and rate limits. The examples assume curl 8 and jq 1.7.

The costly pattern

for id in $(curl -s "$API/items" | jq -r '.[].id'); do
  name=$(curl -s "$API/items/$id" | jq -r '.name')
  owner=$(curl -s "$API/items/$id" | jq -r '.owner')
  echo "$id $name $owner"
done

For 500 items this makes 1,001 requests: one for the list and two per item for the same data. Each request costs latency (often 100 ms or more), counts against a quota and may be rate limited. The loop also makes the second fetch of the same item redundant.

Fetch once, query many

items=$(curl -fsS "$API/items?expand=owner")
echo "$items" | jq -r '.[] | [.id, .name, .owner.login] | @tsv'

detail=$(curl -fsS "$API/items/42")
name=$(jq -r '.name' <<< "$detail")
owner=$(jq -r '.owner' <<< "$detail")

Store the response in a variable (or a file for large ones) and run jq over it as often as needed; jq is local and fast. Many APIs offer a parameter to include related data (expand, include, fields) that removes the per-item calls entirely. Read the documentation for bulk endpoints before writing a loop; GraphQL APIs let one request fetch the nested data. Parsing patterns are in jq for API responses.

Conditional requests

etag_file=.cache/items.etag
body_file=.cache/items.json
mkdir -p .cache
status=$(curl -sS -o .cache/new.json -w '%{http_code}' \
  -H "If-None-Match: $(cat "$etag_file" 2>/dev/null || true)" \
  -D .cache/headers.txt "$API/items")
if [[ $status == 304 ]]; then
  echo "unchanged"
elif [[ $status == 200 ]]; then
  mv .cache/new.json "$body_file"
  awk 'tolower($1) == "etag:" {print $2}' .cache/headers.txt | tr -d '\r' > "$etag_file"
fi

unchanged

The server returns an ETag header identifying the version of the resource. On the next request, sending it in If-None-Match lets the server answer 304 Not Modified with no body, which is cheap and, on GitHub's API for example, does not count against the rate limit. If-Modified-Since works the same way with a date. Not every API supports them, so test with curl -I. The output shown is the script's message on a cache hit.

Batch and paginate sensibly

curl -fsS "$API/items?ids=1,2,3,4,5"
curl -fsS "$API/items?per_page=100&page=1"

If the API has a batch endpoint, use it: one request with 100 IDs beats 100 requests. For lists, request the maximum page size the API allows to cut the number of pages. Follow the pagination links the API returns rather than guessing page numbers; see paginating an API in a shell script. Run independent requests in parallel only after reducing their number and within the documented limits.

Find hidden calls

curl() { echo "curl $*" >> /tmp/api-calls.log; command curl "$@"; }
export -f curl
./sync.sh
sort /tmp/api-calls.log | uniq -c | sort -rn | head

    501 curl -s https://api.example.test/items/42
      1 curl -s https://api.example.test/items

Wrap curl in a function that logs each call, run the script and count duplicates. Repeated identical requests are the first thing to remove. Libraries and SDKs often make requests you did not write, so use a proxy or --trace if the count looks higher than the code. The sample counts above are illustrative.

How to tell that a script is too chatty

Before changing anything, count. Total requests, duplicate requests and requests per item are numbers that expose the problem faster than reading code. A script that handles 500 items and makes 1,001 calls has a ratio of two per item, and any ratio above the number of distinct resources suggests that some responses are fetched more than once. Write the count down, change the script, count again, and report the difference; it is the one benchmark that does not depend on network conditions, because request counts are deterministic where latency is not.

Latency multiplies the effect. At 120 ms per round trip, a thousand sequential calls take two minutes of pure waiting, before the server does any work. Connection reuse matters here: starting a new curl process per request pays for a TCP and TLS handshake each time, which can cost more than the request itself. A single curl invocation given several URLs reuses its connection, and a language client with a session object does the same, which is one reason a loop rewritten in Python often runs several times faster with no change in the number of requests.

Respecting the service

An API is a shared resource, and its limits protect other users. Read the documented quota and design under it, not near it: a script that uses ninety-five percent of the hourly allowance leaves no room for a second run after a failure. Send an identifying User-Agent with a contact address if the provider requests one, and honour Retry-After and rate-limit headers when they appear. Providers notice abusive clients and may revoke keys, so behaving well is also self-interest.

When you must make many calls, spread them out. A short pause between requests, or a small fixed concurrency of two to four, stays well within most limits and finishes faster than a serial loop. A burst of fifty parallel requests will trigger throttling and then take longer than the polite version after retries and back-off. Add jitter to the delays so several copies of the script do not align and hit the service together.

Keeping reused data correct

Reuse introduces a staleness question. Data fetched at the start of a long run may change while the script works through it, and later steps then act on an old view. For a report, this does not matter. For an operation such as deleting resources that look unused, it does: re-read the item immediately before the destructive call, or use an If-Match header with the ETag you saw, so the server rejects the change if the item has been modified in between. That turns a race into a clean failure you can handle.

Scope the reuse. Cache for the length of one run in memory or a temporary file that is deleted at the end, and keep any cache that lasts between runs keyed by everything that changes the answer, including the account and the base URL. A cache shared between two environments is a classic way to read production data in a staging report. Document the expiry next to the code.

Designing the script for the next change

Separate fetching from processing. One function or one script downloads data and writes it to a file, and a second reads the file and produces the report. The split makes development fast, since the processing can be rerun a hundred times against saved data without a single request, and it makes testing possible with a small fixture instead of a live service. It also gives a natural place for a cache: if the file is recent enough, skip the fetch. Most of the speed and politeness improvements fall out of that one structural decision.

Where this does not apply

Data that must be current (a payment status, a lock, a one-time token) must not be cached or reused. Check the API's terms: some forbid storing responses beyond a certain time. For idempotent writes, retry carefully, since a repeated POST can duplicate work. See the MDN guide to conditional requests and the GitHub REST API best practices.