How to Cache API Responses in Shell Scripts the Right Way

A script that calls an API every time it runs wastes quota, time and the server's patience. A cron job that fetches the same list every minute, or a prompt function that asks a weather service on every keypress, is the same mistake. Caching the response locally is easy to do and easy to get wrong: stale answers, half-written files, secrets left in a shared directory. This post builds a cache in layers, from a time-based file to proper revalidation. It extends rate limits and retries and caching command output. Examples use curl 8.11, jq 1.7 and Bash 5.2.

A time-based file cache

#!/usr/bin/env bash
# usage: cached-get TTL_MINUTES URL   prints the (possibly cached) response body
set -euo pipefail
ttl=$1; url=$2
dir="${XDG_CACHE_HOME:-$HOME/.cache}/api-cache"
install -d -m 700 "$dir"
key=$(printf '%s' "$url" | sha256sum | cut -d' ' -f1)
file="$dir/$key"

if [[ -s $file && -z $(find "$file" -mmin +"$ttl" -print) ]]; then
  cat "$file"; exit 0
fi

tmp=$(mktemp "$dir/.tmp.XXXXXX")
trap 'rm -f "$tmp"' EXIT
if curl -fsS --max-time 20 "$url" -o "$tmp"; then
  mv "$tmp" "$file"; trap - EXIT
  cat "$file"
elif [[ -s $file ]]; then
  echo "api unavailable, serving stale cache" >&2
  cat "$file"
else
  exit 1
fi

cached-get 10 https://api.example.org/v1/rates | jq '.eur'

The key is a hash of the full URL, so different parameters get different files. find -mmin +N prints the file only if it is older than the TTL, so an empty result means fresh. The download goes to a temporary file in the same directory and is moved into place only on success, so a failed or interrupted request never leaves a corrupt entry, because mv within a filesystem is atomic. If the API is down, the script serves the old copy and says so on stderr, which keeps dashboards and prompts working through an outage. The cache directory has mode 700 because responses may be private.

Revalidate instead of guessing a TTL

curl -sS --fail \
  --etag-save "$dir/$key.etag" --etag-compare "$dir/$key.etag" \
  -o "$dir/$key.body" -D "$dir/$key.headers" \
  https://api.example.org/v1/rates

Servers send an ETag header that identifies the current version of a resource. With --etag-compare, curl (7.68 and later) sends it back in If-None-Match; the server answers 304 Not Modified with no body if nothing changed, and curl leaves the output file as it is. With --etag-save it records the new tag after a full response. The request still costs a round trip, but it saves the download, and the data is always current. Many APIs also do not count a 304 against rate limits, though some do; check the documentation. Another option is time-based: -z file (--time-cond) sends If-Modified-Since using the file's modification time.

Respect what the server says

grep -i '^cache-control:' "$dir/$key.headers"

cache-control: private, max-age=300

The Cache-Control header tells you how long a response may be reused: max-age=300 is five minutes, no-store means do not cache at all, and private means it is for one user (fine for your own per-user cache, wrong for a shared one). A script that honors it can set its TTL from the header instead of a constant. Also watch Retry-After and rate-limit headers; a cache reduces calls and does not replace backoff.

What not to cache

• Responses with credentials, personal data or one-time tokens, unless the directory is private to the user (mode 700) and you accept the risk on a shared machine.
• Error responses: cache only successful results (--fail prevents writing a 4xx or 5xx body).
• Results of requests that change state: only cache safe, idempotent GETs.
• Anything where staleness is dangerous, such as a price used to place an order or an access decision.

A refresh switch and cleanup

[[ ${REFRESH:-0} == 1 ]] && rm -f "$file"
find "$dir" -type f -mtime +7 -delete

Give users a way to bypass the cache (an environment variable or a flag), since stale data confuses debugging. Delete old entries on a schedule so the directory does not grow without end. If keys include the Authorization header's identity, hash it too, so two users on one machine do not share responses.

Where this does not apply

A process that runs for a long time is better off caching in memory. Shared caches across machines belong in Redis or a CDN, not a directory; see redis-cli. Data that must be real time, such as a queue or an order status, should not be cached. For pagination and large downloads, caching the pages follows the same idea, with the cursor in the key. If the API is yours, add proper Cache-Control and ETag headers: it is the cheapest fix for all clients.

Deciding what a cache must guarantee

Before writing a cache, decide how stale an answer may be. A list of public holidays can be a month old. A deployment status should be seconds old. An authorization decision should not be cached at all. The answer sets the design: a plain time-to-live suits data where staleness is tolerable and the cost of a call is moderate, and conditional requests suit data that changes unpredictably and where you want to be exact without paying for the full transfer each time. A cache in a script has two jobs, to speed it up and to stay inside the rate limit of the API, and both matter more as the script runs more often, for example from a prompt or a status bar that calls an API on every redraw. The simplest implementation is a file whose modification time says when it was fetched. If the file exists and is younger than the limit, read it, and otherwise fetch, write the result to a temporary file and rename it into place so a reader never sees a half-written cache.

Keep the cache in the right place. The convention is ${XDG_CACHE_HOME:-$HOME/.cache} with a directory named for your tool, with mode 700 if the responses are private, and the key derived from the request, for instance a hash of the URL and the relevant headers.

Conditional requests with ETag and Last-Modified

HTTP has a built-in mechanism for exact caching. A server that sends an ETag header, an opaque identifier of the version of the resource, lets a client ask later whether it has changed by sending If-None-Match with that value. If it has not changed, the server replies with 304 Not Modified and no body, which costs almost nothing and does not count against many rate limits, and you use your saved copy. Last-Modified and If-Modified-Since work the same way with dates. With curl, --etag-save stores the tag in a file and --etag-compare sends it on the next call, available from version 7.68, and -z sets a time condition. Combine both with a short time-based layer: within a minute or two, serve the cache without asking, and after that revalidate. Revalidation gives the exactness of a fresh fetch with the cost of a small request, and the script should handle the 304 explicitly, reading from the cache file.

Different servers implement validators unevenly. Some send a tag that changes on every request, which defeats the cache, and some ignore the conditional header, so check the behavior with a few requests and read the status codes.

Respecting the server, and what not to cache

Read the caching headers the server sends. Cache-Control: max-age=300 says how long the response may be reused, no-store says not to keep it at all, and private says it is specific to one user. A script that ignores no-store and writes sensitive data to disk is doing something the server asked it not to. In any case, avoid caching certain things: responses to requests that change state, authentication tokens and anything containing personal or secret data unless the cache is protected like the data itself. Do not cache errors, or cache them only briefly, so a temporary failure is not remembered as the answer for an hour, and never cache an empty result produced by a failed call. Include in the cache key everything that changes the response: the URL, the query parameters, the account, the language and any header that affects the body. A cache keyed only on the URL returns one user's data to another, which is a security bug and not only a staleness one.

Refreshing on demand and cleaning up

Give the user a switch to bypass the cache, such as a --refresh flag or an environment variable, because there is always a moment when stale data is wrong and waiting for the time limit is not acceptable. Provide a way to clear it too, one command that removes the cache directory for the tool. Clean up automatically: a scheduled find "$cache" -type f -mtime +7 -delete, or a check on each run that removes entries older than a limit, keeps the directory from growing without bound. Put a size limit on the cache if responses are large. Log cache hits and misses when a debug flag is set, which makes it clear whether the cache is working, and measure the effect, since a cache that hits one time in a hundred costs complexity and gains nothing. For several scripts or machines sharing data, a local file is too limited, and a small shared store such as Redis, with expiry built in, is the better home, and the script then asks it for the value before calling the API.

See curl --etag-compare and RFC 9111, HTTP caching.