How to Rotate Credentials and Keys Without Downtime

Changing a password in one place is easy; changing one that twelve services read, without a gap where something fails, is a procedure. The failure mode is a rotation that works for the service you were thinking of and breaks a cron job nobody remembered. The solution is a rollover pattern: both values valid for a while. It applies to API keys, database passwords, SSH keys and signing keys, and ties into leaked secrets in CI logs and compromise response.

The overlap pattern

1. Create credential B            (A still works)
2. Store B where consumers read   (secret manager, env file)
3. Roll consumers to B            (restart or reload, one at a time)
4. Verify: no use of A in the last N minutes
5. Revoke A
6. Record the date and next rotation

Both A and B are valid between steps 1 and 5, so a consumer that has not yet reloaded keeps working. Systems that allow only one active value need a different approach: a brief window with a graceful retry, or a proxy that holds the connection while the secret changes. Databases usually allow two users: create app_v2 with the same grants, switch the application, then drop app_v1.

Database password example

new=$(openssl rand -base64 24 | tr -d '/+=' | cut -c1-24)
psql -U postgres -c "CREATE ROLE app_b LOGIN PASSWORD '$new' IN ROLE app_rw;"
printf 'DATABASE_URL=postgres://app_b:%s@db:5432/app\n' "$new" | install -m 600 /dev/stdin /etc/app/db.env
systemctl restart app.service
psql -U postgres -c "SELECT usename, count(*) FROM pg_stat_activity GROUP BY usename;"

 usename | count
---------+-------
 app_b   |     8
 app     |     0

The last query confirms no connections remain under the old role. Only then drop it with DROP ROLE app;. Generate secrets with a strong source, as in generating strong passwords, and avoid putting the value on a command line where ps shows it (secrets in the process list); the example's -c string is a simplification, and a file with \password is safer.

SSH keys

ssh-keygen -t ed25519 -f ~/.ssh/deploy_2026_11 -C "deploy 2026-11" -N ''
ssh-copy-id -i ~/.ssh/deploy_2026_11.pub [email protected]
ssh -i ~/.ssh/deploy_2026_11 -o IdentitiesOnly=yes [email protected] true
ssh [email protected] "sed -i '/deploy 2025-11/d' ~/.ssh/authorized_keys"

Add the new public key next to the old one, test with the new private key alone, update every machine that holds the old private key, then remove the old public key from authorized_keys. The date in the comment tells you later which key is which. SSH certificates with short validity remove the need for this, see SSH certificates.

API keys and signing keys

Many providers allow two active keys; use that. For signing keys (JWT, webhooks), publish the new verification key before signing with it, wait for the cache time to pass, switch signing, then retire the old key after tokens signed with it expire. This is the rollover used by JWKS endpoints, where each key has an ID so verifiers pick the right one. Webhook secrets should accept both signatures during the window.

After a leak: speed first

If the credential is exposed, skip the graceful steps for the exposed one: revoke it now, accept the outage and fix the consumers, then review how it leaked. Check access logs for use of the old credential between the leak and revocation. Routine rotation (every 90 days, say) limits how long a secret that leaked unnoticed stays useful, but only if the process is easy; automate it with the secret manager's rotation feature, and test that the procedure works before you need it in a hurry.

Why rotation is hard and what makes it easier

Rotation goes wrong because credentials are copied. The password in the database is also in the application's environment file, in a deploy pipeline's secret store, in a colleague's local configuration, in a monitoring tool and in a backup script. Changing the first without finding the others breaks something, usually something that runs rarely and fails at night. The preparation, therefore, is an inventory: for each credential, where it is stored and which systems use it. If the answer is a long list, the credential is a candidate for consolidation, so that only one place holds it and the others read it at run time.

Centralising reduces the work to one update, and it also reduces the risk. A secret manager gives one place to rotate, an audit trail of who read the value and a way to give each consumer its own access. Where the manager can generate and rotate database credentials itself, handing out short-lived accounts, you arrive at the best state, in which rotation is automatic and nobody ever sees the value.

Testing the procedure

A rotation procedure that has never been run is a guess. Run it first against a staging environment, with the same number of consumers, and check each one afterwards. Time the process, note which step needed a restart, and write the commands in a runbook that someone else can follow. Include the check that confirms the old credential is no longer in use, because that check decides when it is safe to revoke it, and skipping it is how a forgotten consumer gets discovered by an outage.

Practise the emergency version as well: a revocation under time pressure, with consumers failing and an on-call engineer fixing them. The graceful procedure takes an hour and the emergency one takes minutes, and the second needs different tooling, such as a way to push a new secret to every consumer at once and restart them in a sensible order.

Detecting stale and orphaned credentials

Credentials accumulate. Keys created for a one-time task, accounts belonging to people who left, tokens for tools no longer in use: each is a standing risk. Audit them on a schedule. List credentials with their creation date and last-used date, which most providers record, and review anything unused for ninety days. Disable first, wait a week for complaints, then delete. This practice finds the key in a forgotten script as reliably as any search of the codebase.

Reducing the need over time

Every rotation is a reminder of the cost of long-lived secrets. Where a platform supports identity federation, replace stored keys with tokens issued to the workload for its lifetime. Cloud instance roles, Kubernetes service accounts with projected tokens and CI systems with OIDC trust relationships all remove a stored secret. The migration takes effort once, and afterwards a whole category of rotation work, and a whole category of leak, goes away.

Where this does not apply

The better fix for credentials that are painful to rotate is not to have them: use short-lived tokens from OIDC, instance roles or workload identity, so nothing persistent exists. Hardcoded copies (in images, scripts, old laptops) defeat rotation; find them with gitleaks and trufflehog. See the OWASP secrets management cheat sheet.

Rotation also needs an owner and a calendar. Record where each credential is used, who owns it and when it was last changed, in a short document or the secret manager itself. The NIST key management guidelines describe cryptoperiods, the recommended lifetime for different key types, if you need a reference for policy.