Rotation habits that stick

Dual-run windows and owners, not calendars alone.

Rotation is changing the locks. The awkward part is counting who holds a key: a deploy pipeline, two services, a scheduled job on a machine nobody has logged into since March, someone's laptop, a backup agent. When an office swaps its badge system, the security team does not switch the readers over at noon and leave staff queueing in the car park. The readers are programmed to accept old badges and new badges for a fortnight, replacements go out, and the old generation stops working on an announced date. Credential rotation runs on the same plan, for the same reason, and it fails in the same way when somebody skips the overlap.

The security meaning is narrower than the everyday one. Rotation means issuing a replacement credential and making the old one stop working. Both halves count, and the second half is the one that protects you. An attacker holding a copy of your database password does not care that you generated a new one. They care whether the copy in their notes still opens the door. Issuing is easy. Killing the old value on purpose, at a moment you choose, is the actual control.

This is why a date in a calendar proves nothing on its own. "Rotate the payments database password, quarterly" measures paperwork. You can change the value in the secret store, close the ticket, and still have the old password serving live traffic, because the application read it once at start-up and handed it to a connection pool (the set of already-open database connections an app keeps on hand so it does not pay the cost of reconnecting for every query). Those connections proved who they were when they opened, and nothing rechecks them afterwards. A pool with no maximum connection lifetime will hold a session open for a week without blinking. The store says rotated. The database disagrees.

A rotation is finished when two things are true at the same moment: every consumer presents the new value, and the old value is refused. Everything in between is a rotation in progress. That state deserves a name out loud, because it needs watching and it needs an end.

What A Rotation Needs Before You Start

Treat each secret the way you treat an on-call rota, the roster that decides who gets woken at 3am. Four attributes, and if any one is missing you have a sticky note rather than a control. An owner, meaning a team that gets paged, not a person who left in 2024. A maximum age for that class of secret. A delivery path that can pick up a new value without a human editing a file on a server. And a revoke step somebody has run in a rehearsal and timed with a stopwatch.

Write it somewhere a machine can read: name, owner, where the value lives, how it reaches consumers, last rotated, next due. Automation beats a spreadsheet, and a spreadsheet beats folklore, so start with whichever one you can finish this week. The field teams forget is the rehearsal date, and it is the field that predicts how your first real incident will go.

# one entry per secret, kept next to the code that uses it
- name: payments-db-app
  owner: team-payments                 # a rota, not a person
  store: vault:database/creds/payments-rw
  delivery: vault agent renders /etc/app/db.env, then signals the app to reload
  max_age: 1h                          # dynamic lease: expiry is the rotation
  revoke: vault lease revoke -prefix database/creds/payments-rw
  revoke_rehearsed: 2026-06-18
 
- name: ci-bot-aws-key
  owner: team-platform
  store: github-actions-secret:AWS_ACCESS_KEY_ID
  delivery: pipeline environment, picked up on the next run
  max_age: 90d
  revoke: aws iam update-access-key --status Inactive, then delete-access-key
  revoke_rehearsed: never              # this is the one that will hurt

An inventory is only as honest as the account it describes, so check it against the real thing. Active cloud keys older than their own maximum age are the cheapest finding in all of cloud security, and every account has a few.

# every active access key in the account, oldest first
aws iam list-users --query 'Users[].UserName' --output text | tr '\t' '\n' |
while read -r u; do
  aws iam list-access-keys --user-name "$u" \
    --query "AccessKeyMetadata[?Status=='Active'].[CreateDate,UserName,AccessKeyId]" \
    --output text
done | sort

2024-03-19T04:12:07+00:00	backup-agent	AKIAJ2N7QVX3ZEXAMPLE
2025-11-02T11:31:44+00:00	ci-bot	AKIAI44QH8DHBEXAMPLE
2026-06-30T08:55:10+00:00	terraform-runner	AKIAZ5T6YW9QREXAMPLE
# 3 active keys, 2 of them past the 90-day maximum for this class

The oldest line is a credential that has outlived two people and one reorganisation. The ci-bot key underneath it is nearly nine months old against a ninety-day limit. Deleting either one right now would be brave. The dual-run window is how you retire a key without guessing.

The Dual-Run Window

Dual-run means two valid credentials at once, on purpose, for a measured period. Four steps, in this order. Issue B while A is still live and serving traffic. Ship B to every consumer and confirm each one is using it. Wait longer than the longest cache or lease that could still be holding A. Then disable A, watch what breaks, and delete it. The pattern fits anything with two slots: API keys (application programming interface keys, the strings one program sends another to prove who it is) where the provider lets you hold a primary and a secondary, database logins where you can create a second user with matching grants, and cloud access keys.

Amazon Web Services (AWS, Amazon's cloud platform) gives every IAM user (Identity and Access Management, the service that stores users, roles and permissions) exactly two access key slots. That limit reads like an annoyance to work around. Treat it instead as one dual-run slot, and it keeps you honest, because you cannot start a second rotation until you have finished the first.

# step 1: issue B while A is still live. Nothing has moved yet.
aws iam create-access-key --user-name ci-bot

{
    "AccessKey": {
        "UserName": "ci-bot",
        "AccessKeyId": "AKIAIOSFODNN7EXAMPLE",
        "Status": "Active",
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "CreateDate": "2026-07-27T09:14:02+00:00"
    }
}
# the secret value is shown exactly once, here, and never again

If the command fails instead, you have learned something useful about the last rotation.

aws iam create-access-key --user-name payments-worker

An error occurred (LimitExceeded) when calling the CreateAccessKey operation: Cannot exceed quota for AccessKeysPerUser: 2

Both slots are full, which means the previous rotation never finished. Somebody created a new key, moved the traffic across, and never came back to clean up. That leftover is a fully privileged credential with nobody watching it, and today it is blocking the rotation you actually came here to do. Unfinished dual-runs compound.

Before you delete anything, ask AWS whether the old key is still being used.

aws iam get-access-key-last-used --access-key-id AKIAI44QH8DHBEXAMPLE

{
    "UserName": "ci-bot",
    "AccessKeyLastUsed": {
        "LastUsedDate": "2026-07-27T09:41:00+00:00",
        "ServiceName": "s3",
        "Region": "eu-west-1"
    }
}

A timestamp from four minutes ago means something out there is still authenticating with that key, and deleting it now converts a rotation into an outage. Read the field the careful way round. Recent use proves the key is alive; silence does not prove it is dead. AWS writes that field asynchronously, so it lags real traffic, a nightly job looks idle all morning, and a key that has genuinely never been used comes back with no date at all and N/A in the service and region fields. The safe middle step is deactivation, which you can undo in seconds.

# reversible: the key stops working but still exists
aws iam update-access-key --user-name ci-bot \
  --access-key-id AKIAI44QH8DHBEXAMPLE --status Inactive
 
# after one full business cycle with no new errors, make it permanent
aws iam delete-access-key --user-name ci-bot \
  --access-key-id AKIAI44QH8DHBEXAMPLE
 
aws iam list-access-keys --user-name ci-bot \
  --query 'AccessKeyMetadata[].[AccessKeyId,Status,CreateDate]' --output text

# update-access-key and delete-access-key print nothing on success
AKIAIOSFODNN7EXAMPLE	Active	2026-07-27T09:14:02+00:00
# SUCCESS: one active key, and the second slot is free for the next rotation

Measuring The Overlap Instead Of Guessing

The overlap length is a number you measure, not a number you feel. It is the longest time any consumer could still be holding the old value, plus deploy time, plus a margin for the slowest corner of the estate. Every layer that caches adds to the total, and most of them cache quietly.

Kubernetes shows the problem clearly, because the answer changes with how the Secret is consumed. A Secret exposed as environment variables is read once when the container starts, so a running pod keeps the old value for as long as it lives, whatever you change in the API. A Secret mounted as a volume gets refreshed by the kubelet (the agent Kubernetes runs on every worker machine) on its periodic sync, one minute by default, plus however long the kubelet's own cache takes to notice, so that file on disk does change by itself. A Secret mounted with subPath is never updated. A Secret created with immutable: true cannot be updated at all, and the only way forward is to delete it and make a new one. Even when the file on disk changes, the process still has to re-read it, which plenty of applications never do after boot.

Worth saying plainly while you are in there. The value in a Secret manifest is base64, an encoding for safe transport, the same idea as spelling a surname out letter by letter over a bad phone line. It is not encryption. There is no key and no secrecy in it, and anyone who can run kubectl get secret -o yaml decodes it with one more command. Turning on encryption at rest protects the copy sitting on disk in etcd (the database Kubernetes keeps its objects in) and changes nothing about who can read the object through the API. Access control is what keeps people out, never the base64.

# write the new value into the Secret
kubectl create secret generic payments-db \
  --from-literal=password="$NEW_PW" \
  --dry-run=client -o yaml | kubectl apply -f -
 
# env-var consumers only see it once their containers are replaced
kubectl rollout restart deployment/payments-api
kubectl rollout status deployment/payments-api --timeout=180s

secret/payments-db configured
deployment.apps/payments-api restarted
Waiting for deployment "payments-api" rollout to finish: 1 old replicas are pending termination...
deployment "payments-api" successfully rolled out

For environment-variable secrets the restart is the delivery path, and it brings a pleasant side effect. The change becomes observable. Either the new pods come up healthy on the new credential or they crash-loop in front of you, which beats finding out at 3am when the old key finally dies.

The same arithmetic runs further out. If a content delivery network (CDN, the layer of edge servers that keeps copies of your responses close to users) holds a config document for thirty minutes, your overlap has to exceed thirty minutes plus deploy time plus margin. That thirty minutes is a TTL (time to live, how long a cached copy may be reused before it has to be fetched again). Write the number into the runbook with its measurement beside it, so nobody trims it later for feeling excessive. "Forty-five minutes, because the edge TTL is thirty and the rollout takes nine" survives review. "About an hour, probably" does not.

Then stop reasoning and ask the system. PostgreSQL will tell you exactly who is connected right now and how long they have been there.

psql -h db.internal -U admin -d payments -c "
  SELECT usename, count(*) AS sessions, min(backend_start) AS oldest_session
  FROM pg_stat_activity
  WHERE usename LIKE 'app_rw%'
  GROUP BY usename ORDER BY usename;"

    usename    | sessions |         oldest_session
---------------+----------+-------------------------------
 app_rw_2026q2 |        4 | 2026-07-27 07:58:03.214887+00
 app_rw_2026q3 |       22 | 2026-07-27 09:48:53.771043+00
(2 rows)

Twenty-two sessions moved. Four did not. Two hours after the deploy those four are a finding rather than a rounding error: somewhere a pool is holding connections open, or a background worker never restarted, or a reporting host keeps its own copy of the config that nobody remembered. Chase those four down before you drop anything. Dropping the role while they are still attached is what turns a tidy rotation into a page.

Databases And Certificates Use The Same Trick

Most databases give a user one password and no spare slot, so you build the spare yourself. One group role holds every permission the application needs, the way a job description carries duties that outlast whoever is doing the job this quarter. Two login roles inherit from that group and take turns. Grants live on the group and never move. The logins are disposable, which is the whole point.

-- app_rw is a group role: it holds every grant the application needs
-- and is never used to log in. Login roles inherit from it.
-- run with: psql -v new_password="$NEW_PW" -f rotate-app-role.sql
 
CREATE ROLE app_rw_2026q3 LOGIN PASSWORD :'new_password';
GRANT app_rw TO app_rw_2026q3;    -- identical rights to the outgoing login
 
-- consumers move across during the overlap, then the old login retires

psql -h db.internal -U admin -d payments -c 'DROP ROLE app_rw_2026q2;'

ERROR:  role "app_rw_2026q2" cannot be dropped because some objects depend on it
DETAIL:  owner of table payments_ledger
privileges for schema reporting

PostgreSQL refuses because the old login still owns things and still holds grants. This is the moment people discover that their "temporary" login role from two quarters ago quietly became the owner of half the schema. Two commands clear it, and the order matters. REASSIGN OWNED BY hands the owned objects to the group role, which is precisely why the group role exists. DROP OWNED BY then removes what reassignment leaves behind, because reassigning ownership does not revoke privileges the login was granted on objects it never owned. Skip that second command and the drop fails again, with those schema privileges still listed. Both commands act on the current database only, so run them in every database the role ever touched.

psql -h db.internal -U admin -d payments <<'SQL'
REASSIGN OWNED BY app_rw_2026q2 TO app_rw;
DROP OWNED BY app_rw_2026q2;
DROP ROLE app_rw_2026q2;
SQL

REASSIGN OWNED
DROP OWNED
DROP ROLE
# SUCCESS: ownership sits on the group, logins stay disposable

Certificates rotate on the same shape. Issue the new one, reload the listener, then let the old one expire. On nginx a reload keeps existing connections on the old worker processes while new connections pick up the new certificate, so nobody's request gets cut in half. Revocation exists, and for a public web certificate you should treat it as a weak backstop rather than an off switch, because clients check revocation unevenly or not at all. Expiry is the mechanism that reliably retires a certificate. What you must not do is look at the file on disk and call the job finished. Ask the socket, from outside, the way a client would.

echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
  | openssl x509 -noout -subject -dates -serial

subject=CN = api.example.com
notBefore=Jul 27 08:14:00 2026 GMT
notAfter=Oct 25 08:13:59 2026 GMT
serial=04C9A2B7E1F58D3067B4A19C

The serial number is the field to record, because it is the only one that proves the edge is serving the certificate you issued rather than a cached copy held by a load balancer that never reloaded. Short-lived certificates shrink the whole problem. One valid for twenty-four hours is one nobody has to revoke in a panic, because expiry does the work unattended.

Prefer Issuance Over Rotation

The best rotation is the one you deleted the need for. A hotel does not change every lock when a guest leaves; reception prints a card that dies on the checkout date, and nobody chases you for it afterwards. Do the same with credentials. Instead of a long-lived value that a human moves on a schedule, have the platform mint a fresh one on demand with an expiry already attached. HashiCorp Vault's database secrets engine creates a real database user per request, hands you the password, and drops that user when the lease ends.

vault read database/creds/payments-ro

Key                Value
---                -----
lease_id           database/creds/payments-ro/9c0a5f2e-1b4d-4a77-9e33-6f1c2a8d5b47
lease_duration     1h
lease_renewable    true
password           A1a-9Zq2VbN7xM4pLr0T
username           v-token-payments-8fKq2mXzR4pLnT7vBw3c-1785142201

That username did not exist ten seconds before the command ran, and it will be gone an hour later without anyone filing a ticket. Rotation becomes expiry. Emergency revocation becomes one command across an entire prefix, which is the response you want available at two in the morning when a laptop goes missing.

vault lease revoke -prefix database/creds/payments-ro

All revocation operations queued successfully!

Continuous integration (CI, the system that builds and tests your code on every push) is where static keys pile up, and it is also where they are easiest to remove entirely. OpenID Connect (OIDC, a standard way for one system to prove its identity to another using a short-lived signed token) lets a pipeline swap that token for cloud credentials while the job is running, so there is no key sitting in the repository settings for anyone to copy.

permissions:
  id-token: write        # let the runner request a short-lived OIDC token
  contents: read
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-deploy
          aws-region: eu-west-1
      # note what is absent: no AWS_SECRET_ACCESS_KEY in repository secrets
      - run: aws sts get-caller-identity

The workflow is half of the setup. On the AWS side you register GitHub's OIDC issuer as an identity provider and write a trust policy on the role with a condition that pins the exact repository and branch allowed to assume it. Leave that condition loose and any repository on GitHub can ask for your role, which trades a stored key for an open door. Underneath, the runner is calling sts:AssumeRoleWithWebIdentity against the Security Token Service (STS, the AWS service that issues temporary credentials), and you can see what it got from inside the job.

aws sts get-caller-identity

{
    "UserId": "AROAV3QK7EXAMPLEIDABC:GitHubActions",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/gha-deploy/GitHubActions"
}
# assumed-role, not user: this session expires on its own in about an hour

For humans, single sign-on (SSO, one central login that every application trusts) does the same job from the other direction: one identity to disable when somebody leaves, instead of forty local passwords with forty rotation dates and one forgotten admin account on a legacy box.

The trade is real and worth stating plainly. Dynamic credentials put a broker in the start-up path of every workload, so if Vault is unreachable your new pods cannot get a password and will not start. You have swapped a calendar problem for an availability problem, and the bill for that comes as running the broker in high availability, choosing lease lengths that are short but survivable, and having a tested answer for what happens when a whole cluster restarts at once and asks for ten thousand database users inside a minute. Break-glass accounts, the emergency logins you keep for the day the identity provider itself is down, have to sit outside SSO by design, and those are ordinary passwords needing an owner, a safe and a date. Short-lived credentials are still the right answer. They are an engineering commitment rather than a free upgrade.

Making It Stick

Rotation survives contact with a busy quarter when it is boring, owned and observed. Four habits carry most of the weight.

Alert on age, not on the calendar. A daily job that lists every credential older than its class maximum and files a ticket with the owning team beats a recurring meeting, because it survives holidays, reorganisations and the person who used to remember.

Canary the result. After every rotation, run a check that authenticates with the new value and confirms the old value is refused. Both assertions matter, and the second is the one people skip.

# 1) the new value works
PGPASSWORD="$NEW_PW" psql -h db.internal -U app_rw_2026q3 -d payments -qtc 'select 1' >/dev/null \
  && echo "new credential: accepted"
 
# 2) the old value is refused
PGPASSWORD="$OLD_PW" psql -h db.internal -U app_rw_2026q2 -d payments -qtc 'select 1' >/dev/null 2>&1 \
  && echo "old credential: STILL WORKS" || echo "old credential: refused"

new credential: accepted
old credential: refused
# SUCCESS: both halves of the rotation are proven, not assumed

Treat a failed rotation as an incident, with the same write-up you would give a customer-facing outage. A job that reports green while half the fleet still uses the old key teaches a whole team to ignore its own automation, and that habit takes a year to undo. The failure deserves more attention than the success.

Keep the evidence: the ticket, the pipeline run, the last-used output taken before deletion, the canary result. Auditors ask for it, which is the boring reason. The useful reason is that the next person to rotate this secret gets to read what actually happened last time, real overlap included, instead of guessing the way you had to.

Try This

Pick one non-production credential with two slots: an API key that supports a secondary, or a pair of database roles you are allowed to create. Run the whole cycle end to end, and time it. Before you start, look at how old the existing sessions are, because that number is the one your runbook has been guessing at.

psql -h db.internal -U admin -d payments -c "
  SELECT usename, date_trunc('second', now() - backend_start) AS session_age
  FROM pg_stat_activity WHERE usename = 'app_rw_2026q2'
  ORDER BY backend_start LIMIT 3;"

    usename    |   session_age
---------------+-----------------
 app_rw_2026q2 | 6 days 04:11:52
 app_rw_2026q2 | 6 days 04:11:52
 app_rw_2026q2 | 2 days 19:02:07
(3 rows)

A six-day-old session is a password you changed five days ago that is still in daily use. Set your overlap window from evidence like that, then run the cycle: issue B, deploy, wait, deactivate A, canary, delete A, and write the measured numbers into the runbook while they are fresh.

All of this assumes you already have somewhere sensible to keep the new value while it travels. Choosing that home, Vault against a cloud-native manager against encrypted files in git, is the next lesson, and it arrives with a decision tree rather than a favourite.

Takeaway

The trap worth remembering here: two Live Keys Is Twice The Blast Radius. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related