BlogSecrets
Automating secret rotation without downtime
Rotate database and API credentials on a schedule using overlapping versions so nothing breaks mid-rotation.
Rotation breaks things when the old credential dies before every client has the new one. The fix is an overlap window: create version N+1, roll it out to all readers, and only then disable N. Nothing ever sees a gap.
1
Create v2
both valid
2
Deploy readers
move to v2
3
Verify
no v1 usage
4
Disable v1
window closes
Naive (breaks)
change password
old dies instantly
in-flight clients fail
a mini outage
Overlapping
two valid versions
roll readers over
confirm, then revoke
no gap
Let the manager do it
Dynamic secrets sidestep rotation entirely — every consumer gets its own short-lived credential. Where you must rotate a shared one, use the platform's staged rotation.
bash
# AWS Secrets Manager: staged rotation with a Lambdaaws secretsmanager rotate-secret \--secret-id prod/db \--rotation-lambda-arn arn:aws:lambda:...:rotate-db \--rotation-rules AutomaticallyAfterDays=30
Give the window room
The overlap must be longer than your longest cache/lease TTL and deploy time. Rotate faster than that and you recreate the outage you were avoiding.