Zero-downtime rotation & the dual-secret pattern

Rotate roots, leases, and app credentials without an outage.

Rotation is where secrets programs quietly fail. Everyone agrees credentials should rotate; then the first rotation causes an outage, and rotation gets disabled "temporarily" for two years. The reason is almost always the same: the system flipped to a new credential before every consumer had picked it up. The fix is a pattern, not a tool — keep two credentials valid at once during an overlap window longer than anything that caches the old one. Get that right and rotation becomes boring, which is exactly what you want.

The dual-secret (two-slot) pattern

Maintain two active credential versions: current and previous. To rotate, create a new version and promote it to current, demoting the old current to previous — both still work. Consumers refresh on their own schedule and start using the new value during the overlap. Only after the overlap window has fully elapsed — longer than your longest cache TTL, lease duration, and deploy time combined — do you retire the previous version. AWS Secrets Manager encodes this directly with AWSCURRENT and AWSPREVIOUS staging labels; you build the same thing by hand elsewhere.

Rotating the things Vault itself holds

Dynamic secrets mostly rotate themselves — leases expire — but the privileged root credentials the engines use do not, and those are the crown jewels. Vault can rotate its own database and cloud engine root credentials so that after rotation nobody, including you, knows the password: it changes the credential in place and stores the new one internally. The barrier encryption keys rotate too (vault operator rotate), adding a new keyring version for future writes without touching old data. The rule: the higher the privilege, the more it must be rotated on a schedule and the less any human should know it.

# rotate the DB engine’s privileged connection password in place:
$ vault write -f database/rotate-root/app-postgres
# Vault sets a new password and keeps it internally — no human copy exists.
 
# rotate cloud engine root creds similarly:
$ vault write -f aws/config/rotate-root
 
# add a new barrier key version for future writes (old data still readable):
$ vault operator rotate
Key Term  3   Install Time  ...

Emergency vs scheduled rotation

Distinguish the two clearly. Scheduled rotation is proactive hygiene: it caps the lifetime of any credential and — crucially — it keeps the rotation path exercised, so the emergency version is not the first real run. Emergency rotation is the compromise response: rotate now and accept a possible blip, because a live attacker outranks a clean deploy. The dual-secret pattern makes scheduled rotation invisible; a rehearsed emergency runbook makes the compromise case fast. If you only build one, build the scheduled path first — an unrehearsed emergency rotation during an incident is how you turn a breach into an outage on top of a breach.