Rotation and break-glass

Root credential rotation and the sealed-glass drill.

The rekey ceremony ended at 16:40 on a Friday. Five new envelopes went out by courier, three holders confirmed receipt, everybody logged off. Eleven days later a bad policy push locks the platform team out of every sys/ path, somebody starts the break-glass drill, and the second holder finds that her share was printed across a page break with the tail of it on a sheet nobody kept. The old shares stopped working the instant that Friday rekey completed. Two usable shares, a threshold of three, and no way to mint a root token to fix the policy.

Break-glass is the fire axe in the case on the wall, and the case is the interesting part. Anyone can smash it in four seconds, so it is not a lock. It is a record. Broken glass tells you the axe left the wall, roughly when, and who was standing nearby. Vault's axe is a root token. Vault's glass is a quorum of key holders plus an audit line that fires the moment somebody starts the ceremony.

Every other lesson in this course hands out credentials that kill themselves: hour-long database users, 24-hour certificates, tokens with a TTL (time to live, the countdown after which a credential stops working). This lesson covers the small set of secrets that sit above that machinery and cannot expire on their own. They rotate on a calendar instead, on five different clocks, and mixing two of them up is how a hygiene task becomes an outage.

Five Things Rotate, And They Are Not The Same Thing

Write the list on the runbook page before you touch anything. The barrier keyring is the set of keys Vault encrypts storage with, and vault operator rotate advances it. The unseal or recovery shares decide who can rebuild the key that opens the barrier, and vault operator rekey replaces them. The root token bypasses every policy, and vault operator generate-root mints one from a quorum. Each secrets engine holds one privileged login of its own, and rotate-root changes it. Static roles keep a fixed database username and rotate only its password. Five clocks: keyring monthly and unattended, shares when a holder leaves, root during an incident or a drill, engine credentials quarterly, static roles on their own schedule.

One boundary explains most of the confusion in this area, so say it out loud. Unsealing is decryption, not login. Vault encrypts everything it writes with a barrier keyring. That keyring is encrypted by a single root key. The root key is either split into Shamir shares (named after Adi Shamir, whose scheme cuts a secret into N pieces of which any T can rebuild it) or wrapped by a cloud KMS (key management service, the cloud's own hardware-backed encryption service). Feeding Vault enough shares rebuilds the root key and opens the barrier. It logs nobody in. Somebody holding three shares still cannot read secret/data/prod/db, because reads need a token and a policy.

There is exactly one place where shares turn into authority, and it is this lesson's subject. A quorum can run generate-root, and the token that comes out ignores every policy you wrote in the ACL lesson. That is why share custody is a security control and not a filing problem. The cryptography is the axe. The ceremony around it is the glass.

There Is No Root Token At Rest

A root token is a bootstrap credential with a very short intended life. Vault prints one at vault operator init so you have something to enable an auth method with, write the first policies with, and then destroy. It carries the root policy, which is not a policy document at all: it is a flag that skips ACL evaluation completely. It has no TTL, so it never expires on its own. No policy can constrain it, no lease reclaims it, and the only thing between a copied root token and your entire secret estate is that somebody remembered to revoke it.

So the target state is blunt. Once OIDC works for humans and Kubernetes auth works for workloads, revoke the initial root token and keep zero root tokens at rest. Not one in a password manager, not one in the platform team's shared entry titled "do not use". When you need root again you make one, hold it for four minutes, and destroy it. The price is a ceremony you have to be able to run under pressure, which is why most of this lesson is about rehearsal rather than commands.

vault operator generate-root -init opens an attempt and prints two things: a nonce (a one-time reference number for this attempt) and an OTP (one-time password). Each key holder submits their share against that nonce. When the threshold is reached, Vault builds the new root token, XORs it with the OTP, and prints the result as an encoded token. XOR against a pad of the same length is the one encryption scheme that cannot be broken when the pad is used once, which is why the OTP is exactly as long as the token it hides. The encoded token is useless without the OTP, and the OTP is useless without the encoded token.

That is what makes the ceremony safe over ordinary channels. Whoever started the attempt keeps the OTP on their own machine, so the encoded token can be pasted into the incident channel without handing root to everybody in it. Decode locally, do the one task, then destroy the token before you close the terminal.

# Kick off the drill: prints a Nonce and a one-time password (OTP)
vault operator generate-root -init
 
# Each key holder runs this and pastes their unseal/recovery key share
vault operator generate-root -nonce=<nonce>
 
# After enough shares, you get an Encoded Token — decode it with the OTP
vault operator generate-root \
  -decode=<encoded-token> \
  -otp=<otp>
 
# Do the one privileged task, then destroy the token immediately
vault token revoke -self
 
# Started a drill by mistake? Abort it
vault operator generate-root -cancel

Nonce         a1b2c3d4-...
Started       true
Progress      0/3
Complete      false
OTP           VU9UUC1leGFtcGxl...
OTP Length    26
 
# after threshold shares + decode
Root token: hvs.CAESIJbreakglass
# SUCCESS — use it, then vault token revoke -self immediately

Three things go wrong here, all of them in the first ten minutes. Two operators both run -init, and the second is told an attempt is already in progress, because Vault keeps one at a time; -status shows you whose and -cancel clears it. Holders submit against a stale nonce copied out of an earlier drill, and the submission is rejected without moving anything forward. And a well-formed share that happens to be wrong does not fail where you typed it: Vault holds the pieces and only combines them once the threshold is reached, so a bad character in the first share surfaces as a failure on the third. Read Progress after every submission instead of assuming.

Prove the drill ended the way you think it did. vault token lookup on the fresh token shows policies [root], ttl 0s and an empty expire_time, which is the whole problem with root spelled out in three lines. After revoking, list the token accessors and check that nothing root-shaped survived. An accessor is a handle to a token that lets you look it up and revoke it without ever seeing the token itself, and listing them needs a policy with sudo on auth/token/accessors.

# what am I actually holding?
vault token lookup
 
# after the one privileged task
vault token revoke -self
 
# is anything root-shaped still alive on this cluster?
vault list auth/token/accessors
vault token lookup -accessor <accessor>

Key                 Value
---                 -----
display_name        root
expire_time         <nil>
explicit_max_ttl    0s
orphan              true
path                auth/token/root
policies            [root]
renewable           false
ttl                 0s
# (trimmed)
 
Success! Revoked token (if it existed)

Rotate The Keyring, Rekey The Shares

These two commands sound like synonyms and do unrelated jobs. vault operator rotate advances the barrier keyring so new writes use a fresh key term. It needs no quorum, finishes in milliseconds, and is safe to run from cron. vault operator rekey mints a brand new set of unseal shares (or recovery shares under auto-unseal) and needs a quorum of the current shares to authorize it. One protects data written from now on. The other protects your ability to open the vault at all. Give them different owners: platform automation rotates the keyring, security and the share holders run rekey.

Here is the part people get wrong about rotate. It does not re-encrypt anything already on disk. Vault keeps every past key term in the keyring precisely so it can still read old values, and a stored secret stays encrypted under whichever term was current when it was written. Rotating after you suspect a key term leaked limits what a future copy of storage gives away; it does nothing for the ciphertext already sitting in the copy somebody took last month. The fix for old data is rewriting it, which for KV v2 means writing a new version, and for dynamic secrets happens on its own as leases turn over.

vault read sys/key-status answers the only two questions worth asking after a rotate: which term is current, and when it was installed. The encryption count beside them is why Vault also advances the keyring on its own once a term has been used for a very large number of operations, in the billions, with nobody scheduling it. Your monthly rotate is a statement about hygiene and a check that the command still works, not a load-bearing safety mechanism.

# Cheap, quorum-free: advance the encryption keyring
vault operator rotate
vault read sys/key-status          # current key term + install time
 
# Heavy: mint new unseal shares. Needs a quorum of the CURRENT shares.
vault operator rekey -init -key-shares=5 -key-threshold=3
vault operator rekey -nonce=<nonce>   # each holder submits a current share
 
# Under auto-unseal, the Shamir shares are recovery keys — target them:
vault operator rekey -target=recovery -init -key-shares=5 -key-threshold=3

Key              Value
---              -----
term             7
install_time     2026-07-24T01:12:00Z
encryption_count 184422
 
Success! Rekey initialized:
Nonce: ...
Key Shares: 5
Key Threshold: 3
# Progress shows as each holder submits until Complete=true

Rekey is the operation that can end a cluster. The moment it completes, the old shares are dead. There is no grace period where both sets work, no undo, and no support ticket that recovers a lost share. Under Shamir the failure is total: lose the new shares, restart a node, and nobody can unseal it, and every byte in storage stays ciphertext forever. Under auto-unseal the failure is quieter and still serious. The cluster keeps running because the KMS keeps opening the barrier, applications never notice, and you find out on the day you need generate-root that the recovery quorum is gone.

-require-verification=true at init is the safety belt for exactly the Friday afternoon in the opening story. Vault issues the new shares and then waits: a quorum has to submit those new shares back against a verification nonce before the change takes effect, so an envelope that never arrived, printed badly, or went to the wrong desk shows up while the old set is still valid. It costs one extra round of confirmations and turns an irreversible step into one you can walk away from.

-backup=true is the other belt, and it has a string attached. It works only alongside -pgp-keys, one key per share, and it stores a PGP-encrypted copy of each new share inside Vault's own storage, retrievable with -backup-retrieve. Genuinely useful when a courier loses an envelope. Also a complete set of your shares living in the system those shares protect. Retrieve, distribute, then -backup-delete as the final step of the ceremony, with a checkbox next to it rather than a note at the bottom of the page.

Two rules keep the ceremony boring, which is the goal. Everyone submits against the same nonce, pasted once into the shared channel by whoever ran -init, and everyone points at the same address. Run -status first, in case a half-finished attempt from last quarter's drill is still sitting there.

# nothing half-finished from an earlier drill?
vault operator rekey -status
 
# start a rekey that does not take effect until the new shares are proven
vault operator rekey -init \
  -key-shares=5 \
  -key-threshold=3 \
  -require-verification=true
 
# each holder submits a CURRENT share against the nonce
vault operator rekey -nonce=<nonce>
 
# then each holder submits their NEW share against the verification nonce
vault operator rekey -verify -nonce=<verification-nonce>
 
# optional: PGP-encrypted copies of the new shares, kept inside Vault
vault operator rekey -init -key-shares=5 -key-threshold=3 \
  -pgp-keys="keybase:alice,keybase:bob,keybase:carol,keybase:dan,keybase:erin" \
  -backup=true
vault operator rekey -backup-retrieve
vault operator rekey -backup-delete
 
# a holder dropped off the call
vault operator rekey -cancel

Rotate What The Engines Hold

The database and cloud engines each hold exactly one long-lived credential: the Postgres superuser Vault logs in as to create throwaway users, or the AWS key it uses to mint short-lived ones. That bootstrap credential is the last standing password in a chain designed to have none, and rotate-root deals with it in one call. Vault changes the password at the database, stores the new value, and never gives it back. There is no read endpoint, no recovery, no way to ask what it set. Afterwards the only copy lives inside Vault's encrypted storage, which is the entire point.

# The Postgres/MySQL superuser configured on the database engine
vault write -force database/rotate-root/app-postgres
 
# Cloud engines expose the same pattern for their root key
vault write -force aws/config/rotate-root
vault write -force gcp/config/rotate-root

Success! Data written to: database/rotate-root/app-postgres
# no password returned — Vault alone knows the new value
# verify by minting a fresh dynamic cred:
vault read database/creds/app-readwrite
# lease_id ... SUCCESS means the new root still talks to Postgres

One precondition decides whether that call is hygiene or an outage. The connection config has to carry the credentials in its own username and password fields, with the connection URL using the {{username}} and {{password}} placeholders, the way the config in the dynamic secrets lesson does. If somebody pasted the real password into the URL instead, the engine ends up dialing with a string that no longer opens anything, and the first symptom is every vault read database/creds/app-readwrite failing authentication against Postgres while Vault itself reports perfect health.

The second way this breaks has a pipeline in it. Whatever tool owns database/config/app-postgres also owns the password field in its state, and a routine apply after a rotation writes the old bootstrap value straight back over the rotated one. Terraform will do this during an unrelated change and show it as a one-line diff nobody reads twice. Pin it: mark that field as ignored for drift, or move the connection config into a one-shot bootstrap job and keep it out of the recurring apply. Then test the fix by running the pipeline immediately after a rotation and minting a credential.

Some databases refuse the password Vault generates. Legacy installs with complexity rules, appliances that cap length, systems that reject a character the generated string happens to contain. The rotation fails at the database side and you learn about it when the engine can no longer log in. Vault password policies fix this: describe the shape the database will accept, attach it to the connection with password_policy, and generate a sample to check before you rotate anything real.

length = 24
 
rule "charset" {
  charset = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
}
 
rule "charset" {
  charset   = "0123456789"
  min-chars = 2
}
 
rule "charset" {
  charset   = "!@#%^&*"
  min-chars = 1
}

vault write sys/policies/password/postgres \
  policy=@/etc/vault.d/password-policies/postgres.hcl
 
# sample one and check the database will accept that shape
vault read sys/policies/password/postgres/generate
 
# attach it: a config write must repeat every field, not only the new one
vault write database/config/app-postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="app-readwrite" \
    connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/appdb?sslmode=require" \
    username="vault_admin" \
    password="$VAULT_DB_BOOTSTRAP_PW" \
    password_policy="postgres"
 
vault write -force database/rotate-root/app-postgres

Static roles cover accounts that cannot take a generated username: a vendor integration, a reporting tool whose grants were set up by hand, anything where the name is part of the contract. Vault keeps the user and rotates only the password on a rotation_period. Force one ahead of schedule the first time, so you find out today whether the consumer re-reads it. A rotation nothing notices is an outage with a timetable.

# rotate a static role's password now, ahead of its schedule
vault write -force database/rotate-role/app-reporting
 
# the current password, and when Vault last changed it
vault read database/static-creds/app-reporting

Every rotation needs a proof step in the same runbook paragraph that describes it. After a database rotate-root, mint a credential and run one harmless query with it. After aws/config/rotate-root, Vault returns the new access key identifier (an identifier, never the secret) so you can match it in CloudTrail, then mint a role credential and make one read-only call. Automation that rotates without verifying finds its own breakage when the payments service restarts on Monday morning.

The Ceremony Is A People Protocol

Five shares with a threshold of three is a staffing plan, not a number. It needs a named holder per share, a deputy behind each holder, storage locations in different buildings, and a rule for the person who is on a plane. Nobody keeps two shares, no two shares live in one password manager, and no share lives in a photo on a phone. Write down who holds which share after every rekey, and treat that roster the way you treat the list of people with datacenter badges.

Split the roles during the drill as well. One person runs -init and holds the OTP, different people submit shares. That is a two-person rule for free: a stolen laptop with one share and no OTP completes nothing, and an attacker holding the OTP still needs a quorum of humans to hand over pieces. If your runbook has one engineer doing all of it because that is faster, you have a single point of compromise wearing a ceremony costume.

Put the seal type at the top of every runbook page that mentions shares, because the two ceremonies look identical and fail differently. A tired operator pasting recovery keys into vault operator unseal at 3 a.m. gets a rejection they will not understand for several minutes, and those minutes are expensive. One line above the fold does it: "seal=awskms → recovery keys + generate-root; seal=shamir → unseal keys + generate-root".

Rehearse quarterly, in a maintenance window, with the real holders and the real envelopes, and put a clock on it. The output of a drill is not the token. It is a list: who answered within ten minutes, who could not find their envelope, which page of the runbook was wrong, how long the whole thing took from decision to working token. A break-glass plan nobody has executed is a document, and the gap between the document and reality gets found either by your drill or by an incident.

Most reaches for root are avoidable, and the drill stays rare if you build the alternatives first. An admin locked out of one mount, or a bad policy push that denied the CI role everywhere, needs a narrow recovery policy held by a small group, not the axe. Keep that list of non-root options on the same page as the ceremony, so the first question in an incident is whether this can be fixed without root rather than who has envelope three.

Make The Glass Audible

A ceremony nobody can see is a back door with good manners. Every step in this lesson leaves an audit path, and they all deserve alerts: sys/generate-root/attempt and sys/generate-root/update for the drill, sys/rekey/init and sys/rekey/update for Shamir shares, sys/rekey-recovery-key/init for recovery shares under auto-unseal, sys/rotate for the keyring, and anything under a rotate-root path for engine credentials. Alert on the first request of an attempt rather than its completion, because the attempt is the earliest warning you get and a cancelled attempt is still worth a phone call.

sudo jq -r 'select(.type == "request")
            | select(.request.path | test("^sys/(generate-root|rekey|rekey-recovery-key|rotate)|/rotate-root/|^auth/token/revoke-self"))
            | [.time, .request.path, (.auth.display_name // "-"), (.error // "-")] | @tsv' \
  /var/log/vault/audit.log

2026-07-24T01:11:58.204Z  sys/rotate                         cron-platform    -
2026-07-27T02:14:06.771Z  sys/generate-root/attempt          -                -
2026-07-27T02:14:41.019Z  sys/generate-root/update           -                -
2026-07-27T02:19:52.663Z  auth/token/revoke-self             root             -
2026-07-27T09:02:33.480Z  database/rotate-root/app-postgres  oidc-sre-oncall  -

Read those lines carefully. The generate-root entries carry no identity, because that endpoint is unauthenticated by design: it is the path you use precisely when nobody has a working token. The log tells you an attempt happened and when, not who ran it, and your ceremony record supplies the rest. The last line of a healthy drill is auth/token/revoke-self, and its absence is the alert worth building: a completed generate-root with no revoke behind it inside the hour means a live root token is loose somewhere.

Try This

On a disposable Vault, run the whole drill end to end. Mint a root token with the real ceremony, use it for one harmless read, revoke it, then rotate the keyring and check that the term moved. Time yourself. If the first attempt takes forty minutes because you were reading flag documentation, that number is exactly what the drill exists to find.

vault operator generate-root -init
# submit shares with -nonce until complete, then decode with OTP
vault token lookup
vault token revoke -self
vault operator rotate
vault read sys/key-status
vault operator generate-root -cancel   # if you aborted mid-drill

Nonce ... OTP ... Progress 3/3 Complete true
Root token: hvs.CAESIJ...
Success! Revoked token
Key              Value
term             8
install_time     2026-07-24T02:01:00Z
encryption_count 12
# SUCCESS — drill complete, no root left at rest, keyring advanced

Then run the exercise that scares people, on a lab cluster with three Shamir shares and nothing real in it. Rekey with verification turned on, seal the node, and unseal it with the new set. Feed it one old share first and watch where the rejection lands: not on the old share, but at the moment the threshold is reached and the rebuilt key fails to open the barrier, with the unseal progress counter dropping back to zero. Seeing that once in a lab is how it stops being a surprise at 3 a.m.

vault operator rekey -init -key-shares=3 -key-threshold=2 -require-verification=true
# submit two CURRENT shares with -nonce=<nonce> to receive the new set
vault operator rekey -verify -nonce=<verification-nonce>   # submit the NEW shares
 
vault operator seal
vault operator unseal      # paste one OLD share first
vault status               # Unseal Progress, then watch it reset
vault operator unseal      # now the new shares, until Sealed false

Takeaway

Keep zero root tokens at rest, and prove it with the accessor list rather than believing it. Cron vault operator rotate, calendar vault operator rekey with named humans and -require-verification=true, and never confuse the two. Give every secrets engine a database or cloud account that nothing else uses, rotate it on a schedule, and mint one credential afterwards to prove the engine still works.

Next comes the other half of recovery: Raft snapshots that restore cleanly, upgrades that hold quorum, and the drill that turns a backup file into something you trust.

Related