Dynamic secrets for databases and cloud
Credentials that expire faster than attackers can use them.
A database password sits in a config map, gets copied into a runbook, and turns up two years later in a .env file on a contractor's laptop. It still works. Nothing about it expired, because passwords do not expire on their own. Somebody has to decide to change one, then tell every service, every batch job and every developer at the same moment, and be right about all of them.
A static database password behaves like a hotel key card that never stops working. You checked out in March, the front desk moved on, the card still opens room 412 tonight. Vault's dynamic secrets replace that with a key cut at the desk while you wait, stamped to stop working at eleven tomorrow, with your name printed on it. Your application asks Vault at startup. Vault creates a brand new database account on the spot, hands back the username and password it generated, and writes down when it intends to delete that account.
A stolen dynamic credential is still a stolen credential, and it works right now. What it stops being is permanent. The countdown started before anyone noticed, and one command can kill every credential a role ever issued. Two things decide whether that promise is real or decorative: a short TTL (time to live, how long a credential is allowed to exist) and revocation logic that actually runs. Most teams get the first one right and lose quietly on the second.
Vault holds the account that makes accounts
Vault does not store your application's database password anywhere, which surprises people who arrive expecting a password manager. It stores something sharper: one privileged account on the database that nothing else uses. The front desk does not keep a drawer of spare keys, it keeps the machine that cuts them. Two objects express that. A connection tells Vault how to reach the database and which login to use as itself. A role holds the SQL (structured query language, the language databases speak) that Vault runs to create a throwaway user, plus the SQL it runs later to delete that user again.
{{name}}, {{password}} and {{expiration}} are placeholders Vault fills in per request, and you never choose any of the three. allowed_roles is the fence between one database and the next: a role can only attach to a connection that names it, so a mistake in one team's role config cannot reach the warehouse cluster next door. Keep that list explicit and short. Give the admin account to Vault alone as well, because the day you rotate it (its own lesson later in this course) every other consumer of that login breaks with no warning at all.
Write revocation_statements yourself rather than accepting the default. The PostgreSQL plugin ships a bare fallback that revokes privileges in the public schema and drops the role, so anything outside that shape (objects the role owns, grants in another schema, sessions still open) leaves an expired account alive in your database unless you spell out the teardown.
# enable the database secrets enginevault secrets enable database# tell Vault how to reach Postgres, using ITS OWN admin login (not the app's)vault write database/config/app-postgres \plugin_name=postgresql-database-plugin \allowed_roles="app-readwrite" \connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/appdb?sslmode=require" \username="vault_admin" \password="$VAULT_DB_BOOTSTRAP_PW"# a role = the SQL Vault runs to create and later drop a throwaway uservault write database/roles/app-readwrite \db_name=app-postgres \creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \default_ttl="1h" \max_ttl="24h"# the app calls this at boot to get a unique 1-hour uservault read database/creds/app-readwrite
Key Value--- -----lease_id database/creds/app-readwrite/abcd1234lease_duration 1hlease_renewable truepassword A1b2C3d4-generatedusername v-root-app-readw-xYZq-1234567890# SUCCESS: unique user, stamped expiry, lease_id for renew/revoke
Read that username slowly, because it is telling you things. v-root-app-readw-xYZq-1234567890 is assembled from a fixed prefix, the display name of the token that asked (here, root), the role name clipped short, random characters, and a unix timestamp. The display name is free attribution: when your DBA (database administrator, the person who owns the database) asks who opened four hundred connections at 3 a.m., the username answers without anybody grepping anything. It also means a credential minted with the root token wears root in its name for as long as it exists. A root token is a bootstrap credential for standing Vault up and nothing else. Revoke it once your auth methods and policies exist, and let real workload identities mint these accounts.
The other half of that response is the lease. lease_duration 1h is when Vault plans to delete the account, lease_renewable true says the deadline can be pushed, and lease_id is the handle for everything you do afterwards. Renew, revoke, look up, audit: all of it hangs off that string. Log the lease id in your application at startup. During an incident it is the difference between killing one credential and guessing.
The username Vault builds may not fit your database
Every database caps how long a user name can be, the same way a paper form caps how many boxes your surname gets. PostgreSQL cuts identifiers at 63 bytes. MySQL 8 allows 32 characters, and older MySQL allowed 16, which is why Vault ships a separate mysql-legacy-database-plugin for those servers. Oracle before 12.2 stopped at 30.
Now count what Vault packs into a generated name: prefix, token display name, role name, random characters, timestamp. A Kubernetes service account called payments-reconciliation-worker reading a role called analytics-reporting-readonly overruns every limit on that list. On MySQL and Oracle the CREATE statement fails and your application gets an error at boot instead of a credential, usually on the Friday you added the service. On PostgreSQL the name is truncated to 63 bytes without complaint, and once the random suffix is the part being cut, collisions between two workloads stop being theoretical.
username_template on the connection is the fix. The template language gives you truncate, random, unix_time, replace and lowercase, so you decide what gets sacrificed when the character budget runs out rather than letting the database decide for you.
# writing the config replaces it, so every field goes back invault write database/config/app-postgres \plugin_name=postgresql-database-plugin \allowed_roles="app-readwrite" \connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/appdb?sslmode=require" \username="vault_admin" \password="$VAULT_DB_BOOTSTRAP_PW" \max_open_connections=8 \username_template="{{ printf \"v_%s_%s_%s_%s\" (.DisplayName | truncate 8) (.RoleName | truncate 8) (random 20) (unix_time) | truncate 63 | replace \"-\" \"_\" | lowercase }}"# prove the shape changed before any application depends on itvault read database/creds/app-readwrite
Two details in that command matter as much as the template. First, the write replaces the whole connection object, password included, so decide your template before you hand the admin account over to Vault for rotation. Once Vault has rotated that password, nobody knows it, and editing the connection turns into a manual reset on the database at an hour nobody enjoys. Second, max_open_connections is Vault's own pool to the database and it defaults to four. Four sockets is plenty for a trickle of mints and a visible stall when three hundred pods restart together after a deploy.
Three ceilings sit above every TTL and the lowest one wins
A role's default_ttl is what a credential gets when the caller asks for nothing special. max_ttl is the hard stop where renewal stops working and the account dies however healthy the service is. Above the role sits the mount's own tuning, and above that the server setting, which defaults to 768 hours (32 days). A parking meter that takes coins up to two hours does not care what the sign on the wall promises. The smallest ceiling applies, and nothing tells you which one clipped your request.
# what does this mount inherit right now?vault read sys/mounts/database/tune# a ceiling nothing under this mount can exceedvault secrets tune -max-lease-ttl=24h database
A 0s in that output does not mean unlimited. It means the mount is inheriting the server default, which is the 32 days above. Set the mount ceiling to the longest credential life you would defend out loud in an incident review, then keep role TTLs well underneath it. Requests for anything longer come back clipped instead of honoured, which is the behaviour you want when somebody copies a role definition from a wiki page written in 2021.
Renewal does not mint a new user, and getting that wrong causes real outages. Vault extends the lease and, on PostgreSQL, runs an ALTER ROLE that pushes VALID UNTIL forward. The plugin ships that statement by default, so it works without you writing a line. Override renew_statements with something that does not move the expiry and you get the meanest failure on this page: Vault reports a healthy renewed lease, the database refuses the login at the original timestamp, and no Vault command anywhere explains the outage you are staring at.
That timestamp is the backstop that makes the whole design trustworthy. If Vault crashes, if the revocation SQL errors, if somebody deletes the lease record by hand, the database still refuses the login once VALID UNTIL passes. Keep it in every creation statement you write. A dynamic user without it is a static password wearing a random name.
Cloud credentials, and the revoke that does not revoke
The same shape maps onto cloud providers. Instead of a long-lived access key pasted into a CI (continuous integration, the robot that builds and tests your code on every commit) variable, Vault mints a key scoped to one policy and expires it. Configure the engine, define a role, read from creds/. GCP and Azure follow the identical config-plus-role pattern, so learning the workflow once carries across all three.
# enable and give Vault a set of AWS creds it can mint children fromvault secrets enable awsvault write aws/config/root \access_key=$AWS_VAULT_ACCESS_KEY \secret_key=$AWS_VAULT_SECRET_KEY \region=us-east-1# a role scoped to exactly the permissions the workload needsvault write aws/roles/s3-reader \credential_type=iam_user \policy_document=-<<'EOF'{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["s3:GetObject", "s3:ListBucket"],"Resource": ["arn:aws:s3:::app-uploads", "arn:aws:s3:::app-uploads/*"]}]}EOF# mint a short-lived AWS key pairvault read aws/creds/s3-reader
Key Value--- -----lease_id aws/creds/s3-reader/efgh5678lease_duration 1hlease_renewable trueaccess_key AKIA...TEMPsecret_key wJal...generatedsecurity_token <none for iam_user>
credential_type=iam_user creates a real IAM (identity and access management, the AWS system of users, roles and permissions) user for each request, attaches that inline policy, and deletes the user when the lease ends. Two things bite. AWS is eventually consistent, so a freshly minted key can answer with an authentication error for several seconds and your client needs a retry loop that expects it. And an AWS account allows 5,000 IAM users in total. A pipeline minting one per job reaches that wall far sooner than anyone plans for, and the error it throws does not mention Vault.
For anything high volume, assumed_role calls STS (security token service, the AWS service that hands out temporary session credentials) instead of creating users. Nothing is left behind in the account, the credentials work immediately, and a session cannot outlive the maximum session duration configured on the AWS role itself.
# no IAM user is created; AWS mints a session and expires it on its ownvault write aws/roles/s3-reader-sts \credential_type=assumed_role \role_arns=arn:aws:iam::111122223333:role/vault-s3-reader \default_sts_ttl=900 \max_sts_ttl=3600vault read aws/creds/s3-reader-sts ttl=900
Key Value--- -----lease_id aws/creds/s3-reader-sts/ijkl9012lease_duration 15mlease_renewable falseaccess_key ASIA...secret_key ...security_token ...# lease_renewable false: an STS session is minted once and cannot be extended
Now the part that belongs on your runbook in bold. Revoking that lease does not revoke those credentials. Vault deletes its own record and AWS keeps honouring the session token until it expires, because nothing in the STS design lets a third party call a live session back. With iam_user, revoke genuinely cuts access, since deleting the user deletes the key. So the two types trade against each other: STS wins on churn and leaves no litter in your account, iam_user wins on immediate containment. If you run STS roles, learn the AWS-side kill switch as well, which is attaching a deny policy to the role that refuses any session issued before a given time. Practising that once is worth more than reading about it twice.
iam_user roles need a reconciliation job on top. A failed delete, a Vault outage or a lease that went irrevocable leaves IAM users sitting in the account with your inline policy still attached, and the thing that finds them is a compliance scanner nine months later filing them as unexplained standing access. Compare the users Vault created against its live leases on a schedule and alert when the two counts disagree.
Leases are rows in storage, and they can drown a cluster
A lease is a coat check ticket and Vault keeps the stub. Each one is a record written into Raft, the replicated log Vault uses for its own storage, and it costs a write to create, another to renew, another to clear. Individually cheap, which is exactly why the failure creeps up on people.
Do the arithmetic on a service that mints a credential per HTTP request instead of once at startup. Fifty requests a second with a 24 hour TTL is more than four million live leases by this time tomorrow: four million records in Raft, and four million roles in PostgreSQL, which it will not enjoy. Nothing falls over at a clean threshold. Snapshots get slower, memory climbs, the expiration manager spends its life catching up, and the eventual page says something vague about latency.
Watch two numbers. The telemetry gauge vault.expire.num_leases is the count Vault itself keeps, and it belongs on the same dashboard as your Raft disk usage. The list endpoint tells you which role is responsible when the graph bends.
# which leases exist under one rolevault list sys/leases/lookup/database/creds/app-readwrite# leases Vault has given up trying to revokevault read sys/leases/count type=irrevocable
Keys----abcd1234efgh5678
That second command exists because revocation can fail permanently. When the revocation SQL errors, Vault retries with a growing backoff, and after six attempts it stops trying and marks the lease irrevocable. The lease then sits in storage indefinitely. The database account it was supposed to delete is still there, still privileged, still able to log in until VALID UNTIL rescues you. Nothing pages anybody. Query that count on a schedule and treat a non-zero answer as a ticket with a name on it, not a curiosity.
DROP ROLE while the account still owns an object or holds a grant somewhere, so if your creation statement ever runs a CREATE TABLE, or the application leaves a transaction open, the revoke errors out and Vault is left holding a lease for an account that is alive and over-privileged. Catch it by alerting on revocation failures in the Vault server log and by diffing pg_roles against Vault's active leases on a schedule. The fix is a fuller teardown that reassigns and drops whatever the role owns and terminates its live backends before the role goes: REASSIGN OWNED BY "{{name}}" TO vault_admin; DROP OWNED BY "{{name}}"; SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = '{{name}}'; DROP ROLE IF EXISTS "{{name}}";. Test revocation deliberately rather than assuming the happy path cleaned up after itself.Once you have cleaned the database by hand, vault lease revoke -force -prefix database/creds/app-readwrite clears Vault's records without running any revocation SQL at all. It needs sudo capability on the path, and the order is not negotiable: force first and you have thrown away the only list of accounts you were about to go hunting for.
One more thing about time. Leases do not expire while Vault is sealed, because nothing is running to expire them. Unsealing is not a login, it is the step that reconstructs the key which decrypts Vault's own storage so the process can read its data again. Once it can read, the expiration manager loads every lease and revokes everything whose clock ran out during the outage. A four hour incident therefore ends with a burst of DROP ROLE statements landing on your database in one go. Work out how large that burst is on your busiest mount before an outage works it out for you.
# inspect a specific lease's remaining TTLvault write sys/leases/lookup lease_id=database/creds/app-readwrite/<lease_id># extend it (capped at the role's max_ttl)vault lease renew database/creds/app-readwrite/<lease_id># kill one credential immediatelyvault lease revoke database/creds/app-readwrite/<lease_id># incident response: nuke EVERY credential from a role at oncevault lease revoke -prefix aws/creds/s3-reader
Key Value--- -----lease_id database/creds/app-readwrite/abcd1234lease_duration 59m42srenewable trueSuccess! Revoked lease: database/creds/app-readwrite/abcd1234# prefix revoke returns success per lease; watch audit log for failures
Prefix revoke is the crowbar. One call burns every live credential a role ever handed out, which is the containment story a shared password can never tell: no coordinated reset, no chasing down which services hold a copy. Rehearse it in staging with a stopwatch. How many seconds until connections start failing, how long until pods are ready again on fresh credentials, and who is allowed to make that call at 2 a.m. without waking a director first.
Your application is the part that breaks
A connection pool opens ten sockets at startup using the username Vault handed over. At minute sixty one the account is gone. Sockets that are already open often keep working until the database notices, so the failure does not arrive when the lease expires. It arrives when the pool decides to open socket eleven, which is during a traffic spike, which is the worst possible moment to learn how your pool handles a vanished user.
Renewal has to be somebody's job, then. A long-running service renews at around two thirds of the TTL, well clear of the edge, and rebuilds its pool when the username changes. A batch job does not bother and lets the credential lapse when it exits. The tidiest arrangement keeps Vault out of your application code entirely and lets Vault Agent write credentials to a file and signal the process, which has its own lesson later in this course. Whichever route you take, the requirement is identical: your service has to survive its own database username changing while it is running.
The failure this causes is more political than technical. A team that cannot rebuild a pool raises default_ttl to 720 hours, declares the migration done, and now runs static passwords with a Vault-shaped invoice attached. If your framework cannot swap a database user without dropping traffic, fix that before the rollout rather than after the first incident review.
Minting is gated by policy, and a dynamic role without a tight one is a self-service admin panel. Read on database/creds/app-readwrite is the entire control. Bind that exact path to the Kubernetes service account or OIDC group that owns the database, and resist granting a wildcard over database/creds/ because it is convenient during the pilot. A compromised recommendations service should not be able to mint payments credentials on the grounds that both engines happen to share a mount.
Cut over with an overlap. Keep the static password working while a canary pod proves it can mint, connect, renew and survive a forced revoke on real traffic. Delete the static password only once that canary has restarted cleanly on dynamic credentials at least once. Migrations that delete it on day one produce an outage that gets blamed on Vault rather than on the plan.
Three numbers say whether any of this landed: how many static database passwords still live in config maps, the median TTL of the roles your services actually call, and the seconds between running revoke and watching a login fail during a drill. Track those and the conversation with leadership stops being about architecture diagrams.
Try this
Ten minutes against a throwaway PostgreSQL and a lab Vault beats rereading this page. Mint a credential, connect with it, revoke it, and watch the login die.
vault read database/creds/app-readwrite# note username + lease_id, then:psql "postgresql://$USER:$PASS@postgres.internal:5432/appdb?sslmode=require" -c 'select current_user'vault lease revoke database/creds/app-readwrite/$LEASE_IDpsql "postgresql://$USER:$PASS@postgres.internal:5432/appdb?sslmode=require" -c 'select 1'
username v-root-app-readw-xYZq-1234567890lease_id database/creds/app-readwrite/abcd1234current_user-----------------------------v-root-app-readw-xYZq-1234567890Success! Revoked lease: database/creds/app-readwrite/abcd1234psql: error: FATAL: password authentication failed for user "v-root-..."# FAIL after revoke, exactly what you want in an incident
Now break it deliberately, because the happy path teaches you nothing about revocation. Mint a second credential, log in as that user, create a table so the role owns an object, and then revoke the lease.
vault read database/creds/app-readwritepsql "postgresql://$USER:$PASS@postgres.internal:5432/appdb?sslmode=require" -c 'CREATE TABLE scratch_owned (id int)'vault lease revoke database/creds/app-readwrite/$LEASE_ID# Vault side: did the lease actually go?vault list sys/leases/lookup/database/creds/app-readwrite# database side: did the account actually go?psql "$ADMIN_URL" -c "SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname LIKE 'v-%'"
The revoke returns an error, the role is still listed in pg_roles, and the credential still logs in. That gap between what Vault believes and what your database is doing is the whole reason the callout above exists. Replace the role's revocation statement with the fuller teardown, mint a fresh credential, repeat the same three commands, and keep going until the account disappears from pg_roles on the first attempt. Then check sys/leases/count type=irrevocable and confirm it is back to zero.
Takeaway
Dynamic secrets swap a shared forever-password for a per-request account with a stub attached to it. Minting is the easy half. The lesson lives in the other half: revocation SQL that survives a user owning objects, a VALID UNTIL the database enforces without Vault's help, TTL ceilings you set on purpose instead of inheriting, and a lease count somebody actually looks at. Creation statements with weak revocation build the same trap you had before, with better paperwork.
Certificates get the same treatment next. Vault becomes an internal certificate authority handing out leaves that live a single day, so a stolen certificate rots on its own before anyone has to file a ticket about it.