PKI and short-lived certificates
An internal CA with 24-hour leaf certs.
The wildcard certificate on your load balancer expires at 03:14 on a Saturday. Nothing pages you, because what expired is not a process, it is a date. What pages you is a flood of customers saying the mobile app will not log in, and forty minutes go by before anyone thinks to point openssl at the listener. There is a quieter version of the same story. The certificate does not expire at all. It sits on nineteen nodes for two years, and the copy an attacker lifted from a stale machine image last spring still terminates traffic that every client accepts as genuine.
A building that takes security seriously does not cut you a permanent key to the front door. Reception prints a day pass with your name and today's date on it, and at midnight that pass is a rectangle of plastic. Vault's PKI secrets engine (public key infrastructure, the machinery that hands out TLS certificates and vouches for them) is the badge office for your machines. Instead of a person generating a certificate once and pasting it into config, Vault signs a fresh certificate the moment a workload asks, with a life measured in hours.
Both failure modes die together. Expiry stops being a calendar entry somebody forgets and becomes something that happens every single day, in automation you have already watched work a thousand times. Theft stops being permanent. A stolen certificate with twenty hours left on it is a problem that solves itself before you finish writing the incident doc. That second property is the bet you already made with dynamic database credentials, moved from identity at the database to identity on the wire.
By the end of this lesson you will have a two-tier certificate authority inside Vault, a role that flatly refuses to sign anything outside one domain, a leaf certificate that dies in a day, and an answer to the question that ambushes every first PKI rollout: what happens to all the certificates Vault keeps writing into storage.
What a certificate actually promises
A TLS (transport layer security, the encryption behind every https:// address) certificate is a signed claim and nothing else. It says: whoever holds the matching private key is entitled to call itself api.svc.example.com, and here is my signature to prove I checked. The "I" is a certificate authority, a CA. A client accepts the claim because it recognises the signature, and it recognises the signature because the CA's own certificate already sits in a trust store on that machine. No magic, no lookup, no phone call to anyone.
Trust runs downhill in a chain. A root CA signs an intermediate CA. The intermediate signs the leaf certificate your web server actually presents. A client that trusts the root will walk that chain down and accept the leaf, provided the server hands over the intermediate as well. Forgetting to send the intermediate is the single most common reason a certificate that looks perfect under openssl fails in a browser, and the error text rarely says so. The expiry date, meanwhile, is the only part of the whole arrangement that enforces itself without a human, which is why short lifetimes buy so much safety for so little work.
Vault does not replace the mathematics. It replaces the people. The CA's private key is generated inside a mount and never leaves it, which is what the generate/internal endpoint means; there is an exported variant that hands you the key along with the problem of keeping it safe, and you almost never want it. Issuance turns into an API call bound by policy, and every issued certificate lands in the audit log next to the identity that asked. During an incident, "which service minted a cert for payments.svc last Tuesday" becomes a search instead of a guess.
Build a two-tier CA: offline root, online intermediate
Never let Vault sign leaf certificates straight from the root. The root is the thing every laptop, container image, phone and partner system has been told to trust. You want it created once, its private key never leaving a controlled mount, and then left alone for a decade. Everything routine runs through an intermediate CA that the root signed on day one.
The reason is recovery. If the intermediate is compromised, you revoke one certificate, sign a new intermediate from the root, and roll it out. Nothing in any trust store on earth has to change, because the root is unchanged. If the root is compromised, you are re-seeding trust on every machine, image and device your company owns, by hand, under time pressure, while people ask when the outage ends. Two tiers convert a catastrophe into a bad week.
Before you generate anything, tune each mount's max_lease_ttl (time to live, how long a thing is allowed to be valid). It is a hard ceiling that silently caps everything issued beneath it, and silence is the part that costs you an afternoon. A role asking for 72h on a mount tuned to 24h receives 24h with no warning and no error. Set the ceilings on purpose: long for the root, shorter for the intermediate, shortest at the role.
# Root CA — long-lived, then leave it alonevault secrets enable -path=pki_root pkivault secrets tune -max-lease-ttl=87600h pki_root # 10y ceilingvault write -field=certificate pki_root/root/generate/internal \common_name="Example Corp Root CA" issuer_name="root-2026" ttl=87600h > root_ca.crt# Intermediate CA — the workhorse Vault signs leaves fromvault secrets enable -path=pki_int pkivault secrets tune -max-lease-ttl=43800h pki_int # 5y ceiling# 1. intermediate generates a key + CSR (private key stays in Vault)vault write -format=json pki_int/intermediate/generate/internal \common_name="Example Corp Intermediate CA" \| jq -r '.data.csr' > int.csr# 2. root signs the CSRvault write -format=json pki_root/root/sign-intermediate \issuer_ref="root-2026" csr=@int.csr format=pem_bundle ttl=43800h \| jq -r '.data.certificate' > int.crt# 3. hand the signed cert back — creates the intermediate issuervault write pki_int/intermediate/set-signed certificate=@int.crt# 4. name the freshly-imported issuervault write pki_int/issuer/default issuer_name="int-2026"
Key Value--- -----issuer_id <uuid>issuer_name int-2026certificate -----BEGIN CERTIFICATE-----...---# root_ca.crt is distributed to trust stores; day-to-day issuance uses pki_int only
Walk those four steps once and the shape sticks. The intermediate mount generates its own key pair and emits a CSR (certificate signing request, a formal "please vouch for this public key" document); the private half stays inside pki_int and is never printed, copied or backed up separately. The root signs that CSR and returns a certificate. You post the signed certificate back into pki_int, which is the moment the mount becomes a working issuer rather than a mount holding an orphan key. Then you name the issuer, because a year from now, mid-rotation, you will have two issuers on that mount and issuer_id UUIDs are unreadable while "int-2026" is not.
About that word "offline". A root CA in a Vault mount is not offline in the way an auditor means it. What you can do is make it inert: no roles on pki_root, a policy that denies every path under pki_root to every day-to-day identity, and an audit alert on any write to it. Higher-assurance shops generate the root on an air-gapped machine or a hardware security module and import only the signed intermediate. Either way, the root token you used to bootstrap this setup is a temporary credential, not an operating account. Revoke it once the mounts, roles and policies are in place, and use identity-based logins from then on.
Define a role and issue a 24-hour leaf
A role is what turns "Vault can sign anything anyone asks for" into "Vault will sign a certificate for api.svc.example.com, RSA-2048, valid for exactly one day, and refuse everything else." It pins the allowed domains, whether subdomains count, the key algorithm and size, and both ttl and max_ttl. Set the two TTLs to the same value and a caller cannot talk its way into a longer certificate by passing its own ttl, because max_ttl is the ceiling the role enforces regardless of what the request asks for.
Point config/urls at your Vault address in the same breath. Those two fields get baked into every leaf as the AIA (authority information access, a pointer to where the issuing certificate can be downloaded) and CRL (certificate revocation list, the published list of certificates the CA has disowned) locations. Clients that need to fetch a missing chain link or check revocation read them out of the certificate itself. They cannot be added afterwards. A certificate issued before you set config/urls carries empty pointers forever, and the only fix is issuing a new one.
# Publish issuer + CRL locations (embedded into every leaf)vault write pki_int/config/urls \issuing_certificates="https://vault.example.com/v1/pki_int/ca" \crl_distribution_points="https://vault.example.com/v1/pki_int/crl"# The role: caps domains, key type, and lifetimevault write pki_int/roles/web-24h \allowed_domains="svc.example.com" \allow_subdomains=true \key_type=rsa key_bits=2048 \ttl=24h max_ttl=24h# A workload requests a leaf — cert + key + issuing CA come back inlinevault write pki_int/issue/web-24h common_name="api.svc.example.com" ttl=24h
Key Value--- -----lease_id pki_int/issue/web-24h/<id>lease_duration 24hcertificate -----BEGIN CERTIFICATE-----...private_key -----BEGIN RSA PRIVATE KEY-----...issuing_ca -----BEGIN CERTIFICATE-----...ca_chain [intermediate, ...]serial_number 11:22:33:...# SUCCESS — 24h leaf; private key shown once, treat like any secret
Read that response field by field, because four of them matter later. private_key is displayed exactly once and Vault will never show it again; if you lose it, the certificate is decoration. serial_number is the handle you need to revoke this specific certificate, so anything that stores issuance events should keep it. issuing_ca is the intermediate, and ca_chain is every link above the leaf, which is what your server needs to send alongside the certificate so clients can build a path to the root. lease_duration mirrors the certificate's own validity window, which is Vault's way of saying it expects this to be replaced, not renewed.
Three role knobs cause most of the surprises. allow_subdomains permits anything under svc.example.com but not the bare name itself, which needs allow_bare_domains. allow_glob_domains accepts wildcard patterns in allowed_domains and quietly widens what the role will sign, so read it twice before enabling it. And allow_any_name does what its name says, which makes the role worthless as a boundary; if you find it set to true in an existing deployment, treat that as a finding rather than a preference.
Key algorithm is worth a second thought too. RSA-2048 is the safe default because every client on the planet handles it. Elliptic curve keys are smaller and much faster to generate, which starts to matter when a mesh sidecar mints a fresh key every hour across two thousand pods. A second role gives you that without touching the one your legacy clients depend on.
# One hour, EC keys, and Vault keeps no record of what it issuedvault write pki_int/roles/mesh-1h \allowed_domains="svc.example.com" \allow_subdomains=true \key_type=ec key_bits=256 \no_store=true \ttl=1h max_ttl=1h
Where all those certificates pile up
That no_store=true on the second role is not decoration, and it deserves the paragraph before the callout rather than after it. By default, every certificate Vault issues is written into storage so the engine can revoke it later and build a CRL that lists it. At a hundred certificates a week nobody notices. At a hundred thousand a week, on Raft (Vault's built-in replicated storage), you have quietly built a write-heavy database whose only purpose is to remember credentials that died yesterday.
For roles where storage is the right choice, tidy is the housekeeping job that keeps the mount from growing forever. It deletes expired entries and revoked entries older than a safety buffer you pick, and it runs in the background rather than blocking. Run it on a schedule from day one instead of discovering it during a capacity incident. Counting what the mount holds before and after tells you whether it did anything.
# every stored cert is one entry in Raftvault list -format=json pki_int/certs | jq 'length'# sweep expired certs and long-revoked entries, keeping a 72h marginvault write pki_int/tidy \tidy_cert_store=true \tidy_revoked_certs=true \safety_buffer=72h# tidy runs in the background, so ask how it wentvault read pki_int/tidy-status
14231# 14231 stored certs on a mount issuing 24h leaves is about six months of neglect
Revocation itself is one call, and it needs the serial number from the issue response. Revoking marks the certificate on the CRL; it does not reach out and delete anything from the machine holding it. Clients only stop trusting the certificate once they fetch the updated CRL, which many of them do lazily, some do never, and almost none do fast enough to help you during an active compromise. That gap is exactly why short lifetimes are the stronger control and revocation is the backstop.
vault write pki_int/revoke serial_number="11:22:33:..."
Key Value--- -----revocation_time 1785283200# the cert is now on the CRL; clients honour that only after they refetch it
Renewal is the entire point
Short-lived certificates are only workable if something replaces them without a human in the loop. That something is Vault Agent, running as a sidecar container or a systemd unit next to your service. It logs in with the workload's own platform identity, the Kubernetes or OIDC methods from earlier in this course, asks the PKI role for a fresh leaf on a schedule, writes it where the server already looks, and signals the server to reload. The application never learns a Vault token exists and never holds a certificate older than a day.
template {source = "/etc/vault.d/templates/bundle.pem.ctmpl"destination = "/etc/ssl/private/bundle.pem"perms = "0640"exec {command = ["systemctl", "reload", "haproxy"]}}
{{ with secret "pki_int/issue/web-24h" "common_name=api.svc.example.com" "ttl=24h" }}{{ .Data.certificate }}{{ .Data.issuing_ca }}{{ .Data.private_key }}{{ end }}
One detail in that template will bite you if you skip it. Because extra arguments turn secret into a write, every evaluation of that block asks Vault for a brand new certificate and a brand new key. Render the certificate from one template file and the key from another and you get two unrelated issuances: the key in the second file does not match the certificate in the first, and your server fails to start with a message about a key mismatch that sends people hunting for a corrupt file. Render all three parts inside a single with block, as above. Servers that want separate files (nginx is the usual one) need a split step in the exec command, not a second template.
Test the reload path under load before you trust it, because renewal has two halves and only one of them is visible. Agent can write a perfect new certificate to disk while the running process keeps serving the old one out of memory, and the file on disk looks so healthy that nobody suspects it. You find out when the in-memory certificate expires, on a schedule that looked handled. The on-call check is short and worth writing down: read notAfter from the live listener rather than from the file, then check Agent's log for its last render, then check the Vault audit log for denied issue requests.
What actually breaks in production
Trust distribution is half of PKI and gets a tenth of the attention. Every client, every load balancer, every partner integration has to carry the root certificate before a single short-lived leaf helps anybody. Plan how root material reaches base container images, machine images, JVM and Python trust stores, mobile apps and the one Windows box nobody documented. Automate trust-bundle updates the way you automate leaf renewal. A leaf that chains to an issuer the client has never heard of is an outage with extra steps.
Verification is cheap, so do it in two places. Check that the leaf builds a path to your root through the intermediate, and separately check what the running listener is serving right now, because those two answers disagree more often than you would like.
# does the leaf chain to your root through the intermediate?openssl verify -CAfile root_ca.crt -untrusted int.crt leaf.crt# what is the running listener actually handing out?echo | openssl s_client -connect api.svc.example.com:443 -servername api.svc.example.com 2>/dev/null \| openssl x509 -noout -subject -issuer -dates
leaf.crt: OKsubject=CN = api.svc.example.comissuer=CN = Example Corp Intermediate CAnotBefore=Jul 27 09:14:02 2026 GMTnotAfter=Jul 28 09:14:02 2026 GMT# SUCCESS - path builds to the root and the live cert has under a day left
Clock skew stops being theoretical the moment certificates live for hours. A node whose clock is four minutes ahead will reject a certificate issued four minutes ago as not yet valid, and the error blames the certificate rather than the clock. With a two-year wildcard nobody ever noticed. With a one-hour mesh certificate, a drifting clock takes a service down and looks exactly like a PKI bug. Time synchronisation is a prerequisite for this design, not a nice-to-have, and it belongs in the same runbook.
Role sprawl in reverse is the other trap: one enormous role that can mint anything under svc.example.com because splitting it seemed like bureaucracy. Whoever compromises a CI (continuous integration) token bound to that role can impersonate every service you run, including the ones that authenticate each other by certificate. Per-team roles with tight allowed_domains turn the same theft into an impersonation of one team's names. The extra roles cost you a few lines of Terraform and buy you a smaller incident.
Split who may issue from who may manage the CA, and express the split in policy paths. A build pipeline needs create on one issue path and nothing else. It has no business rewriting a role's allowed_domains to a wildcard, generating a new intermediate, or reading anything under the root mount.
# CI can ask for a leaf under exactly one rolepath "pki_int/issue/web-24h" {capabilities = ["create", "update"]}# and cannot rewrite the role that constrains itpath "pki_int/roles/*" {capabilities = ["deny"]}# and cannot touch the root mount at allpath "pki_root/*" {capabilities = ["deny"]}
Machines should reach the issue path through Kubernetes auth or a cloud identity, never a token a person pasted somewhere. Keep human logins on a separate, rarely used break-glass role, because if any engineer can mint certificates for production hostnames from a laptop, one phishing email becomes a CA compromise. Watch issuance volume as a signal in its own right. Ten thousand leaves a day across two hundred services means a client retrying in a loop or an Agent whose renewal is failing silently, and the fix is in the client, not in more storage.
Rotating the intermediate is a planned event, so rehearse it. Sign a second intermediate from the root while the first is still valid, publish the new chain everywhere, move roles to the new issuer_ref, let a canary service prove the new path works, and revoke the old intermediate only after the overlap window closes. What you are really testing is which clients pinned a chain they should not have pinned. Discovering that list on the day the old intermediate expires is the version of this exercise that involves a bridge call.
Last, treat the leaf private key like the secret it is for the whole time it exists. Agent should write it mode 0640 onto a memory-backed volume, owned by a group the application belongs to and nobody else does. Never let it reach a ConfigMap, an image layer, a log line, or a chat message. Audit devices HMAC sensitive response fields precisely so a key never lands in the audit log in readable form. A one-day certificate whose key was pasted into a ticket is still a breach, only a shorter one.
Try this
In a lab Vault, build pki_root and pki_int from the commands above, issue one 24-hour leaf, then confirm two things with openssl: the certificate really does expire tomorrow, and the AIA pointer really did get baked in. Then ask the same role for a name it has no business signing and watch it refuse.
vault write pki_int/issue/web-24h common_name="api.svc.example.com" ttl=24h -format=json \| jq -r '.data.certificate' > leaf.crtopenssl x509 -in leaf.crt -noout -dates -ext subjectAltName,authorityInfoAccessvault write pki_int/issue/web-24h common_name="evil.example.net" ttl=24h
notBefore=Jul 24 01:00:00 2026 GMTnotAfter=Jul 25 01:00:00 2026 GMTX509v3 Subject Alternative Name: DNS:api.svc.example.comAuthority Information Access: CA Issuers - URI:https://vault.example.com/v1/pki_int/caError: 1 error occurred:* common name evil.example.net not permitted by CA# FAIL on foreign domain — role policy working
For a second round, tune pki_int down to -max-lease-ttl=12h, issue from the same unchanged 24h role, and read notAfter again. Watching a role you did not touch hand back a twelve-hour certificate teaches the mount ceiling better than any paragraph can.
Takeaway
Keep the root inert and the intermediate online, let roles decide which names and which lifetimes are allowed, and pick no_store deliberately per role rather than by default. Then hand renewal to Vault Agent so that a certificate expiring becomes the most boring event in your week.
Next you take the same expiry-first thinking to credentials that cannot be short-lived, and rehearse generate-root before an incident is the reason you need it.