CoursesVault from dev to productionAuth methods: OIDC and Kubernetes

Auth methods: OIDC and Kubernetes

Humans via SSO, workloads via service accounts.

Intermediate30 min · lesson 4 of 13

A laptop goes missing on a Tuesday. The engineer who owned it had a Vault password, and that password was the same string they used on three other sites, saved in a notes app with no lock on it. By Wednesday morning somebody on a café Wi-Fi is walking secret paths one directory at a time. Nothing in Vault was broken. The problem was that Vault had its own password database at all.

That is the whole argument for this lesson. Production Vault should not store credentials for people. It should ask somebody else who the person is, and it should ask the platform who the machine is.

Treat Vault as a secured building with two entrances. Humans walk in through the front lobby, where the badge reader is wired straight into your corporate identity provider, so there is no separate Vault password to remember or leak, only the single sign-on login your engineers already have. Workloads use the loading dock around back, where the only ID that counts is the service account token the pod was born with. Same building, two very different doors, and each needs a different lock.

Say your pipeline deploys a payments API into the Kubernetes namespace prod. The humans who debug it at 2am should get their permissions from their Okta group. The pods themselves should never hold a long-lived Vault token sitting in a Kubernetes Secret where anyone with read access on that namespace can print it. OIDC (OpenID Connect, the standard that lets one system prove your identity to another using a signed token) covers the people. Kubernetes auth covers the machines. Wire both, and disabling an employee upstream or deleting a service account becomes the revoke path you actually want, rather than a wiki page nobody follows.

Why Vault refuses to be your password database

Every system that stores its own passwords becomes a system somebody has to offboard from. You have felt this. An engineer leaves, HR closes the ticket, and eight months later somebody finds their account still live in the tool nobody remembered was in the list.

Vault sidesteps that by treating authentication as a question it forwards, not one it answers. An auth method is a plug-in whose only job is to accept some proof of identity from the outside world, verify it against a source of truth that lives elsewhere, and hand back a Vault token carrying policies. The token is the currency inside Vault. Everything before the token is somebody else's problem, by design.

This is worth separating from unsealing, because people mix them up constantly. Unsealing is Vault reconstructing the encryption key that decrypts its own storage after a restart. It has nothing to do with who you are. An unsealed Vault with no valid token is still useless to you. A sealed Vault will reject your perfectly good OIDC token because it physically cannot read the data behind it. Two different problems, two different failures, two different 3am pages.

The root token is the other thing worth naming here. It is a bootstrap credential, nothing more. You use it to enable your first auth method and write your first policies, then you revoke it with vault token revoke and you never mint another one except through a deliberate break-glass ceremony. A root token sitting in someone's shell history is the same failure mode as the stolen laptop, only with no policy limits at all.

The front door: OIDC for humans

OIDC auth lets Vault delegate the question "who are you?" to an identity provider you already trust, such as Okta, Entra ID, Google, or Keycloak. Vault never sees a password. It redirects the browser to the IdP (identity provider, the system that actually holds your staff accounts), receives a signed ID token back, and reads claims out of it. A claim is one labelled fact inside that token: an email address, a user ID, a list of groups. You bind a role to the IdP's client ID and tell Vault which claim uniquely identifies the person (user_claim) and which claim carries their group membership (groups_claim).

The handshake is worth walking through slowly, because almost every failure you will hit lives in one of these steps. You run vault login -method=oidc. The CLI opens a tiny web server on your own machine at port 8250 and pops your browser at the IdP. You do whatever your company requires, password, hardware key, push notification. The IdP redirects your browser back to that little local server with a code. The CLI hands the code to Vault. Vault exchanges it with the IdP for a signed ID token, checks the signature against the IdP's published keys, checks the audience, reads the claims, matches them to a role, and mints a Vault token. Your CLI stores it. You are in.

The allowed_redirect_uris must list both the CLI's localhost callback and your Vault UI address, exactly, including scheme, port, and path. Get one character wrong and the IdP refuses the handoff before Vault ever sees a token, which is why the error message you get looks like it came from a system you were not even configuring. Because the Vault token that comes out is short-lived, the human re-authenticates through single sign-on when it expires, so disabling their account upstream cuts off Vault on its own. That is the operational win: identity lifecycle stays in one place.

enable and configure OIDC
# Turn on the OIDC auth method
vault auth enable oidc
# Point Vault at your identity provider
vault write auth/oidc/config \
oidc_discovery_url="https://login.example.com" \
oidc_client_id="$OIDC_CLIENT_ID" \
oidc_client_secret="$OIDC_CLIENT_SECRET" \
default_role="engineer"
# A role: which claims to trust, where to redirect, what token to mint
vault write auth/oidc/role/engineer \
user_claim="sub" \
groups_claim="groups" \
oidc_scopes="openid,profile,groups" \
bound_audiences="$OIDC_CLIENT_ID" \
allowed_redirect_uris="http://localhost:8250/oidc/callback" \
allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
token_policies="engineer" \
token_ttl=1h
# Engineers now log in with:
vault login -method=oidc role=engineer
output — successful OIDC login
Complete the login via your OIDC provider. Launching browser to:
https://login.example.com/...
Success! You are now authenticated.
The token information displayed below is already stored in the token helper.
Token: hvs.CAESIJexample
Token TTL: 1h
Token Max TTL: 0s
Token Policies: ["default" "engineer"]
token_accessor: ...

Notice oidc_discovery_url. Vault does not need you to type in the IdP's token endpoint, key endpoint, and issuer separately. It fetches a well-known discovery document from that base URL and learns them. Which means two things in practice. First, Vault must be able to reach your IdP over the network, so if Vault lives in a private subnet with no egress, the login fails in a way that looks like a configuration typo but is actually a firewall rule. Second, when your IdP rotates its signing keys, Vault picks up the new ones without you doing anything, because it re-fetches the key set. That is one fewer expiry to track.

Reading the claims before you guess at them

Here is the mistake that eats an afternoon. You look at your Okta admin console, see a group called "Platform Engineering", and write that string into your Vault config. Then every login succeeds and every user comes back with only the default policy, no error anywhere, no clue why.

The cause is that identity providers do not always emit the friendly name. Entra ID, in its default configuration, emits group object IDs, which look like a3f9c1e2-4b7d-4e88-9c31-6d0f2a5b8e14. Okta can emit either, depending on how the claim is configured. Your alias must match the exact string that arrives in the token, not the string a human reads in a console.

So do the boring thing first. In a lab, capture one real ID token, decode the payload, and read it. Never do this against production and never let a decoded token land in a log file, because that token is a live credential for as long as it has not expired. Once you know what sub, email, and groups actually contain for your provider, the rest of the configuration is mechanical.

There is a second silent failure right next to this one. bound_audiences tells Vault to only accept tokens whose aud claim matches. If your IdP is issuing tokens for a different application, or you registered two apps and pasted the wrong client ID, Vault rejects the login even though the browser dance completed and the user saw their own IdP login screen. The user reports "SSO is broken". SSO is fine. The audience does not match. Check that before you check anything else.

Turning SSO groups into Vault policy

Granting policy to individual users does not scale, and it drifts the moment somebody changes teams. Instead, let group membership in your IdP drive Vault authorization. When a user logs in through OIDC, Vault reads the groups_claim and looks for a matching group alias. You pre-create an external identity group, attach the policies you want it to carry, and map the IdP's group name to it through a group alias keyed on the OIDC mount accessor.

The mount accessor deserves a plain explanation, since it shows up in the commands below. When you enable an auth method, Vault gives that specific mount a stable internal ID, something like auth_oidc_aabbccddee. It is how Vault tells your OIDC mount apart from a second OIDC mount you might add later for contractors. The group alias is keyed on it so that "platform-eng from our staff IdP" and "platform-eng from the contractor IdP" can never be confused with each other.

map an IdP group to a Vault policy
# Grab the accessor of the OIDC mount
ACCESSOR=$(vault auth list -format=json | jq -r '."oidc/".accessor')
# External group that carries the policy
vault write identity/group name="platform-eng" \
type="external" \
policies="platform"
GROUP_ID=$(vault read -field=id identity/group/name/platform-eng)
# Alias ties the IdP's group name to the Vault group
vault write identity/group-alias \
name="platform-eng" \
mount_accessor="$ACCESSOR" \
canonical_id="$GROUP_ID"
output — auth list accessors
Path Type Accessor Description
oidc/ oidc auth_oidc_aabbccddee n/a
kubernetes/ kubernetes auth_kubernetes_ff001122 n/a
# after group write
Key Value
--- -----
id <GROUP_ID>
name platform-eng
policies [platform]
type external

The word external on that group is the important part. An internal identity group is one whose membership you maintain by hand inside Vault, which puts you right back in the offboarding business. An external group has no member list at all. Membership is decided fresh at every login by whatever the identity provider says. Somebody added to platform-eng in Okta inherits the platform policy on their very next login. Somebody removed loses it the same way. No Vault change, no ticket, no stale grant left behind.

That also tells you the honest limit of this design. Removing somebody from a group in Okta does not kill a Vault token they are already holding. It stops the next login from succeeding with those policies. The token they have keeps its policies until its time to live runs out. For a real offboarding you revoke their existing tokens explicitly. This is exactly why token_ttl=1h on the role matters rather than being a nice-to-have: it is the maximum window between somebody losing access upstream and losing it in Vault, unless you go revoke by hand.

Keep policies on the OIDC role thin. The role's token_policies are a floor that everybody who logs in through that role receives. Identity groups stack additional policies on top at login time. Design it that way and the role definition stays stable for years while team membership churns around it. An engineer moving from payments to platform drops the old paths automatically on their next single sign-on, because the group they left stopped contributing its policy.

The loading dock: Kubernetes auth for workloads

Machines cannot do a browser redirect. Nobody is there to click. So workloads authenticate with the one credential Kubernetes already hands every pod: its projected service account token, a short-lived signed JWT (JSON Web Token, a compact signed blob of claims) that the kubelet mounts into the pod's filesystem and refreshes on its own. Vault takes that JWT, calls the cluster's TokenReview API to confirm it is genuine and unexpired, and if the pod's service account name and namespace match a role's bindings, issues a Vault token in return. There is no secret to distribute and no static credential to rotate. The identity is the pod's own.

Since Kubernetes 1.21 these tokens are time-bound and audience-scoped, so one scraped out of a running pod cannot be replayed forever or against a different audience. Bind each role narrowly to a service account name and namespace, so only the webapp pods in prod can ever assume the webapp policy, and a compromised pod in another namespace gets nothing. Wide bindings like bound_service_account_names="*" feel convenient in a demo and become a gift to an attacker in production, because now any pod anywhere that can reach Vault holds your webapp policy.

enable Kubernetes auth, bind a role, log in
vault auth enable kubernetes
# In-cluster: you supply the host; Vault defaults to its own pod's
# service account token and the local cluster CA to reach the API
vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes.default.svc:443"
# Only the 'webapp' SA in 'prod' may assume this role
vault write auth/kubernetes/role/webapp \
bound_service_account_names="webapp" \
bound_service_account_namespaces="prod" \
token_policies="webapp" \
token_ttl=1h
# From inside the pod, exchange the SA token for a Vault token
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
vault write auth/kubernetes/login role=webapp jwt="$TOKEN"
output — kubernetes login
Key Value
--- -----
token hvs.CAESIJpodtoken
token_accessor ...
token_duration 1h
token_renewable true
token_policies ["default" "webapp"]
token_meta_role webapp
token_meta_service_account_name webapp
token_meta_service_account_namespace prod

Read the metadata on that response, because it is doing real work for you later. token_meta_service_account_namespace: prod is stamped on the token by Vault, not claimed by the app. When you are reading an audit log during an incident, that field tells you which namespace minted the token that read the secret, and no application can lie about it. It is also usable inside policy templates, so a single policy can grant secret/data/{{identity.entity.aliases.<accessor>.metadata.service_account_namespace}}/* and each namespace lands in its own lane without you writing one policy per team.

How a pod authenticates to Vault
1kubelet mounts a projected token
Short-lived signed JWT, audience-scoped, refreshed automatically into /var/run/secrets
2Pod POSTs it to auth/kubernetes/login
Sends the JWT plus a role name; holds no Vault credential of its own
3Vault calls TokenReview
Uses its reviewer identity (its own SA, or an explicit token_reviewer_jwt) to ask the API server if the JWT is genuine
4API server answers with identity
Returns the service account name and namespace, or 403 if the reviewer lacks system:auth-delegator
5Vault matches the role bindings
Name must be in bound_service_account_names, namespace in bound_service_account_namespaces
6Vault mints a scoped token
1h TTL, webapp policy, metadata stamped with the real SA name and namespace
The pod never holds a Vault secret of its own — its Kubernetes identity is the credential, and it expires on its own.

Who actually does the reviewing

This is the part that generates support tickets, so slow down here. When Vault verifies a pod's token, somebody has to be allowed to ask the Kubernetes API server "is this token real, and whose is it?" That permission comes from the system:auth-delegator ClusterRole. The question is which identity Vault uses to ask.

When Vault runs inside the cluster with default settings, it uses its own pod's service account token to make the TokenReview call. Vault's service account needs the delegator binding. The application's service account does not. The Vault Helm chart wires this up for the server service account by default, which is why in-cluster setups usually work the first time and why the failure, when it comes, is so confusing: it appears only after somebody overrides the chart's service account or moves Vault out of the cluster.

If you set an explicit token_reviewer_jwt on the mount, Vault always uses that one privileged identity for every review, no matter where Vault runs. This is the durable pattern. One dedicated reviewer service account, one binding, one thing to audit. Treat that reviewer token like a secrets engine root credential, because it is long-lived compared to the pod tokens it validates, and write down how you rotate it before you need to.

TokenReview returns 403 and every pod login fails
With no token_reviewer_jwt on the mount, Vault does not review a pod's token with that same token. When Vault runs inside the cluster it uses its OWN pod's service account token to call the TokenReview API. So it is Vault's service account, not each application's, that must be bound to the system:auth-delegator ClusterRole. The Vault Helm chart wires this binding for the server service account by default, so in-cluster logins usually work out of the box, until someone overrides the chart's service account, or runs Vault outside the cluster. Out-of-cluster (or with disable_local_ca_jwt set) and still no reviewer JWT, Vault falls back to reviewing each login with the caller's own token, and now every authenticating service account needs the delegator, which is a brittle pattern. The durable fix is to set an explicit token_reviewer_jwt (a dedicated reviewer service account) so Vault always reviews with one privileged identity. Either way Kubernetes answers 403 as a misleading "permission denied" even when your role bindings are perfect, so check the RBAC before you burn an afternoon re-reading correct-looking bindings.

For Kubernetes auth outside the cluster, meaning Vault on virtual machines talking to EKS, GKE, or AKS, you must supply kubernetes_host, the certificate authority material so Vault trusts the API server's TLS certificate, and usually an explicit token_reviewer_jwt from a dedicated reviewer service account. Managed control planes rotate their endpoints and certificates on their own schedule, so treat the CA bundle as something your automation refreshes rather than something you pasted in once during setup. On the application side, projected tokens should list Vault as an audience when your cluster version supports it, which shrinks the blast radius if one leaks: a token minted for Vault cannot be replayed against another API that checks audiences.

Living with short tokens

A one hour token is only useful if something renews it. An app that logs in at startup, caches the token forever, and never handles a 403 will run beautifully for 59 minutes and then fall over in a way that looks random and is not.

Vault Agent or the Vault Secrets Operator usually sits in front of this exchange so the app never speaks Vault's API directly. The agent authenticates with the pod identity, renders a template to a file or environment variable, and renews before the time to live expires. That pattern closes the secret-zero problem, where you need a credential to get a credential: bootstrap is a projected JWT the platform already mounts, not a token you pasted into a Deployment manifest and then had to rotate across forty services.

Two numbers on the role decide the shape of this. token_ttl is how long a token lives before it needs renewing. There is also a maximum lifetime beyond which no amount of renewal helps and the client must log in again from scratch. Set the TTL short enough that a leaked token is boring, long enough that you are not hammering TokenReview on every request. An hour with renewal at the halfway mark is a reasonable starting point for most workloads. Batch jobs that run for six hours need the maximum raised deliberately, not the TTL stretched, because those are different controls.

When a namespace is compromised, you have real revoke paths. Revoke by policy, or by token accessor if you know which token to kill, drop the role's TTL so anything still alive dies sooner, and delete the offending service account so no new login can succeed. Because authentication is identity-based, you are not hunting a static password that might still be sitting in a continuous integration variable from last quarter. Pair OIDC group aliases with Kubernetes role bindings and you get one coherent story: humans inherit from single sign-on, machines inherit from the pod they are.

Audit both doors from day one

Turn on an audit device before you roll OIDC out to the company, not after. You want auth/oidc/login and auth/kubernetes/login events flowing to storage nobody can quietly edit, starting with the very first login. During an incident those events answer who minted what policy, from which service account, at what minute. Without audit you are reconstructing the story from application logs while an attacker still holds a valid token until its time to live runs out.

One caution about audit devices, since it bites people during exactly the wrong week: if every configured audit device fails to write, Vault stops serving requests rather than serving them unlogged. That is deliberate. It is also a great way to take an outage if your only device is a log file on a disk that filled up. Two devices, different destinations, and an alert on the disk.

Then kill userpass for staff once OIDC works. Leaving a parallel password database "for emergencies" means it becomes the path everyone uses the first time single sign-on flakes, and then the emergency is permanent and you are back to the stolen laptop. Break-glass for humans should be a generate-root ceremony with multiple key holders, or a tightly controlled batch token issued for a named incident, and neither of those is a shared password in a wiki.

Try this

In a lab cluster with Vault installed, enable both auth methods and prove a human login and a pod login mint different policies. Then break the TokenReview RBAC on purpose once, by deleting the ClusterRoleBinding on Vault's service account, so you recognise the 403 signature when it turns up for real at a worse time. Put it back and confirm pod logins recover without restarting anything.

terminal
vault auth list
vault login -method=oidc role=engineer
vault token lookup -format=json | jq '{policies:.data.policies, ttl:.data.ttl}'
# from a pod with SA webapp in prod:
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
vault write auth/kubernetes/login role=webapp jwt="$TOKEN"
output
Path Type
oidc/ oidc
kubernetes/ kubernetes
policies: ["default","engineer"]
ttl: 3200
token_policies: ["default","webapp"]
token_meta_service_account_namespace: prod
# SUCCESS — two doors, two identities, no shared password

Two checks tell you the wiring is right rather than accidentally permissive. First, log in as a user who is in no mapped group at all and confirm they come back with ["default"] and nothing else, which proves your group aliases are granting rather than everybody inheriting from a fat role. Second, try the Kubernetes login from a pod in a different namespace using the same role name and confirm Vault refuses it. If that second one succeeds, your bindings are wider than you think and it is better to learn that in a lab.

Takeaway

Humans authenticate through OIDC and inherit policy from identity provider groups, so offboarding happens once, upstream. Workloads authenticate through Kubernetes service account JWTs bound tightly to a name and a namespace, verified by one reviewer identity you can point at. Neither path should rely on a long-lived Vault password or a token stuffed into etcd where a namespace read grants it to anyone.

Next: once identity works, stop handing apps static database passwords. Dynamic secrets mint a disposable credential per lease, which is the natural follow-on to short-lived tokens.

Quick check
01Why bind Kubernetes auth roles to both service account name and namespace?
Incorrect — TokenReview returns those fields; it does not demand them as a binding condition.
Correct — Name plus namespace is the least-privilege boundary for machine auth.
Incorrect — True but unrelated. The two auth methods are independent.
Incorrect — Bindings do not affect TTL, and non-expiring tokens are the opposite of the goal.
02OIDC login works fine in the Vault UI but vault login -method=oidc fails from a terminal. What do you check first?
Incorrect — OIDC never consults userpass. The two are separate mounts.
Correct — The UI and CLI use different redirect URIs, so one can work while the other is rejected at the IdP.
Incorrect — Unrelated. Kubernetes auth has no bearing on a browser-based OIDC login.
Incorrect — A quorum loss breaks every request, including the UI login that just succeeded.
03Vault runs in-cluster with default local JWT review. Which identity needs the system:auth-delegator ClusterRole?
Incorrect — That is the fallback pattern when there is no reviewer JWT and Vault is out of cluster, and it is brittle on purpose to avoid.
Correct — Vault calls TokenReview with its own pod's token, so that account carries the delegator binding.
Incorrect — Kubernetes RBAC has no view of Vault identity groups.
Incorrect — kube-proxy handles service networking and plays no part in token review.

Related