Why production Vault exists
Standing secrets, sprawl, and the platform you need.
Almost nobody adopts Vault because they wanted another service to keep alive. They adopt it the Monday after a database password turns up in a public GitHub gist, or after an AWS (Amazon Web Services, the cloud most companies rent servers from) access key that somebody pasted into a CI (continuous integration, the robot that builds and tests your code on every commit) variable three years ago starts renting graphics cards to mine cryptocurrency on the company card. The bill lands before the alert does.
The thing that failed in both stories has a name: standing privilege. A credential that works forever, gets copied into five places, and belongs to nobody in particular once the person who created it moves to another team. It behaves like the master key to an office building that was cut fourteen times over the years, with no list of who holds a copy. You cannot change the locks without breaking something, and after a break-in you cannot say whose copy opened the door. Every other problem in this course grows out of that one.
So the goal is not a tidier password file. A secrets platform has five jobs, and skipping any one of them leaves you with the master-key problem in a nicer box. It authenticates the caller, so it knows who is asking. It authorizes path by path, so the service that reads one database password cannot read the payroll signing key. It encrypts everything before it touches disk, so a stolen drive is a stolen brick. It records every read in an audit log, so you can answer "who used this" nine months later. And it hands out credentials that expire on their own, so most secrets are already dead by the time anyone finds them.
HashiCorp Vault does all five in one system. This course walks from the toy vault server -dev to something you can defend at 3 a.m. during an outage and across a table from an auditor. If you already keep secrets in AWS Secrets Manager, Google Secret Manager, or SOPS (Secrets OPerationS, a tool that encrypts secret values inside files you can safely commit to Git), the mental model carries over unchanged: identity in, least privilege out, short lifetime by default. What Vault adds is doing that for databases, cloud APIs and your own certificate authority through a single control plane with a single audit trail.
How One Password Becomes Fifteen Copies
Follow one ordinary Postgres (PostgreSQL, the open source database sitting under a large share of the world's web applications) password through one ordinary company. Someone creates it during setup and drops it into a .env file. That file gets base64-encoded into a Kubernetes Secret so the application can read it, Kubernetes being the system most teams use to run containers across a fleet of machines. A screenshot of the running pod, one pod being a single live instance of your application, ends up in a wiki runbook with the password visible in the shell prompt. The .env file sits on a laptop that backs up to iCloud. A contractor gets a copy over Slack during onboarding, because the wiki page was out of date. The same string of characters now exists in fifteen places, and nobody remembers at least four of them.
Two properties are gone at that point, and neither comes back on its own. The first is rotation. Changing the password means every consumer has to change at the same moment, so the work gets scheduled, postponed, and quietly dropped. Static database passwords tend to survive for years for exactly that reason. The second is attribution. When the leak surfaces, your Postgres log shows a successful login from an address inside your own network. Which copy was that? The pod, the contractor's laptop, or the backup? You cannot tell, because all fifteen copies are byte-for-byte identical.
Dynamic secrets go after both problems at once. Rather than storing one shared password, Vault holds administrative access to Postgres and creates a brand new database user every time a service asks, with a TTL (time to live, how long the credential stays valid) measured in hours. The payments API gets one user. The nightly reporting job gets a different one. When the lease runs out, and a lease is Vault's own record that it handed something out with an expiry stamped on it, Vault connects to Postgres and runs the revocation SQL (Structured Query Language, the language you use to talk to a relational database) that you configured. The built-in default for Postgres strips the role's grants and privileges first, then drops the role. Rotation stops being a project and becomes the resting state, and attribution comes free, because the Postgres log already holds a username that belongs to exactly one caller and one lease.
Here is the honest cost. You have made Vault a hard dependency of every service start-up, and you have handed Vault credentials strong enough to create and drop database users. That is real concentration of risk. It is still the better trade, because concentrated risk can be sealed, protected by a quorum of key holders, watched and audited, while fifteen scattered copies of a password cannot be watched at all. You are choosing one door you guard properly over fifteen doors you have lost track of.
Identity Is The Other Half
A wine cellar with an excellent lock and one key hanging on a nail beside the door is technically locked. Encrypted storage behind a single shared key is that same picture. Vault's answer is that every request carries a token, every token was issued to a specific identity by a specific auth method, and every token is bound to policies that spell out which paths it may touch.
People and machines come in through different front doors. Humans log in with OIDC (OpenID Connect, the standard behind "sign in with Google" and most company single sign-on, usually shortened to SSO), so their Vault identity is their real identity and switching off their SSO account switches off their Vault access at the same moment. Workloads use the Kubernetes auth method or a cloud auth method, where the platform itself vouches for the pod or the virtual machine and nothing has to be baked into a container image. Everything else uses AppRole, where a role ID and a short-lived secret ID are traded for a token. Nobody uses the root token in production. It is created once at initialisation, it carries the root policy, it never expires, and the right move is to revoke it with vault token revoke once setup is finished and mint a fresh one with vault operator generate-root on the rare day you need it, which takes a quorum of unseal key holders standing there with you.
This is the part that trips people up. A Vault token is not "logged in as an administrator". A token carries an explicit list of policy names, and a policy is a list of path rules with capabilities attached. If no rule grants a path, the answer is no, because Vault denies by default. Matching happens by specificity first: an exact path beats a wildcard, and a longer prefix beats a shorter one. Then, among the policies on your token that define a rule for that same path, a deny capability wins over any allow. Memorise the second half. It is how one broad deny added to a shared policy locks out an entire team on a Friday afternoon.
# What can the token in my environment actually do?vault token lookup
Key Value--- -----accessor 8dR2mVqK9wZbTn4XcLpF7yGhcreation_time 1785142800creation_ttl 768hdisplay_name approleentity_id 8f4a1b2c-6d0e-4a71-9c33-2e5b7d0a1f88expire_time 2026-08-28T09:00:00.117492Zexplicit_max_ttl 0sid hvs.CAESIHQ7bV2mKp...issue_time 2026-07-27T09:00:00.117492Zmeta map[role_name:payments-api]num_uses 0orphan truepath auth/approle/loginpolicies [default payments-read]renewable truettl 767h58m12stype service
Three lines on that output decide almost everything. policies is the entire permission surface: payments-read plus the built-in default policy, and nothing else is implied or inherited. ttl counts down in real time. renewable true means the holder can call vault token renew to push expire_time forward, but only up to a ceiling, and the ceiling here is the auth mount's maximum lease TTL measured from creation_time, because explicit_max_ttl is 0s and therefore unset. Renewable does not mean immortal. A renewable token behaves like a library book you can keep extending until the library says no. There is one exception worth carrying with you: a periodic token, issued by a role configured with a period, ignores the max TTL and can be renewed indefinitely as long as each renewal lands inside that period. Vault Agent, the helper process that logs in on behalf of an application and keeps its token alive, leans on exactly that. One housekeeping note on the output above: the id line is trimmed for the page. Your terminal prints the whole token, which is a decent reason never to paste a lookup into a chat window.
Leases behave the same way one level down. When Vault issues a dynamic database credential it hands back a lease_id alongside the username and password. Renewing that lease keeps the database user alive. Letting it lapse, or running vault lease revoke, makes Vault execute the revocation statements immediately. That is what revocation means here, and it is why it beats changing a shared password. Revoking a lease removes the ability to log in, full stop. Changing a shared password only helps once you have hunted down and updated every consumer, which drops you straight back into the fifteen copies.
# Ask for a fresh, short-lived Postgres loginvault read database/creds/payments-readonly
Key Value--- -----lease_id database/creds/payments-readonly/2f6a614c-4aa2-7b19-24b9-ad944a8d4de6lease_duration 1hlease_renewable truepassword A1a-8fJk2mQx7vRtusername v-approle-payments-9tPq3kXm2vNwLbT4sYzD-1785142980
Read that username from left to right and it tells you its own story. The v- marker, then the token's display name cut to eight characters (approle), then the Vault role name cut to eight (payments, from payments-readonly), then twenty random characters, then the Unix timestamp of the moment it was issued. That layout is the database plugin's default username template, and you can change it. When something odd shows up in the Postgres log at 2 a.m., that one string walks you back through the Vault audit log to the exact role, the exact request and the exact minute. No shared password buys you that at any price.
Watch Dev Mode Do Everything Wrong
The quickest way to understand production Vault is to run the version that skips all of it. vault server -dev is a driving-school car with the engine lifted out: same steering wheel, goes nowhere, and it vanishes the moment you shut the door. Run it once, read what it prints, and you are holding a checklist of every shortcut the rest of this course removes.
vault server -dev
==> Vault server configuration:Api Address: http://127.0.0.1:8200Cgo: disabledCluster Address: https://127.0.0.1:8201Go Version: go1.22.5Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")Log Level: infoMlock: supported: true, enabled: falseRecovery Mode: falseStorage: inmemVersion: Vault v1.17.2, built 2024-07-05T15:19:12Z==> Vault server started! Log data will stream in below:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memoryand starts unsealed with a single unseal key. The root token is alreadyauthenticated to the CLI, so you can immediately begin using Vault.You may need to set the following environment variable:$ export VAULT_ADDR='http://127.0.0.1:8200'The unseal key and root token are displayed below in case you want toseal/unseal the Vault or re-authenticate.Unseal Key: mDl4Kx0Vv1sZ7Yb2Qn8Rp3Tc6Wf9Ha5Jg1Ld4Ne7OiQ=Root Token: hvs.QkX2m7ZpR4tLv9NbCw3Ys6DgDevelopment mode should NOT be used in production installations!
Count the problems in that banner. Storage is inmem, so every secret lives in memory and dies with the process. TLS (Transport Layer Security, the encryption behind the padlock in your browser) is switched off on the listener, so tokens cross the network in the clear. It starts unsealed with a single unseal key, so there is no ceremony and no split of trust between people. The root token is printed to your terminal, which on most machines means your shell history and your system journal as well. Dev mode also mounts the KV v2 secrets engine at secret/ for you, which is why the commands further down work with no setup at all. Every one of those shortcuts is deliberate. The danger lives entirely in how convincing the thing looks once it is running.
export VAULT_ADDR='http://127.0.0.1:8200'vault status
Key Value--- -----Seal Type shamirInitialized trueSealed falseTotal Shares 1Threshold 1Version 1.17.2Build Date 2024-07-05T15:19:12ZStorage Type inmemCluster Name vault-cluster-3f2c1a94Cluster ID 6b1d0e2f-83a7-4c11-9d5e-0f7a2c46bb90HA Enabled false
Total Shares 1 and Threshold 1 give the game away. Vault encrypts your data with an encryption key, keeps that key encrypted under a root key, and keeps the root key encrypted under the unseal key. Shamir's Secret Sharing, the default seal, splits that unseal key into numbered shares and demands a quorum of them back before it will rebuild it, so no single person can open a sealed Vault alone. Five shares with a threshold of three, held by five different people, is an ordinary production shape. Dev mode sets both numbers to one, which deletes the entire point of the mechanism. HA Enabled false and Storage Type inmem are the other two tells, HA meaning high availability, more than one node so the service survives losing one. Here you have a single node, no replication, and nothing written to disk.
vault server -dev keeps everything in memory, starts unsealed with a single key share, turns TLS off, and prints a root token to standard output. Learning on it is fine. The trouble starts when somebody runs it on a shared virtual machine "for now", two other teams point their applications at it, and six months later a production service depends on an unsealed, unencrypted, in-memory Vault holding real credentials that evaporate at the next reboot. If you see Storage Type inmem anywhere other than a laptop, treat it as an incident rather than a chore.Try This: Make The Secret Disappear
Two minutes of typing makes the case for durable storage better than another paragraph will. Write a secret, read it back, kill the process, and watch the data stop existing. Dev mode drops the root token into ~/.vault-token, so your CLI (command line tool) is already authenticated. The script below lifts the token out of the log anyway, because the VAULT_TOKEN environment variable takes priority over that file and you may already have one set from something else.
# start dev mode in the background and keep its logvault server -dev > /tmp/vault-dev.log 2>&1 &VAULT_PID=$!sleep 2export VAULT_ADDR='http://127.0.0.1:8200'export VAULT_TOKEN=$(awk '/Root Token:/ {print $NF}' /tmp/vault-dev.log)vault kv put secret/demo password=s3cret
== Secret Path ==secret/data/demo======= Metadata =======Key Value--- -----created_time 2026-07-27T09:14:03.117492Zcustom_metadata <nil>deletion_time n/adestroyed falseversion 1
vault kv get secret/demo
== Secret Path ==secret/data/demo======= Metadata =======Key Value--- -----created_time 2026-07-27T09:14:03.117492Zcustom_metadata <nil>deletion_time n/adestroyed falseversion 1====== Data ======Key Value--- -----password s3cret
Look at the path split before you go any further. You wrote to secret/demo, and Vault reports the real path as secret/data/demo. That is KV v2 (key-value version 2, the versioned secrets engine) keeping live values under data/ and version history under metadata/. It matters the first time you write a policy: a rule on path "secret/demo" grants nothing whatsoever, because the API (application programming interface, the URL paths the Vault server actually serves) path the client calls is secret/data/demo. That single mismatch is behind most of the "my policy does not work" tickets in a Vault rollout, and it will find you again the first time you write HCL (HashiCorp Configuration Language, the format Vault policies and server config files are written in).
kill "$VAULT_PID"sleep 1vault status
Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused
Start it again and secret/demo is gone, along with the root token, the unseal key and the cluster ID. Nothing ever reached disk, so there is nothing to recover and no snapshot to restore. Everything the next few lessons build exists to change one of those facts: integrated storage on Raft, the consensus algorithm that keeps several Vault nodes agreeing on the same data, so writes land on disk and replicate to peers; a seal, so those files on disk are useless to whoever steals them; auto-unseal, so a reboot does not need a human holding a key share; and audit devices, so every read leaves a trail that outlives the process that served it.
What This Course Covers, And What It Does Not
You will stand up a three-node cluster on integrated Raft storage and watch what happens when one node dies, and then what happens when two do. You will put real TLS certificates on the listeners and configure auto-unseal against a cloud KMS (Key Management Service, a managed service that holds an encryption key you can use but never export), which swaps Shamir unseal keys for recovery keys. You will write ACL (access control list) policies in HCL and prove they deny what you expect them to deny. You will use KV v2 without falling into the data/ trap, mint dynamic Postgres and cloud credentials, run a short-lived internal PKI (public key infrastructure, the machinery that issues and signs TLS certificates), deliver secrets to workloads with AppRole and Vault Agent auto-auth and templating, turn on audit devices, rehearse break-glass recovery for the day normal access is gone, and take and restore Raft snapshots.
Left out on purpose: namespaces, performance and disaster recovery replication, Sentinel policies, and SPIFFE (Secure Production Identity Framework For Everyone) workload identity. Those belong in an advanced track, and they are the wrong thing to reach for before your single cluster has TLS, real policies and a restore you have actually practised. No outage was ever survived by replication while the operators were passing a root token around.
Three beliefs to drop before lesson two. Vault does not encrypt your Kubernetes Secrets for you: Kubernetes stores those as base64 text inside etcd, the key-value database a cluster keeps its state in, and base64 is an encoding rather than a lock, so Vault helps only if you deliberately wire the two together. Dev mode is not almost-production, as its own banner spells out at length. And a secret is not safe because it arrived in Vault. The policy that grants access, the TTL that bounds it, and the audit device that records it are what decide whether a mistake stays small.
Next you swap the in-memory toy for three nodes on Raft storage, and find out what a quorum failure looks like from the on-call phone at 3 a.m.
vault token lookup shows renewable true, ttl 767h58m12s and explicit_max_ttl 0s. What does renewable actually guarantee?path "secret/demo" { capabilities = ["read"] } and gets a 403 permission denied, even though the secret exists and the token is valid. What do you fix?list controls directory enumeration only. Adding it changes nothing about this denial.vault kv get secret/demo calls the API path secret/data/demo, and policies are matched against the API path, so the rule as written grants nothing at all.Try this
Run vault token lookup on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: dev mode is a lab tool, and it sticks around. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.