Dev server to HA raft

Storage, unseal, and what breaks when a node dies.

It is 3:14 a.m. and the on-call phone is buzzing because half the platform cannot fetch secrets. Vault is not down in the way a crashed process is down. The surviving node is running, listening on port 8200, and answering vault status with Sealed: false. Writes still fail. Clients hang. The load balancer keeps politely handing traffic to a lonely machine that cannot form a raft quorum. That is the production shape of a Vault failure, and it looks nothing like the friendly vault server -dev you ran in a tutorial last week.

A vault server -dev is a driving-school car with the instructor's pedals still bolted in. It starts unsealed, keeps everything in memory, prints your root token onto the terminal, and forgets all of it the moment you close the door. Perfect for learning. Useless for anything anyone would be paged about. Production Vault reverses every one of those choices: durable storage on local disk, a sealed store that somebody has to deliberately open, and several machines that vote before they trust a single write. Teams ship the driving-school car anyway, because the dashboard looked identical, and because a -dev process on a spare VM (virtual machine) has a way of quietly becoming load-bearing.

This lesson walks that toy up to a three-node cluster on integrated storage, which is Vault's own implementation of raft (a consensus algorithm: a set of rules that lets several machines agree on one ordered list of changes even while some of them are broken or unreachable). You get the failure modes in order. One node dead. Two nodes dead. A rebooted node that comes back sealed with nobody holding keys. And the join that fails for a reason the error message only half explains.

What The Dev Server Decides Without Asking

vault server -dev makes five decisions for you and mentions none of them. Storage is in memory, so your secrets live inside the process and die with it. The listener binds to 127.0.0.1 on port 8200 with TLS (transport layer security, the encryption behind the padlock icon in a browser) switched off. The cluster is initialized for you. It is unsealed for you. A root token is minted and printed straight onto your screen, which is roughly the security posture of taping the master key to the front door. Every one of those is right for a ten-minute experiment and wrong for everything after it.

In production you make all five calls yourself, starting with where the data lives. Vault has carried a long list of storage backends over the years. The file backend writes to local disk and offers no HA (high availability, meaning the service keeps answering when a machine dies), so it gives you exactly one node and a bad night when that node dies. Consul, HashiCorp's service catalog, was the traditional answer: a second distributed system with its own quorum, its own access rules and its own failure modes, run purely to hold Vault's brain. Integrated storage folded that job back inside Vault.

The payoff is one process, one data directory, one thing to upgrade. Nodes replicate among themselves, so every member keeps a complete encrypted copy of the store on its own disk, and a backup becomes an API (application programming interface, the machine-facing door into Vault) call instead of a filesystem trick. When vault status prints Storage Type raft, that is what you are looking at, and everything below assumes it.

Raft Is A Room Taking Minutes

A raft cluster behaves like a committee with a single secretary. Anybody may propose a change, only the secretary writes in the minute book, and a proposal counts as recorded once a majority of the members have copied the same line into their own books in the same order. Vault's version of that: one node is the leader and shows HA Mode active, the others are followers sitting in standby, and every write becomes a numbered entry in an append-only log that the leader ships out to everyone. Reads can be answered locally. Writes have to go past the secretary.

A commit happens in four beats. The leader appends the entry, sends it to the followers, waits until a majority reply that it is written to their disk, then applies it to the store and answers the client. The word disk is doing real work in that sentence. Followers flush the entry before they acknowledge it, so Vault's write latency is your storage's fsync latency (the time it takes for a write to become genuinely durable rather than sitting in a cache). Put a raft cluster on a slow network volume and you get election churn, request timeouts and a cluster that behaves as though it is haunted. Give it local SSD.

Leadership is decided by the same majority rule. Followers expect a heartbeat from the leader, and when the heartbeats stop arriving one of them nominates itself and asks the others to vote. It wins only with a majority. That single rule is what stops a partitioned network from producing two leaders writing two different histories. It is also why an even voter count is a bad trade: four voters need three to agree, exactly like five voters do, while leaving you one fewer spare machine.

One idea before the config file, because it is the one most people have backwards. Unsealing is not logging in. Everything Vault writes to storage is encrypted with a key that itself sits on disk in encrypted form, and unsealing is the act of reassembling the key that decrypts that key. It turns the bytes on disk from noise into something Vault can read. Nobody gains a single privilege by unsealing. An unsealed Vault with no token still answers every request with permission denied. Seal state is about storage encryption; who you are is a separate question, answered later by auth methods.

Trade -dev For A Real Storage Backend

The dev server hid three decisions you now own: where data lives, how the node is reached, and how it gets opened. Two address fields in the config carry far more weight than their size suggests. api_addr is the address this node advertises to clients and to peers as the place to reach it. cluster_addr, port 8201 by default, is the private machine-to-machine channel raft uses for replication and for forwarding writes to the leader. One is the street address on your business card. The other is the service corridor behind the building.

Point either of them at localhost, or at a hostname that resolves inside one subnet only, and the node still initializes perfectly. It never becomes useful. You get standbys that never turn ready, or clients that authenticate against one node and are then redirected to an address they cannot reach from where they happen to be standing. Treat api_addr as client-reachable identity and cluster_addr as private fabric. They are not two spellings of one thing, and nothing in Vault validates either value for you.

# /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"            # unique per node
 
  # add leader_ca_cert_file below if the leader's TLS cert
  # is signed by a private/internal CA (not in the system trust store)
  retry_join { leader_api_addr = "https://vault-2.internal:8200" }
  retry_join { leader_api_addr = "https://vault-3.internal:8200" }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
}
 
# auto-unseal so a reboot doesn't page a human
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
}
 
api_addr     = "https://vault-1.internal:8200"  # client / standby reachability
cluster_addr = "https://vault-1.internal:8201"  # raft replication channel

Read that file from the top. path points at local disk, never at NFS (network file system, a folder shared over the network) or anything else where two machines could touch the same bytes. node_id has to be unique across the cluster; leave it out and Vault picks one for you and remembers it inside the data directory, which is fine right up until you rebuild a machine and find a stranger in the peer list. The retry_join blocks let a rebooted node find its way home on its own, so you list its peers once in configuration management and stop running manual joins at 4 a.m.

Key                     Value
---                     -----
Seal Type               awskms
Recovery Seal Type      shamir
Initialized             true
Sealed                  false
Total Recovery Shares   5
Threshold               3
Version                 1.17.x
Storage Type            raft
Cluster Name            vault-cluster-prod
HA Enabled              true
HA Cluster              https://vault-1.internal:8201
HA Mode                 active
Active Since            2026-07-20T08:12:01Z

Four lines in that table tell you the cluster is real. Storage Type raft confirms integrated storage. HA Enabled true says this node is allowed to take leadership. HA Mode active marks the leader, where a standby would print standby instead. HA Cluster echoes the cluster-side address, which makes it a free sanity check on the cluster_addr you set. The two seal lines say a cloud key opens this vault while a set of human-held recovery shares authorizes privileged operations, and that pair is the next lesson's whole subject.

Initialize Once, And Only Once

Initialization happens one time for the entire cluster, against a single node, and it is the moment the cryptography is born. vault operator init generates the root key that protects everything in storage, splits the material guarding it using Shamir's secret sharing (an algorithm that cuts a secret into parts where any threshold of them rebuilds it and fewer than that reveal nothing at all), and hands back an initial root token. Until enough holders present their shares, the data on disk stays unreadable, including to Vault itself.

Which flags you use depends on your seal. With no seal stanza you are on Shamir, so -key-shares=5 -key-threshold=3 cuts five keys and demands three of them at every start. With an auto-unseal seal stanza, a cloud KMS (key management service, a managed store that holds a key and will decrypt things on request) does the opening, and you size the human key set with -recovery-shares and -recovery-threshold instead. Confuse the two ceremonies and you will be pasting recovery keys into vault operator unseal while an incident clock runs in the corner of the screen.

# run ONCE, against a single node
 
# Shamir seal (no seal stanza): split the root key into 5 unseal keys
vault operator init -key-shares=5 -key-threshold=3
#   prints 5 unseal keys + initial root token
#   supply 3 of 5 keys to open the vault:
vault operator unseal   # run 3x, one key per holder
 
# Auto-unseal (awskms seal stanza): use RECOVERY flags, not key flags
vault operator init -recovery-shares=5 -recovery-threshold=3
#   prints 5 recovery keys + root token; unseal is automatic on boot
 
vault status

Unseal Key 1: xxxxxxxx...
Unseal Key 2: yyyyyyyy...
...
Initial Root Token: hvs.CAESIJexample
 
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
HA Enabled      true
HA Mode         active
Storage Type    raft

The initial root token is a bootstrap credential, the same species as the temporary password a new laptop ships with. It exists so you can configure the first auth method and the first policies, and then it should stop existing. Run vault token revoke -self the moment real logins work. Leaving root parked in a shared password manager means whoever phishes one laptop owns the whole secret plane, with no policy in the way. If you need root again years later, a threshold of key holders can mint a fresh one, which is the right amount of friction for that credential.

Add Nodes: Raft Join And Quorum

New nodes start empty and sealed, then join the cluster that already exists, and the join copies the encrypted store over the cluster port rather than the API port. The tidy way is the retry_join blocks already sitting in the config. The manual way is vault operator raft join pointed at any live node. On a Shamir cluster a freshly joined node still has to be unsealed before it does anything, because it received a pile of encrypted data and no way to read it. On an auto-unseal cluster it opens itself and starts catching up.

Quorum is the entire reason for running more than one node. A write commits only once a majority of voters, (n/2)+1, has it on disk, and that majority rule is what stops two halves of a split cluster from inventing two different pasts. Three voters need two, so you survive one loss. Five voters need three, so you survive two. Nothing in between buys anything, which is why real clusters run three or five voters and never two or four.

# if you didn't use retry_join, join explicitly against any live node:
vault operator raft join https://vault-1.internal:8200
 
# confirm membership and who can vote
vault operator raft list-peers
 
# autopilot: quorum, healthy voters, dead-server cleanup
vault operator raft autopilot state

Node     Address                  State     Voter
----     -------                  -----     -----
vault-1  vault-1.internal:8201    leader    true
vault-2  vault-2.internal:8201    follower  true
vault-3  vault-3.internal:8201    follower  true
 
# autopilot state (trimmed)
Healthy:                 true
Failure Tolerance:       1
Leader:                  vault-1
Voters:                  [vault-1 vault-2 vault-3]

Write forwarding is the part that ambushes people mid-incident. A standby does not accept a write of its own; it hands the request to the leader over the cluster port and passes the answer back. So a wrong api_addr on a quiet standby costs you nothing at all, and costs you everything the second that node wins an election, because now every client is being sent to an address that may not exist. A landmine that only arms itself during failover is the worst kind, since failover is precisely when nobody has attention to spare.

The Joins That Fail, And What They Are Telling You

The most common first-day disaster is running vault operator init on all three machines. Each one succeeds. Each one prints its own root token and its own key shares. You now own three separate one-node clusters that will refuse to join each other, and they are not being awkward: every one of them holds a different root key and a different history, and raft has no rule for merging two pasts into one. The fix is to choose one as the real cluster, wipe the other two back to empty, start them, and let retry_join do the work. The same reasoning covers any node still carrying a data directory from a cluster that no longer exists.

Second failure: duplicate node_id values. This one arrives when somebody bakes a machine image after Vault has already run once on it, so every instance built from that image insists it is vault-1. The peer list ends up with entries fighting over a single identity, and membership flaps in a way that reads like a network fault for as long as you let it. Set node_id from the hostname in configuration management, and check vault operator raft list-peers after any image rebuild.

Third failure: the cluster port. Firewall rules and security groups get written for 8200, because that is the port everyone knows, and 8201 quietly stays shut between nodes. The symptom is oddly precise. Nodes appear in the peer list and then drop out of it, because the API port is fine while the replication channel is not. Test reachability directly rather than reasoning about the rule set, and question each node separately about what it believes, since sys/leader answers without a token at all.

# is the cluster port actually open between nodes?
nc -vz vault-1.internal 8201
 
# ask each node who it thinks the leader is (sys/leader needs no token)
curl -s https://vault-1.internal:8200/v1/sys/leader
curl -s https://vault-2.internal:8200/v1/sys/leader
 
# a node that was initialized by mistake carries its own cluster in here
ls /opt/vault/data
 
# reset that node so it can join the real cluster (it holds nothing unique yet)
systemctl stop vault
rm -rf /opt/vault/data/*
systemctl start vault
vault operator raft list-peers

You know the fix landed when three separate things agree. vault operator raft list-peers returns the same rows no matter which node you ask it from, autopilot reports a failure tolerance of one, and the repaired node survives a deliberate restart without dropping out of the list. Checking only from the leader is how people declare victory over a cluster that is still broken from the far side of a one-way firewall rule.

The Load Balancer Is Part Of The Cluster

Vault tells a load balancer what it is through the status code on /v1/sys/health, and those codes say more than a normal health check does. 200 means initialized, unsealed and active. 429 means unsealed but standby, which is a strange number until you read it as this node saying it is healthy and not in charge. 501 means not initialized. 503 means sealed. A balancer configured to accept 200 and nothing else will mark two of your three perfectly healthy nodes as dead, and the team will spend an hour staring at raft before anyone looks at the health check.

# 200 active, 429 unsealed standby, 501 not initialized, 503 sealed
curl -s -o /dev/null -w '%{http_code}\n' https://vault-1.internal:8200/v1/sys/health
 
# count standbys as healthy too (they forward writes to the leader)
curl -s -o /dev/null -w '%{http_code}\n' 'https://vault-2.internal:8200/v1/sys/health?standbyok=true'

Decide deliberately which nodes take traffic. Sending everything to the active node is easy to reason about and turns the leader into a bottleneck for reads. Accepting standbys with standbyok=true spreads the load and leans on write forwarding, which is fine as long as api_addr is correct on every node. The one thing to avoid is monitoring Vault only through the balancer, because a working balancer hides a sealed node by pulling it out of rotation, and that node is the one you needed to hear about.

Autopilot And The Number That Decides Your Night

Autopilot is the shift supervisor built into integrated storage. It tracks which servers are alive, decides when a newly joined node has been steady long enough to be trusted as a voter, and reports the one number worth pinning to a dashboard: failure tolerance, meaning how many more machines can die before writes stop. A healthy three-node cluster reads 1. When it reads 0 you are one failed disk away from an outage, and nothing else on your screen will look the least bit wrong.

Dead voters are how that number slides without anyone noticing. Replace a VM without telling Vault and the old node sits in the peer list indefinitely, counted in the majority, incapable of voting. Autopilot can clear those out, and it will not until you ask: cleanup_dead_servers is off by default, and it also wants min_quorum set so it never prunes the cluster below a size you can survive. For a machine that is genuinely gone, vault operator raft remove-peer takes it out by node id and gives you your tolerance back.

# current settings
vault operator raft autopilot get-config
 
# prune servers that have been unreachable past the threshold,
# but never drop the cluster below min-quorum voters
vault operator raft autopilot set-config \
  -cleanup-dead-servers=true \
  -dead-server-last-contact-threshold=10m \
  -min-quorum=3
 
# a machine that is never coming back
vault operator raft remove-peer vault-3

Alert on failure tolerance hitting zero, on leadership changes happening more than a handful of times a day, and on any node reporting sealed for longer than a normal boot takes. Read all three as security signals rather than infrastructure noise. Somebody who can seal a node or partition the cluster steals nothing at all and still takes your platform off the air, and the incident channel will spend its first twenty minutes certain this is a networking problem.

Try This

Build a disposable three-node lab with no route to anything real, initialize once, join the peers, then break it on purpose. Start with a clean failover: kill the leader process and watch a new one appear. What you want is boring. Clients retry, HA Mode moves to another node, list-peers shows a new leader within seconds. What you do not want is a mystery hang, which almost always means api_addr is lying to somebody.

export VAULT_ADDR=https://vault-1.internal:8200
vault status
vault operator raft list-peers
# stop the leader process (lab only), then:
vault status
vault operator raft list-peers
vault operator raft autopilot state

Sealed          false
HA Mode         active
 
Node     Address                  State     Voter
vault-1  vault-1.internal:8201    leader    true
vault-2  vault-2.internal:8201    follower  true
vault-3  vault-3.internal:8201    follower  true
 
# after leader kill (example)
Sealed          false
HA Mode         active          # now on vault-2
vault-2  ...                  leader    true
vault-3  ...                  follower  true
# vault-1 missing or nonvoter until it returns and rejoins

Then run the two experiments that teach the most. Kill a second node and try to write a secret: the survivor stays unsealed and reports itself happily while every write hangs, which is the exact shape of the 3 a.m. page this lesson opened with. Bring one peer back and watch writes resume with nobody doing anything clever. Finally, if your lab runs Shamir rather than a cloud seal, reboot one node and leave it sealed for ten minutes while the other two keep working, so you can see a sealed node holding a voter slot in the configuration while being unable to acknowledge a single entry.

Takeaway

Production Vault is three things stacked together: raft for agreement, a deliberate unseal for the storage encryption, and advertised addresses that are true from wherever your clients actually stand. Size the voter count for the failures you expect rather than the ones you hope for, keep the count odd, and watch autopilot's failure tolerance with the attention you already give to free disk space. Once quorum is gone, no amount of restarting brings it back.

Next you hang real doors on this building. TLS on the listener so nothing crosses the wire in clear text, and an auto-unseal seal backed by a cloud key so a Tuesday reboot stops being a ceremony with three envelopes and a bridge call.

Related