TLS listeners and auto-unseal

api_addr, seals, and recovery keys done right.

Someone runs tcpdump (a tool that records raw network packets as they pass an interface) on a host to chase a slow API call, saves the capture to a shared drive, and picks up the next ticket. Sitting in that file, in plain readable text, is an X-Vault-Token header belonging to a live service account, and a JSON response body holding your production database password. No exploit. No zero-day. Nobody escalated any privileges. A cleartext listener handed all of it to anyone who could watch the wire.

Vault encrypts everything it writes to disk, so the raft data directory on each node (integrated storage, Vault's own replicated database) is unreadable ciphertext. The catch is that Vault's entire job is to decrypt those secrets and hand them to callers. You built a bank vault with an excellent door, then carried the cash across the lobby in an open bucket. TLS (Transport Layer Security, the encryption behind the padlock icon in your browser bar) is the armored corridor from the vault door to the client's hand.

Three settings decide whether a Tuesday-morning reboot is boring or a pageable ceremony: what the listener presents and demands, the address Vault advertises to everyone who talks to it (api_addr), and which seal protects the key that opens the barrier. They fail in combination. Get the seal right and the certificate wrong, and clients cannot reach a perfectly healthy cluster. Get the certificate right and the seal wrong, and a 3 a.m. reboot needs three humans holding three envelopes.

The Listener Decides What Reaches the Wire

A listener stanza is the front desk of the building. It sets which network address Vault answers on, what certificate it shows to prove who it is, and whether callers have to show identification of their own. The parameter that causes the most damage is the one people set "temporarily": tls_disable = true. Its default is false, and every cleartext Vault in the world got there because somebody was losing a fight with a certificate at 11 p.m.

Two version knobs already ship with sane defaults: tls_min_version is tls12 and tls_max_version is tls13. Raising the floor to tls13 is a real improvement if every client can reach it, and older SDKs (software development kits, the client libraries your apps import), appliance load balancers and Java 8 runtimes often cannot. There is a trap in tls_cipher_suites. Go, the language Vault is written in, does not let anyone configure TLS 1.3 cipher suites, so that setting quietly applies to TLS 1.2 connections only and does nothing at all for 1.3.

The other silent failure is the certificate chain. tls_cert_file points at one PEM file (Privacy Enhanced Mail, the plain-text format certificates ship in, all -----BEGIN CERTIFICATE----- blocks), and Vault serves exactly what is in it. Put only the leaf certificate there and browsers still work, because they cache intermediate certificates from earlier visits or fetch them on demand. Your Go and Python clients will not. The bug report arrives as "it works on my laptop but the pods cannot connect," and the cause is a missing intermediate nobody thought to check.

listener "tcp" {
  address       = "0.0.0.0:8200"
 
  # leaf certificate FIRST, then any intermediates, all in one PEM file
  tls_cert_file = "/etc/vault.d/tls/vault-fullchain.pem"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
 
  tls_min_version = "tls12"   # default; "tls13" if every client can handle it
  tls_max_version = "tls13"   # default
 
  # optional mutual TLS: callers must present a cert signed by this CA
  # tls_client_ca_file                 = "/etc/vault.d/tls/client-ca.pem"
  # tls_require_and_verify_client_cert = true
 
  # both default to false; leave them off unless you control who reaches the port
  unauthenticated_metrics_access = false
  unauthenticated_pprof_access   = false
 
  # cluster_address defaults to one port above `address`, so 8201
}
 
api_addr     = "https://vault-1.internal:8200"   # where callers get sent
cluster_addr = "https://vault-1.internal:8201"   # private node-to-node fabric
 
disable_mlock = true    # HashiCorp's recommendation when using integrated storage

Mutual TLS on the listener (tls_require_and_verify_client_cert, where the server demands a certificate from the caller as well) is a strong control with a cost you should price in first. Every caller now needs its own certificate, including your load balancer's health check. If the balancer probes /v1/sys/health without one, it marks every node unhealthy and pulls the whole cluster out of rotation. That reads as a total outage rather than as a TLS setting, and people will spend an hour looking at raft before anyone looks at the listener.

The SAN List Is the Real Contract

A certificate's Subject Alternative Name list (SAN, the set of hostnames and IP addresses that certificate is allowed to answer to) is the guest list at the door. The older Common Name field is a nickname that modern clients no longer read. Go dropped its Common Name fallback in version 1.15 and removed the escape hatch in 1.17, and the Vault command-line tool is a Go program, so a certificate carrying a Common Name and no SANs now fails against Vault's own tooling.

Everything a client might put in VAULT_ADDR has to appear in that list: the load balancer hostname, each node's own DNS name (Domain Name System, the phone book that turns names into addresses) if anything talks to nodes directly, and raw IP addresses as IP SANs rather than DNS entries wherever health checks or retry_join blocks connect by address. Add 127.0.0.1 if a local Vault Agent hits loopback. Wildcards cover exactly one label, so *.internal does not match vault-1.eu.internal. Miss one name and some callers succeed while others fail, which looks like a network flap for two days.

# what does the load balancer actually present, and for how long?
openssl s_client -connect vault.example.com:8200 -servername vault.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -dates -ext subjectAltName

subject=CN = vault.example.com
notBefore=Jul  1 00:00:00 2026 GMT
notAfter=Sep 29 23:59:59 2026 GMT
X509v3 Subject Alternative Name:
    DNS:vault.example.com, DNS:vault-1.internal, DNS:vault-2.internal, DNS:vault-3.internal, IP Address:10.20.0.11

# does the server actually send its intermediates, or only the leaf?
openssl s_client -connect vault.example.com:8200 -servername vault.example.com \
  -CAfile /etc/vault.d/tls/ca.pem </dev/null 2>&1 \
  | grep -E "verify error|Verify return code"

verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate
    Verify return code: 21 (unable to verify the first certificate)
 
# translation: the chain is incomplete. Rebuild vault-fullchain.pem as
# leaf + intermediate(s) in that order, then reload the listener.

When a name is missing, Go tells you exactly which one, which makes this one of the friendlier failures in the whole stack. Read the error literally. It prints the certificate's opinion of itself and the address you asked for, side by side. Then decide which side is wrong: reissue the certificate when the address is one clients are meant to use, and fix VAULT_ADDR when somebody dialed a node directly or by raw IP that should have gone through the balancer name. A SAN list that grows an entry every time a new address gets typed has stopped being a contract. The staging cluster below is the untended version of this. Its certificates came from a per-node template, so each node carries its own name and nothing else: no balancer name, no IP SAN.

# ask staging who it is: once through the load balancer, once by IP
vault status -address=https://vault-lb.example.com:8200
vault status -address=https://10.30.0.11:8200

Error checking seal status: Get "https://vault-lb.example.com:8200/v1/sys/seal-status":
tls: failed to verify certificate: x509: certificate is valid for vault-1.internal, not vault-lb.example.com
 
Error checking seal status: Get "https://10.30.0.11:8200/v1/sys/seal-status":
tls: failed to verify certificate: x509: cannot validate certificate for 10.30.0.11 because it doesn't contain any IP SANs

api_addr Is the Return Address on the Envelope

api_addr is the return address Vault writes on every envelope it sends: this node announcing "here is where I can be reached" to clients, to its peers, and to any tool that asks who the leader is. Nothing validates it. Vault will cheerfully advertise https://127.0.0.1:8200, and every caller that believes it dials its own loopback interface and gets connection refused from a machine that has never run Vault in its life.

The reason this bug survives so long is that it usually hides. By default a standby node forwards writes to the active node over the private cluster channel, so clients never see a redirect, and a wrong api_addr on a standby costs nothing at all. The value surfaces in three places: when that node becomes leader, when request forwarding is unavailable and Vault falls back to an HTTP 307 redirect, and whenever tooling reads sys/leader to find the active node. A landmine that goes off only during failover is worse than a constant error, because failover is exactly when nobody has spare attention.

cluster_addr is a different animal, and the difference catches out experienced people. It points at the private server-to-server channel, port 8201 by default, and raft replication does not use your listener certificate at all. Once a node has joined, the cluster speaks mutual TLS using certificates Vault generates for itself and keeps inside the encrypted barrier. Your tls_cert_file covers the API port and the initial join handshake, which is why retry_join offers a leader_ca_cert_file parameter: a joining node has to verify the leader's public certificate before it is trusted enough to receive the internal cluster credentials. Rotating your public certificate leaves raft replication undisturbed, and it can still stop a brand new node from joining if you forget to update that CA file.

# vault-2 was rebuilt from a stale template last week. Force a leader change
# and watch what the new active node advertises to the world.
vault operator step-down -address=https://vault-1.internal:8200
 
curl -sS --cacert /etc/vault.d/tls/ca.pem \
  https://vault-3.internal:8200/v1/sys/leader | jq

Success! Stepped down: https://vault-1.internal:8200
 
{
  "ha_enabled": true,
  "is_self": false,
  "leader_address": "https://127.0.0.1:8200",
  "leader_cluster_address": "https://vault-2.internal:8201",
  "performance_standby": false,
  "performance_standby_last_remote_wal": 0
}

Nothing here is broken in a way any dashboard would notice. Raft is committing, vault status reports Sealed false on all three nodes, autopilot calls the cluster healthy. And every client that follows the advertised leader address is dialing its own machine. Fix the value in /etc/vault.d/vault.hcl and restart that node, because api_addr is read once at startup and a reload signal will not touch it.

Pick a Seal, and Own Its Failure Mode

Every secret in Vault is protected by an encryption key that lives inside the encrypted barrier, and that encryption key is itself wrapped by the root key. The seal is whatever guards the root key while Vault is switched off. With the default Shamir seal, the key that unwraps the root key is cut into shares held by different people, like a safety deposit box that needs three branch managers standing at the door with three different keys. With auto-unseal, the root key sits in storage in wrapped form, and the only thing that can unwrap it is a key held somewhere else: AWS KMS (Key Management Service, Amazon's managed key store), Google Cloud KMS, Azure Key Vault, or a transit secrets engine running on a separate Vault.

The boot sequence is short. A node starts sealed, reads the wrapped root key out of raft storage, calls the KMS Decrypt API, gets the root key back, opens the barrier, rejoins the cluster and starts serving. Nobody gets woken up. That is the difference between a rolling operating system patch you schedule for Tuesday lunchtime and one you schedule for a Saturday with three key holders on a bridge call.

Say the trade-off out loud in the design review. You swapped a human dependency for a cloud dependency, and KMS availability is now Vault availability. The nastier part is that the failure is delayed. Removing kms:Decrypt from the instance role does not disturb a running Vault, because the root key is already sitting in memory. It breaks the next reboot, possibly six weeks later, during an unrelated machine image rollout that nobody connected to Vault. That gap between cause and symptom is why KMS policy changes belong in the same change-control lane as Vault's own configuration.

A security payoff hides in the same place, and it points straight at your least guarded path. Anyone who holds kms:Decrypt on that seal key and can also read a raft snapshot owns your entire secret store. That attack never touches Vault's API, never authenticates, and never appears in any audit device you have enabled. Scope the KMS key policy to the Vault instance role and nothing else, grant kms:Encrypt, kms:Decrypt and kms:DescribeKey only, and alert on CloudTrail Decrypt calls from principals you did not expect.

# --- Option A: cloud KMS, the common choice ---
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234-ef56-7890"
 
  # credentials come from the instance role; never hardcode access_key here.
  # Route through a VPC endpoint so unsealing never needs the public internet:
  # endpoint = "https://vpce-0abc123-xyz.kms.us-east-1.vpce.amazonaws.com"
}
 
# --- Option B: a transit mount on a separate Vault, no cloud dependency ---
# seal "transit" {
#   address         = "https://vault-seal.internal:8200"
#   token           = "hvs...."        # or VAULT_TOKEN via the unit EnvironmentFile
#   key_name        = "autounseal"
#   mount_path      = "transit/"
#   disable_renewal = "false"
#   tls_ca_cert     = "/etc/vault.d/tls/seal-ca.pem"
# }
 
# Permissions Vault needs on the KMS key, and nothing more:
#   kms:Encrypt, kms:Decrypt, kms:DescribeKey

Recovery Keys Are a Signature, Not a Key

Recovery keys look identical to unseal keys. Same base64 soup, same ceremony of printing them on cards for different people. They do a completely different job. An unseal key is a fragment of the material that opens the barrier. A recovery key is closer to a notary's signature: proof that a quorum of humans approved a privileged operation. The split is mechanical rather than a policy someone chose. When you initialize with a seal stanza, the root key is wrapped by the KMS key, and the recovery shares are never part of that wrapping, so there is nothing inside them for Vault to reassemble.

Learn what they do authorize before the night you need it: minting a new root token with vault operator generate-root, rotating the recovery set with vault operator rekey -target=recovery, and carrying a cluster through a seal change with vault operator unseal -migrate. You size the set at initialization with -recovery-shares and -recovery-threshold. Reaching for -key-shares while a seal stanza is configured is the classic day-one mistake, and it fails loudly, which is a mercy.

# ONE node, ONE time, with the seal stanza already in the config file
vault operator init -recovery-shares=5 -recovery-threshold=3

Recovery Key 1: IWFFX9K8ot9Mirii5nxA8iWYP4Sv7r6OUyPxPpWSXolp
Recovery Key 2: 3b7hfLAc+A778EsKIPI9c6bUd5hVyOivgMcRF89Ce++R
Recovery Key 3: Uox5m+JQbFnQTFTWSV3n5pyycBz1BfEbtSjA6E2V40FM
Recovery Key 4: MeD2wsJmCG5CPMX/ERN9Uwj9qDoFrYF+L8dSx/BpHE82
Recovery Key 5: syGIEWNIKDlvYdeHJHMWqMHRSb3LnlTtFxvbKKe3A/4B
 
Initial Root Token: hvs.oeRIhQy9AYgSpuJCSRgDxgeP
 
Success! Vault is initialized
 
Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

vault status

Key                      Value
---                      -----
Seal Type                awskms
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.20.4
Storage Type             raft
Cluster Name             vault-cluster-prod
HA Enabled               true
HA Cluster               https://vault-1.internal:8201
HA Mode                  active
Active Since             2026-07-20T08:12:01.482Z
Raft Committed Index     4182
Raft Applied Index       4182

Two lines in that output are worth committing to memory. Seal Type awskms says a cloud key holds the thing that opens this vault. Recovery Seal Type shamir says a recovery set exists, split five ways, and it authorizes operations rather than opening doors. A cluster showing Seal Type shamir with no recovery lines at all is a Shamir cluster, and there the envelopes in your safe really do open the door. Write down which one you run on the cover page of the runbook, because nobody reads page four during an outage.

Here is what recovery keys genuinely buy you. Someone deleted the last policy binding that granted admin rights, or the root token from initialization went missing years ago. generate-root runs a threshold ceremony against the recovery set and returns a token encoded with a one-time password (OTP, a short throwaway secret printed at the start of the ceremony), so no single terminal ever sees the raw root token in flight.

# start the ceremony; note the nonce and the OTP it prints
vault operator generate-root -init
 
# each key holder runs this on their own machine and pastes their recovery key
vault operator generate-root -nonce=5827a3f9-2d1b-4c66-9d7e-8a0f1e2c3b44
 
# once the threshold is met, decode the result with the OTP from step one
vault operator generate-root -decode=<encoded-token> -otp=<otp>

Nonce         5827a3f9-2d1b-4c66-9d7e-8a0f1e2c3b44
Started       true
Progress      0/3
Complete      false
OTP           BMjefDrkFsbFRXaGqAPYd0Xs4d
OTP Length    26

Custody rules for recovery shares match Shamir shares exactly: different humans, different physical locations, no two shares living in one password manager. Encrypt them at initialization with -recovery-pgp-keys (PGP, Pretty Good Privacy, the long-standing standard for encrypting things to a named person) so that a single terminal scrollback never holds all five in clear text. When a holder leaves the company, rotate the set with vault operator rekey -target=recovery -init -key-shares=5 -key-threshold=3. The default target is barrier, which means something only on a Shamir cluster, so forgetting that flag on an auto-unsealed Vault costs you a confusing five minutes.

Prove It Works, Then Rotate Without an Outage

A control you cannot verify is a belief. vault status returns exit code 0 when the node is unsealed, 2 when it is sealed, and 1 on error, which makes a clean monitoring probe with no output parsing. Run it against each node's own address rather than through the load balancer, because a healthy balancer quietly removes a sealed node from rotation and hides the exact machine you need to hear about. Alert on exit code 2 lasting longer than a normal boot, and alert separately on KMS Decrypt errors in the Vault log, since those show up before anybody reboots into them.

Certificate rotation is the operation people fear most, and it turns out to be low risk once you know the mechanism. SIGHUP (the hang-up signal, a polite nudge that tells a running process to re-read its configuration) reloads the listener certificate and key with no restart, no re-seal and no leadership change. systemctl reload vault sends that signal on the standard systemd unit, which maps reload to kill -HUP. Two limits worth writing down: api_addr, the listener address and the seal stanza are not applied by a reload and need a real restart, and Windows has no equivalent signal, so plan a restart there. One node at a time, autopilot healthy before you touch the next.

# 1. how much runway is left on the current leaf?
openssl x509 -in /etc/vault.d/tls/vault-fullchain.pem -noout -enddate
 
# 2. write the renewed leaf+chain and key to the SAME paths, then reload
sudo systemctl reload vault        # SIGHUP: no restart, no re-seal
 
# 3. confirm the live cert changed and the node never left the cluster
openssl s_client -connect vault-1.internal:8200 -servername vault-1.internal </dev/null 2>/dev/null \
  | openssl x509 -noout -enddate
vault operator raft autopilot state

notAfter=Sep 29 23:59:59 2026 GMT
 
notAfter=Dec 28 23:59:59 2026 GMT
 
Healthy:                         true
Failure Tolerance:               1
Leader:                          vault-1
Voters:
   vault-1
   vault-2
   vault-3

Try This

Two lab exercises, both of them failures you want to have seen once before you see them for real. First, set api_addr = "https://127.0.0.1:8200" on one standby node, restart it, and confirm that everything still looks perfect from every angle. Then run vault operator step-down against the leader until that broken node takes over, and watch sys/leader advertise loopback to the entire cluster while vault status cheerfully reports active and healthy on all three.

Second, break the seal on purpose. Remove kms:Decrypt from one node's instance role, restart Vault on that node, and read what comes back.

sudo systemctl restart vault
vault status -address=https://vault-3.internal:8200 ; echo "exit code: $?"
sudo journalctl -u vault -n 20 --no-pager | grep -i unseal

Key                      Value
---                      -----
Seal Type                awskms
Initialized              true
Sealed                   true
Total Recovery Shares    5
Threshold                3
Unseal Progress          0/3
Unseal Nonce             n/a
Version                  1.20.4
Storage Type             raft
HA Enabled               true
exit code: 2
 
[WARN]  core: failed to unseal core: error="failed to decrypt encrypted stored keys:
  error decrypting seal wrapped value: operation error KMS: Decrypt, https response error
  StatusCode: 400, api error AccessDeniedException: User:
  arn:aws:sts::111122223333:assumed-role/vault-server/i-0abc123def is not authorized to
  perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:111122223333:key/abcd-1234"

Now try your recovery keys against that node and watch them get rejected, which is the whole point of the exercise. Then restore the IAM permission, restart, and confirm the node unseals on its own within seconds. Time both halves and write the numbers down. The difference between them is how long a seal outage lasts once somebody reads the log line instead of hunting for envelopes.

Takeaway

Listener, api_addr and seal fail as a set, so test them as a set. Dial every address a client might reasonably use and confirm each one is on the certificate's SAN list. Force a leader change and read what sys/leader hands out. Pull kms:Decrypt from one node and restart it, so you have watched an auto-unsealed node sit at Sealed true before an outage shows it to you. None of the three announces itself on a dashboard, which is why an afternoon spent breaking them on purpose is the cheapest hour in the build.

Next up: the corridor is encrypted and the doors open cleanly on reboot, so the question becomes who gets to walk through them. OIDC for humans, Kubernetes auth for workloads.

Related