KV v2 versioning and check-and-set

Static secrets without stampeding them.

A payments engineer rotates the Stripe key at 4:00 p.m. Three minutes later a platform engineer pastes the old key back, working from a terminal tab that has been open since lunch, because the old value is what their screen still showed. Nothing errors. No alert fires. Card charges start failing forty minutes later, and the only surviving copy of what the key used to be is sitting in somebody's clipboard history. That is an ordinary Tuesday for a team running a shared secret store the way they would run a network file share.

KV version 2 is Vault's answer to that afternoon. KV is short for key-value, the same shape as a coat check ticket: a short label maps to the one thing you want back later. Version 2 behaves like a shared document with revision history switched on rather than a file you overwrite. Every save is kept and numbered. You can open last Thursday's copy without asking anyone. And when two people write at the same moment, the store can refuse the second write and tell it that the world moved. Vault calls that last feature check-and-set, or CAS: a conditional write that lands only if the version you believed was current really is.

Be honest about what that buys you. Versioning does not make a stored secret safe. A leaked API key (application programming interface key, the long random string one service shows another to prove who it is) is leaked whether or not Vault kept twelve copies of it. What you get is a provable rollback, a timestamped record of who changed what, and a mechanism that stops one careless write from erasing a value another team depends on. What you pay is that every superseded secret is still sitting in Vault, readable by anyone whose policy allows the path, until you deliberately erase it. Every version you keep is a version an attacker with a stolen token can read.

KV is for material Vault cannot mint on its own: third-party API keys, partner credentials, a license string for a vendor appliance from 2014. If a credential can be generated on demand and expire by itself, reach for the database or cloud secrets engines instead. A stored secret is a secret you have to remember to rotate.

Turn It On And Watch The Version Counter Move

Most Vault installs already have a KV v2 mount at secret/. If you are creating one, name the engine type explicitly. The older spelling was vault secrets enable -path=secret -version=2 kv; the current shorthand is kv-v2, and both produce the same mount. One detail trips up nearly everyone on day one: the path you type is not the path Vault stores at. Writing to secret/payments/stripe actually hits secret/data/payments/stripe. The CLI (command-line interface, the vault binary you type into) inserts that data/ segment for you without mentioning it. Your policies will not.

vault secrets enable -path=secret kv-v2
vault kv put -mount=secret payments/stripe api_key=sk_live_4eC39H

Success! Enabled the kv-v2 secrets engine at: secret/
 
== Secret Path ==
secret/data/payments/stripe
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:12:44.113472Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

The -mount=secret flag tells the CLI where the mount ends and the key name begins. Without it, vault kv put secret/payments/stripe still works, but the CLI has to guess the split, and it guesses badly on nested mounts like secret/team/prod. Build the -mount habit now and your commands survive somebody reorganizing the mount tree. Notice the header Vault printed back: it names the real storage path, which is the one your policy has to match.

Write a second value, then read the first one back. Nothing was overwritten. Version 1 is still there, and -version=1 fetches it.

vault kv put -mount=secret payments/stripe api_key=sk_live_9xQ71B
vault kv get -mount=secret -version=1 payments/stripe

== Secret Path ==
secret/data/payments/stripe
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:18:02.884201Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2
 
== Secret Path ==
secret/data/payments/stripe
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:12:44.113472Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
===== Data =====
Key        Value
---        -----
api_key    sk_live_4eC39H

Check-And-Set Is A Seatbelt Somebody Has To Buckle

When two people edit the same shared document, a decent editor stops the slower save and says the page changed while you were typing. CAS is that prompt, expressed as a number. -cas=N means accept this write only if the current version is exactly N. If somebody slipped a write in between your read and your write, N is stale and Vault rejects you instead of quietly stacking your value on top of theirs. One special value: -cas=0 means write only if this key has never existed, which is how you create a secret with no risk of clobbering one.

# our terminal still thinks version 1 is current, but someone wrote version 2
vault kv put -mount=secret -cas=1 payments/stripe api_key=sk_live_stale_paste

Error writing data to secret/data/payments/stripe: Error making API request.
 
URL: PUT https://vault-1.internal:8200/v1/secret/data/payments/stripe
Code: 400. Errors:
 
* check-and-set parameter did not match the current version

That 400 is the 4:03 p.m. incident, caught. The catch is that CAS is off by default, and a seatbelt nobody buckles saves nobody. You can make it mandatory for a whole mount by writing to the mount's own config path, or for a single key with vault kv metadata put -cas-required=true. Mount-wide is the setting worth defending in a design review, because it also covers the keys nobody has created yet.

vault write secret/config cas_required=true
vault read secret/config
vault kv put -mount=secret payments/stripe api_key=sk_live_no_cas

Success! Data written to: secret/config
 
Key                     Value
---                     -----
cas_required            true
delete_version_after    0s
max_versions            0
 
Error writing data to secret/data/payments/stripe: Error making API request.
 
URL: PUT https://vault-1.internal:8200/v1/secret/data/payments/stripe
Code: 400. Errors:
 
* check-and-set parameter required for this call

Two different error strings, two different problems. "did not match the current version" means you raced someone. "required for this call" means you forgot the flag. Once cas_required is on, every writer has to read the current version first, which is a two-step your pipelines now have to perform on purpose. That is the point of it. A rotation job that cannot be bothered to look at the current state is a rotation job that will one day flatten an emergency fix.

CUR=$(vault kv metadata get -mount=secret -format=json payments/stripe \
        | jq -r .data.current_version)
vault kv put -mount=secret -cas="$CUR" payments/stripe api_key=sk_live_9xQ71B

== Secret Path ==
secret/data/payments/stripe
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:26:41.006123Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            3

While you are in the metadata, use it as an inventory. Custom metadata attaches to the key rather than to any single version, so an owner and a rotation date survive every write. Put the on-call team, the change ticket and the date the secret goes stale in there, and your rotation reminders come from Vault instead of a spreadsheet nobody has opened since March.

vault kv metadata put -mount=secret \
  -max-versions=5 \
  -custom-metadata=owner=payments-team \
  -custom-metadata=rotate_after=2026-10-01 \
  payments/stripe
vault kv metadata get -mount=secret payments/stripe

Success! Data written to: secret/metadata/payments/stripe
 
========== Metadata ==========
Key                     Value
---                     -----
cas_required            false
created_time            2026-07-27T09:12:44.113472Z
current_version         3
custom_metadata         map[owner:payments-team rotate_after:2026-10-01]
delete_version_after    0s
max_versions            5
oldest_version          0
updated_time            2026-07-27T09:31:55.201884Z
 
====== Version 1 ======
Key              Value
---              -----
created_time     2026-07-27T09:12:44.113472Z
deletion_time    n/a
destroyed        false
 
====== Version 2 ======
Key              Value
---              -----
created_time     2026-07-27T09:18:02.884201Z
deletion_time    n/a
destroyed        false
 
====== Version 3 ======
Key              Value
---              -----
created_time     2026-07-27T09:26:41.006123Z
deletion_time    n/a
destroyed        false

Read that output slowly and one line looks wrong: cas_required is false on a mount where you switched it on a few commands ago. Both settings are real and they live in different places. The value you wrote to secret/config applies to the whole mount. The field printed here is the key's own override, and false means fall back to the mount, which is set to true. Check the key alone and you will conclude the enforcement is off when it is very much on, so read both before you decide anything during an incident.

Delete, Destroy, And The Recycle Bin People Assume Exists

There are three different things an operator can mean by "delete that secret," and Vault implements all three as separate API paths with separate permissions. Soft delete is the recycle bin: the value stays on disk, marked with a deletion timestamp, and one command brings it back. Destroy is the shredder: the bytes of that version are erased and the version number stays behind as a tombstone. Metadata delete burns the folder and the index card together, removing every version and the history in one call. If your runbook says "delete the compromised key" without naming which one, half your team will pick the wrong one under pressure.

Watch what a soft delete looks like on a read, because the output surprises people expecting a plain not-found.

vault kv put -mount=secret -cas=0 lab/demo password=one  > /dev/null
vault kv put -mount=secret -cas=1 lab/demo password=two  > /dev/null
vault kv delete -mount=secret lab/demo
vault kv get    -mount=secret lab/demo

Success! Data deleted (if it existed) at: secret/data/lab/demo
 
== Secret Path ==
secret/data/lab/demo
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:41:20.331Z
custom_metadata    <nil>
deletion_time      2026-07-27T09:41:30.918Z
destroyed          false
version            2

There is no Data section, deletion_time is filled in, and destroyed is still false. Vault is telling you plainly that the value exists and is being withheld. An application calling the same endpoint over HTTP gets a 404 with that metadata in the body, which is why a soft delete looks like an outage to your service and looks like nothing at all in your compliance evidence. The underlying data was never touched, so a later write still has to pass the current version to CAS as though the delete had not happened. Undelete puts it straight back.

vault kv undelete -mount=secret -versions=2 lab/demo
vault kv get      -mount=secret lab/demo

Success! Data written to: secret/undelete/lab/demo
 
== Secret Path ==
secret/data/lab/demo
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:41:20.331Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2
 
===== Data =====
Key         Value
---         -----
password    two

Destroy is the one with no undo. Run it against version 1 and read that version back: the tombstone survives, the bytes do not.

vault kv destroy -mount=secret -versions=1 lab/demo
vault kv get     -mount=secret -version=1 lab/demo

Success! Data written to: secret/destroy/lab/demo
 
== Secret Path ==
secret/data/lab/demo
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:41:12.550Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          true
version            1

When you want to undo a bad write rather than hide it, vault kv rollback -version=N is the honest move. It reads version N and writes those contents back as a brand new version, so the history shows a correction instead of a gap. It also sends the current version as a CAS parameter on your behalf, which means it fails if somebody wrote while you were deciding, and it keeps working on a mount with cas_required=true for exactly that reason.

One more thing about destroy that catches teams during an audit: it does not reach into your backups. Take a Raft snapshot on Sunday, destroy version 4 on Monday, restore Sunday's snapshot on Tuesday after a bad upgrade, and version 4 is back with its value intact. That is correct behavior for a backup and a nasty surprise for anyone who told a regulator the value was erased. The contents of a snapshot are still encrypted by the cluster's barrier keys, so the file is not a plaintext dump, but it restores into any cluster that can unseal with the same key material, which means the snapshot bucket deserves the protection you give Vault itself. Rehearse a restore followed by a read of an old version, so you know what your runbook actually promises.

Policies Split Along Those Prefixes, Not Along Intent

This is where the security control either exists or does not. A policy written in HCL (HashiCorp Configuration Language, the syntax Vault uses for policies and config files) matches literal request paths. A rule for secret/payments/stripe grants nothing at all, because no request is ever sent there. The reading application and the rotation job need different capabilities, and neither of them should be able to shred history. These are three separate policy files, shown together so you can see how they fit.

# payments-app.hcl -- what the running service gets
path "secret/data/payments/stripe" {
  capabilities = ["read"]
}
 
# it may read ownership and rotation notes, never change them
path "secret/metadata/payments/stripe" {
  capabilities = ["read"]
}
 
# ---------------------------------------------------------
# payments-rotator.hcl -- what the nightly rotation job gets
path "secret/data/payments/stripe" {
  capabilities = ["read", "create", "update"]
}
path "secret/metadata/payments/stripe" {
  capabilities = ["read", "update"]
}
 
# ---------------------------------------------------------
# shared-guardrails.hcl -- attached to both roles
# irreversible operations belong to a human with dual control
path "secret/destroy/*" {
  capabilities = ["deny"]
}
path "secret/metadata/*" {
  capabilities = ["deny"]
}
path "secret/delete/*" {
  capabilities = ["deny"]
}

Those files look like they contradict each other, and understanding why they do not is worth two minutes. Vault ranks every rule that could match a request and applies exactly one of them, the most specific. A path with no glob outranks a path ending in *. So secret/metadata/payments/stripe keeps its read grant even though secret/metadata/* says deny, and every other key under metadata stays blocked. Deny outranks other capabilities when two policies define the same path, which is not what is happening here. Guess at this rather than testing it and you will ship a policy that either locks out your own application or leaves destroy wide open. Test it with the token itself.

vault token capabilities "$APP_TOKEN" secret/data/payments/stripe
vault token capabilities "$APP_TOKEN" secret/metadata/payments/stripe
vault token capabilities "$APP_TOKEN" secret/metadata/payments/other
vault token capabilities "$APP_TOKEN" secret/destroy/payments/stripe

read
read
deny
deny

Four lines, and the control is proven rather than assumed. Those commands use your admin token to ask Vault what the application's token can do, so put them in your policy test suite and a future edit that widens secret/* trips a red build instead of a red incident review. Pay attention to the list capability while you are there: a token with list on secret/metadata/ can enumerate every key name in the mount without reading a single value, and names like payments/stripe-live-backup-DO-NOT-DELETE are a map of your infrastructure handed to whoever holds that token.

Stop The Read Stampede With Vault Agent

Three hundred pods roll at once after a deploy and every one of them shells out to vault kv get on startup. In a high availability cluster on integrated storage (Raft, the built-in replicated log that keeps every node's copy of the data in step), standby nodes forward client requests to the active node, so your carefully balanced load lands on one server anyway. That node handles three hundred logins and three hundred reads in a four-second window, your login rate limit quota starts answering 429, and half the fleet crash-loops. Meanwhile every pod that did succeed holds the plaintext key in an environment variable for the life of the process, where a crash dump, a debug endpoint or anything that can read /proc/PID/environ picks it up for free.

Vault Agent turns that into one login and one read per node, refreshed on a timer, written to a file with tight permissions. The application never speaks to Vault and never handles a token.

vault {
  address = "https://vault.internal:8200"
}
 
auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/etc/vault-agent/role_id"
      secret_id_file_path = "/run/vault-agent/secret_id"
      # this is the default; kept explicit so nobody tidies it away
      remove_secret_id_file_after_reading = true
    }
  }
 
  sink "file" {
    config = {
      path = "/run/vault-agent/token"
      mode = 0640
    }
  }
}
 
template_config {
  # KV reads carry no lease, so Agent re-reads on a timer instead of
  # renewing the way it would with a database credential. 5m is the default.
  static_secret_render_interval = "5m"
}
 
template {
  source               = "/etc/vault-agent/stripe.env.ctmpl"
  destination          = "/run/secrets/stripe.env"
  perms                = "0400"
  error_on_missing_key = true
 
  # the bare `command = "..."` string form is deprecated; use exec
  exec {
    command = ["systemctl", "reload", "payments-api"]
    timeout = "30s"
  }
}

{{- with secret "secret/data/payments/stripe" }}
STRIPE_API_KEY={{ .Data.data.api_key }}
{{- end }}

.Data.data.api_key reads like a typo the first few times. It is not. The API response wraps your key-value pairs in a second data object, so the word turns up once in the request path and once in the response body. Write .Data.api_key, the KV v1 habit, and the template renders an empty value without complaining. That is what error_on_missing_key = true is for: it turns a silently empty secret into a template failure you see at deploy time instead of a payment gateway rejecting every request at 2 a.m.

The trade-off is real and you should say it out loud in review. Agent swaps a secret in memory for a secret on disk. A file at /run/secrets/stripe.env with mode 0400 lives on tmpfs, disappears on reboot and is readable by exactly one user, which is stronger than an environment variable inherited by every child process. It is still a file, and anyone with code execution as that user reads it. The five-minute render interval is the other trade: rotate a key and up to five minutes of requests still carry the old one, so plan rotations as overlap windows where both values work.

Prove Somebody Would Notice

A control you cannot verify is a belief. Enable a file audit device and every request and response gets a line, with sensitive values replaced by an HMAC (hash-based message authentication code, a one-way fingerprint computed with a salt only Vault holds). You cannot read a secret out of the log, which is the point, and you can still ask Vault to fingerprint a value you already know so you can search for it.

vault audit enable file file_path=/var/log/vault/audit.log
vault write -field=hash sys/audit-hash/file input="sk_live_4eC39H"

Success! Enabled the file audit device at: file/
 
hmac-sha256:2e4b7c1f0a9d6e35b8c47f0d2a1e93b6f8c5d0e2a7b41c93f6e08d5a2c7b1e40

That hash answers the question "did this leaked key ever come out of Vault, and through whom." Search the audit log for the fingerprint and you find every request that carried that exact value. Query by path instead and you get the read history of the secret, with the auth method and role attached to every line. Request paths are written in the clear; the values are not.

jq -r 'select(.request.path == "secret/data/payments/stripe")
       | [.time, .type, .request.operation, .auth.metadata.role_name] | @tsv' \
  /var/log/vault/audit.log | tail -4

2026-07-27T09:26:41.006123456Z	request	update	payments-rotator
2026-07-27T09:26:41.041882301Z	response	update	payments-rotator
2026-07-27T09:44:07.552117490Z	request	read	payments-app
2026-07-27T09:44:07.559043118Z	response	read	payments-app

The operation reads update rather than create because the key already existed. Vault runs an existence check before it writes the audit line, so a brand new key logs as create and every write after that logs as update. It is a small detail that matters when you are proving to an auditor that a secret was changed rather than created.

Now the honest part. Audit devices are blocking by design: if every enabled device fails to write its line, Vault stops answering requests rather than operating unaudited. A full disk on the audit volume takes your cluster down as surely as a lost quorum does. That is the right default for a system of record, and it means the audit volume needs its own free-space alert, its own log rotation, and ideally a second device such as syslog or a socket, so one wedged filesystem cannot stop the world.

Try This

Ten minutes against a lab cluster and you will have felt every failure mode above. Create a key with a create-only write, add a second version, race yourself on purpose with a stale CAS number, then roll back to the original value and confirm the counter moved forward rather than backward.

vault kv put -mount=secret -cas=0 lab/cas-drill password=one   > /dev/null
vault kv put -mount=secret -cas=1 lab/cas-drill password=two   > /dev/null
vault kv put -mount=secret -cas=1 lab/cas-drill password=three   # stale on purpose
vault kv rollback -mount=secret -version=1 lab/cas-drill
vault kv get -mount=secret -field=password lab/cas-drill

Error writing data to secret/data/lab/cas-drill: Error making API request.
 
URL: PUT https://vault-1.internal:8200/v1/secret/data/lab/cas-drill
Code: 400. Errors:
 
* check-and-set parameter did not match the current version
 
== Secret Path ==
secret/data/lab/cas-drill
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T10:02:55.311Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            3
 
one

Version 3 now holds the contents of version 1, the failed write left no trace in the data and a clear trace in the audit log, and nothing was overwritten anywhere along the way. Repeat the drill with a token whose policy grants read but not update on that path, and the rollback fails at the write step while the reads keep working, which is exactly the split you want in production.

Takeaway

The trap worth remembering here: max_versions 0 does not mean unlimited. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related