Snapshots, upgrades, and DR

Raft snapshots and rehearsed recovery.

At 14:20 on a Thursday, an engineer runs vault secrets disable database/ against what they are certain is staging. VAULT_ADDR still points at production. Four hundred applications lose their database credentials, every lease from that mount is revoked, and the mount configuration is gone. A cron job has been writing Raft snapshots to a bucket every night for eighteen months, which sounds like the happy ending. Nobody in the room has ever restored one, and nobody knows yet that the restore needs a key living in a different cloud account.

A backup you have never restored is a photograph of a spare tire. It looks like the thing that saves you. Raft, the agreement protocol behind Vault's built-in storage where a majority of nodes must sign off before a write counts, keeps a full live copy of your data on every voting node, and teams quietly file that under "backup". It is not one. Replication saves you from a dead disk, a dead node, a rack losing power. It also copies a mistaken secrets disable to all three nodes faster than you can lift your finger off the return key.

This lesson closes the production loop: snapshots that are genuinely consistent, a backup job carrying the smallest token that can do the work, storage custody fit for key material, a restore rehearsed with a stopwatch running, and a version upgrade that never drops the cluster below quorum. You will also meet the failure that catches almost everyone on their first real restore, which is a perfectly good snapshot installed into a cluster that cannot unseal it.

What Is Actually Inside A Snapshot

A Raft snapshot is a photocopy of the whole filing cabinet, taken while people are still using it, including the locked drawer and the lock itself. In Vault terms that means every secrets engine mount, every policy, every auth method configuration, your KV v2 (key/value secrets engine, version 2) data with its version history, identity entities and groups, every token, and every lease record. It also carries the encrypted barrier keyring and the seal-wrapped root key, and that last item decides whether a restore works at all.

The file is ciphertext from end to end, so a stolen snapshot does not hand an attacker readable secrets. It hands them an offline copy of your entire secret plane and unlimited time to work on the lock. Someone holding a snapshot plus enough Shamir shares, or a snapshot plus permission to call the right KMS (key management service, the cloud service that holds an encryption key you can use but never download) key, rebuilds every secret you own without ever touching your network again.

Take the snapshot through the API (application programming interface, the machine-facing door into Vault) from the active node, while Vault is running. Do not tar up /opt/vault/data on a live server and call it a backup. The snapshot endpoint hands you a consistent point in time built from committed Raft state, while a file copy gives you a Bolt database (the embedded key/value file Vault keeps on disk) caught mid-write. Standby nodes forward the request to the active node, but the reply is one long stream and load balancers sitting in the middle of it have surprised people, so point VAULT_ADDR at the leader and take the doubt off the table.

export VAULT_ADDR=https://vault-1.internal:8200
vault operator raft list-peers
vault operator raft snapshot save /var/backups/vault/raft-$(date +%F-%H%M).snap
ls -lh /var/backups/vault/

Node       Address                  State       Voter
----       -------                  -----       -----
vault-1    vault-1.internal:8201    leader      true
vault-2    vault-2.internal:8201    follower    true
vault-3    vault-3.internal:8201    follower    true
 
# snapshot save prints nothing at all when it works
total 43M
-rw------- 1 vault vault 43M Jul 27 02:00 raft-2026-07-27-0200.snap

That silence is a trap for cron jobs. vault operator raft snapshot save prints nothing on success and exits with a non-zero status on failure, so a backup script has to check the exit code, then check the file size, then page a human when either one looks wrong. A zero-byte snapshot written faithfully every night for a year is an expensive kind of quiet.

Inspect the file rather than trusting it. snapshot inspect reads a snapshot locally, with no running cluster involved, and tells you the Raft log position it was captured at along with a rough map of what is inside.

vault operator raft snapshot inspect /var/backups/vault/raft-2026-07-27-0200.snap

 ID           bolt-snapshot
 Size         44695552
 Index        184213
 Term         9
 Version      1
 
 Key Name                                          Count    Size
 ----                                              -----    ----
 sys/expire                                         9214    21.4MB
 sys/token                                          3877    9.1MB
 logical/9c2b7a41-3c2e-4e0e-9f6b-7f0a2d1c55e8        812    4.3MB
 logical/4424d327-7320-7eed-6955-7cf9554ab30e        407    1.9MB
 auth/ba6064c0-7b95-c9ab-42b5-59139a68d169            96    38KB
 sys/policy                                           41    62KB
 core/cluster                                          2    236B

Index is the Raft log position at the moment of capture, so an index that has barely moved since yesterday means Vault took almost no writes: a very quiet cluster, or a broken one. Size gives you a free trend line, and a snapshot that suddenly shrinks by ninety percent is an alarm you want at 02:05 rather than at 14:20 on a Thursday. In the breakdown, sys/expire is lease bookkeeping and sys/token is exactly what it sounds like, which is why those two dominate a busy cluster. Each logical/ entry is one secrets engine mount and each auth/ entry is one auth method, listed under the internal identifier Vault assigned the mount rather than the path you know it by. Add -depth if you want the breakdown to go finer. Compare the shape against last week instead of against a fixed list, because it shifts every time you mount something new.

Give The Backup Job Read, Not Write

Your nightly backup runner should never hold a token that can overwrite Vault. One path does double duty: an HTTP GET (a plain read request) on sys/storage/raft/snapshot downloads a snapshot, and an HTTP POST (a write request) to the same path installs one. In policy terms that is read against update, so the distance between a backup account and a doomsday account is a single word in an HCL (HashiCorp Configuration Language) file. Adding -force to a restore sends the write to sys/storage/raft/snapshot-force instead, so a policy written with care names both paths and grants neither.

# Nightly snapshot runner. Read only: it can download the cluster, never replace it.
path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}
 
# Deliberately absent. These two live in a separate break-glass policy that only
# the on-call operator role can be granted, and only during a declared incident:
#   path "sys/storage/raft/snapshot"       { capabilities = ["update"] }
#   path "sys/storage/raft/snapshot-force" { capabilities = ["update"] }

vault policy write vault-backup /etc/vault.d/policies/vault-backup.hcl
vault write auth/approle/role/vault-backup \
  token_policies=vault-backup token_ttl=15m token_max_ttl=30m secret_id_ttl=24h
 
# now prove the limit, using a token issued to that AppRole
VAULT_TOKEN="$BACKUP_TOKEN" vault operator raft snapshot restore /tmp/raft.snap

Success! Uploaded policy: vault-backup
Success! Data written to: auth/approle/role/vault-backup
 
Error installing the snapshot: Error making API request.
 
URL: POST https://vault-1.internal:8200/v1/sys/storage/raft/snapshot
Code: 403. Errors:
 
* 1 error occurred:
	* permission denied

That 403 is the control working. Run it the day you write the policy and again after anyone edits it, because policy drift is silent and nothing warns you when a tired engineer widens ["read"] to ["read","update"] at midnight. The blast radius you are containing here is specific. A compromised build runner holding a restore-capable token can roll your entire secret plane back to an older state, quietly reinstating every credential you revoked and every policy you tightened since that snapshot was taken, and the whole thing reads like routine maintenance in the audit log.

Retention is a security decision wearing a storage-cost costume. An attacker who plants a backdoor policy in March and is caught in June has poisoned every snapshot taken after March, so a rolling seven days of dailies leaves you nothing clean to go back to. Thirty dailies plus twelve weeklies is a sane starting shape, and the weeklies are the part that saves you from the patient attacker.

Be honest about your RPO (recovery point objective, the most data you are willing to lose, measured in time). A 02:00 nightly snapshot means a worst case of nearly a full day of writes gone. For static key/value data that is irritating. For dynamic secrets it gets stranger: restoring an old snapshot rewinds Vault's lease bookkeeping but does nothing to the systems those leases point at. Credentials Vault issued after the snapshot still exist in your database while the restored Vault holds no record of them, so nothing ever revokes them. Orphan accounts, real access, no expiry. Going the other way, leases inside the snapshot may name database users that were already dropped, so revocation fails loudly. Every restore runbook needs a reconciliation step that lists database users matching your Vault role prefix and removes the ones no live lease accounts for.

Restore Is Not Undo

Restore replaces the cabinet; it does not undo a drawer. vault operator raft snapshot restore installs the entire snapshot over the running cluster's data. There is no partial restore, no way to bring back a single mount, and no confirmation prompt. Like save, it prints nothing on success, which is a strange feeling the first time you run the most destructive command in your toolbox.

The first surprise lands a second later, when your own token stops working. Tokens live in Vault's storage, so a snapshot taken before you logged in does not contain yours. The restore succeeds and your next vault kv get comes back with permission denied. Work out in advance which login method existed at snapshot time and which humans can use it, because figuring that out mid-incident costs twenty minutes you do not have.

The Seal Follows The Data

The second surprise is bigger, and it is where most first restores die. Vault encrypts everything in storage with a barrier keyring, the keyring is encrypted with the root key, and the root key is wrapped by your seal: Shamir shares held by humans, or a KMS key in the cloud. The snapshot carries that wrapped root key along with the data. Restore a snapshot from cluster A into cluster B and cluster B is now holding cluster A's locked drawer while carrying only its own key.

This is what the restore command's -force flag is for. Without it, Vault refuses when the seal does not match the snapshot, and the flag's own help text says it bypasses checks ensuring the auto-unseal or Shamir keys are consistent with the snapshot data. Read that twice. -force skips the check. It does not make the seal match. You will reach for it legitimately on every restore into a rebuilt cluster, which is exactly why it deserves care rather than avoidance: force a restore across genuinely mismatched seals and the command reports nothing, the data lands, and the cluster then fails to unseal while the process looks perfectly healthy from the outside.

# The isolated lab cluster is already initialized and unsealed, it is wired to the
# SAME AWS KMS key as production, and you are about to overwrite everything in it.
vault operator raft snapshot restore -force /var/backups/vault/raft-2026-07-27-0200.snap
vault status

Key                      Value
---                      -----
Seal Type                awskms
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.17.6
Storage Type             raft
Cluster Name             vault-cluster-prod
HA Enabled               true
HA Mode                  active

Cluster Name is the tell that it worked. This machine answered to vault-cluster-lab five minutes ago and now reports the production cluster's name, because that value came out of the snapshot along with everything else. Had the lab's seal pointed at a different KMS key, you would be reading the server log instead of a status table. Exact wording varies by seal type and version, but the shape is always the same.

[ERROR] core: failed to unseal: error="failed to decrypt encrypted stored keys:
        error decrypting seal wrapped value: InvalidCiphertextException: "
[INFO]  core: vault is sealed

Seal locality is a design decision you make long before you need it. If auto-unseal depends on a single-region KMS key in us-east-1 and the DR (disaster recovery) plan says "stand the cluster up in eu-west-1", the restore will succeed and the unseal will not, and you will be reading key policy documentation with an incident bridge listening. The fixes are unglamorous and all of them need doing in advance: a multi-region KMS key with a replica in the recovery region, a Transit seal served by a small separate Vault cluster both regions can reach, or a written seal migration you have already rehearsed.

Rehearse It Like A Fire Drill

A drill has a clock and produces a document. Time it from "we decide to restore" to "an application reads a real secret", because that number is your true RTO (recovery time objective, how long recovery is allowed to take), and it usually runs two or three times what the runbook claims. Write down every surprise while it is fresh: the KMS key nobody could reach, the login method nobody remembered existed, the bucket policy that denied the break-glass role, the twelve minutes spent finding who holds share three.

Verify with a checklist rather than a feeling. Prove that the seal type is what you expect, that mounts and auth methods came back, and that both a static secret and a freshly minted dynamic credential still work.

# your pre-restore lab token is gone; log in with an identity from the snapshot
vault login -method=userpass username=breakglass
vault auth list
vault kv get -mount=secret app/payments/db

Path         Type        Accessor                    Description                Version
----         ----        --------                    -----------                -------
approle/     approle     auth_approle_1a2b3c4d       n/a                        n/a
oidc/        oidc        auth_oidc_5e6f7a8b          n/a                        n/a
userpass/    userpass    auth_userpass_9c0d1e2f      break-glass only           n/a
token/       token       auth_token_3a4b5c6d         token based credentials    n/a
 
== Secret Path ==
secret/data/app/payments/db
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-26T22:14:03.918Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            7
 
====== Data ======
Key         Value
---         -----
password    r0tated-2026-07-26
username    payments_app

Upgrades Without Losing Quorum

Three voters tolerate exactly one failure. Take a node down for an upgrade and your slack is gone: the two survivors still form a majority and everything looks fine, but the next hiccup, a reboot, a full disk, a five-second network partition, stops writes across the whole platform. Upgrades are therefore a sequence with waiting deliberately built into it, not a parallel rollout across an inventory file.

Autopilot, the cluster supervisor built into integrated storage, tells you when it is safe to move. vault operator raft autopilot state reports a Failure Tolerance number, which is how many nodes could still die without taking you down. Upgrade the next node only once that number is back where it started.

vault operator raft autopilot state

Healthy:                      true
Failure Tolerance:            1
Leader:                       vault-1
Voters:
   vault-1
   vault-2
   vault-3
Servers:
   vault-1
      Name:            vault-1
      Address:         vault-1.internal:8201
      Status:          leader
      Node Status:     alive
      Healthy:         true
      Last Contact:    0s
      Last Term:       9
      Last Index:      184213
      Version:         1.17.6
      Node Type:       voter

Upgrade the standby nodes one at a time and leave the active node for last, because a failover onto a node still running the old version is the thing you are trying hardest to avoid. Pay attention to how you stop the process, too. systemctl stop sends SIGTERM (the polite "please exit" signal), and Vault uses it to shut down cleanly, handing off leadership and telling its peers on the way out. Kill it with SIGKILL (the un-catchable "stop right now" signal) and the survivors have to notice the silence and hold a fresh election before writes resume, which is seconds of downtime you were never obliged to take.

# 0. snapshot first: the data store carries no backward-compatibility promise
vault operator raft snapshot save /var/backups/vault/pre-1.18.5-$(date +%F).snap
 
# 1. one standby at a time (vault-3, then vault-2). Active node goes last.
sudo systemctl stop vault              # SIGTERM: clean shutdown, clean handover
sudo apt-get install -y vault=1.18.5-1
sudo systemctl start vault             # auto-unseal means no human keys needed
 
VAULT_ADDR=https://vault-3.internal:8200 vault status

Key                     Value
---                     -----
Seal Type               awskms
Initialized             true
Sealed                  false
Version                 1.18.5
Storage Type            raft
Cluster Name            vault-cluster-prod
HA Enabled              true
HA Mode                 standby
Active Node Address     https://vault-1.internal:8200

Sealed false and standby mode look like success, but the node still has to catch up on the Raft log before it counts as a healthy voter. Ask autopilot instead of trusting the status table, and see what the cluster actually thinks of you right now.

vault operator raft autopilot state

Healthy:                      false
Failure Tolerance:            0
Leader:                       vault-1
Voters:
   vault-1
   vault-2
   vault-3
Servers:
   vault-3
      Name:            vault-3
      Address:         vault-3.internal:8201
      Status:          voter
      Node Status:     alive
      Healthy:         false
      Last Contact:    12.884s
      Last Term:       9
      Last Index:      183980
      Version:         1.18.5
      Node Type:       voter

HashiCorp's upgrade guidance is blunt about the one move operators reach for by instinct: do not issue a step-down at any point during the upgrade. Let the active node hand over leadership by shutting down cleanly when its turn arrives, and treat the upgrade as finished only once an upgraded standby has taken over active duty. If you run Vault Enterprise 1.11 or later with autopilot's automated upgrades switched on, disable that feature before starting a manual rolling upgrade, or two systems will argue about which nodes should be voters and you can lose quorum watching them.

That pre-upgrade snapshot is your only real rollback. Vault makes no backward-compatibility guarantee for its data store, so an upgrade can change on-disk structures the older binary cannot read. Reinstalling the old package is not a rollback; restoring the pre-upgrade snapshot onto the old binary is. Large version jumps are supported, but read the upgrade notes for every version you cross rather than only the one you are landing on, because storage and autopilot behavior changes hide in the intermediate releases. Then rehearse the whole jump on a scratch cluster built from a production snapshot and run your restore-drill smoke tests, meaning an OIDC (OpenID Connect, the browser-based single sign-on protocol) login, a key/value read, a dynamic database credential, and a PKI (public key infrastructure, the machinery that issues certificates) certificate issue, before any production node is touched.

What A Snapshot Does Not Cover

Snapshots restore Vault's data and nothing else. They do not restore the machine, the TLS (transport layer security) certificate and private key files on disk, the systemd unit, the vault.hcl config, or the cloud identity that lets a node call KMS. Those belong in configuration management and version control, and your real recovery time is the sum of both halves: rebuilding nodes that can start and unseal, then installing the snapshot into them. Teams who time only the second half get surprised by a factor of four.

Snapshots are also not replication. Enterprise performance replication and disaster recovery replication keep a warm second cluster in step continuously, with lag measured in seconds, and they answer a different question than a nightly file does. Community Edition teams get snapshots plus a documented rebuild, which is perfectly defensible as long as leadership hears the honest RPO number rather than the one they hoped for. Vault writing its own snapshots on a schedule straight to Amazon S3 object storage, Azure Blob, or Google Cloud Storage through sys/storage/raft/snapshot-auto/config/<name> is an Enterprise feature; on Community Edition you own the cron job, the retention, and the alerting.

Try This

Prove to yourself that a restore really is a rewind, on a disposable lab cluster with no route to anything real. Write a canary after the snapshot, restore, and watch the canary cease to have ever existed.

vault operator raft snapshot save /tmp/before.snap
vault kv put -mount=secret canary v=1
vault kv get -field=v -mount=secret canary
 
# same cluster, same seal, so no -force is needed here
vault operator raft snapshot restore /tmp/before.snap
 
# your session began after the snapshot, so your token is not in it
vault login "$ROOT_TOKEN_FROM_BEFORE"
vault kv get -mount=secret canary

== Secret Path ==
secret/data/canary
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:41:22.117Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
1
 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. ...
 
No value found at secret/data/canary
# the cluster is back at the moment before the canary was written

Book the next drill before you close this page: a date, two named people, a scratch cluster, and a stopwatch. The only snapshot you can trust is one you have already restored, and the only upgrade path you can trust is one you have already walked on a throwaway cluster. From here, advanced secrets management picks up with namespaces, replication topologies, and deeper identity work, all of which assume this loop already closes cleanly.

Takeaway

The trap worth remembering here: a snapshot is the whole vault in one file. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related