Audit, tamper-evidence, break-glass & DR

Audit devices, HMAC log integrity, recovery keys, and the DR drill.

The last mile of a secrets platform is proving what happened and surviving disaster. Auditors ask "who could read this, who did, and can you prove the record was not altered?" Incident responders ask "can we get back to a known-good state if the whole cluster is lost?" Both come down to trustworthy audit logs, a tested break-glass path, and a rehearsed disaster-recovery drill. A secrets manager you cannot audit or recover is a liability wearing the costume of a control.

Audit devices and tamper-evidence

Vault’s audit devices log every request and response, and they do it in a way built for trust: sensitive values are HMAC-ed (hashed with a per-Vault key) so the log records that a secret was accessed and by whom without printing the secret itself — the log is safe to ship to a SIEM. Audit logging is also fail-closed by design: if Vault cannot write to at least one audit device, it stops serving requests rather than operate blind. That is the correct default for a system of record, even though it means you must run redundant audit sinks so a single log outage does not take Vault down.

$ vault audit enable file file_path=/var/log/vault/audit.log
$ vault audit enable syslog tag="vault" facility="AUTH"
 
# a log line records the access without the secret (HMAC of the value):
{ "type":"response","auth":{"display_name":"oidc-alice","policies":["payments-read"]},
  "request":{"operation":"read","path":"database/creds/payments-ro"},
  "response":{"data":{"password":"hmac-sha256:5f2e..."}} }
# ship both sinks to a SIEM; two devices so one outage doesn’t fail-close Vault.

Guarantee integrity: the log must resist your own admins

An audit log an insider can quietly edit proves nothing. Make the record tamper-evident: forward audit events off-box in real time to append-only, access-controlled storage (a SIEM, an immutable bucket with object lock) so the copy of record is not on a host any Vault admin can reach; and monitor for the tells of tampering — gaps in the sequence, an audit device being disabled, log files truncated. The threat model here explicitly includes privileged insiders, so the integrity control has to live outside the trust boundary of the people it is auditing.

Break-glass and the disaster-recovery drill

Two emergencies need a rehearsed answer. Break-glass is "we need root right now": generate a temporary root token from a quorum of recovery-key holders (vault operator generate-root), use it under four-eyes observation, and revoke it the moment the emergency passes — root is a fire axe behind glass, not a tool on the desk. Disaster recovery is "the primary is gone": promote the DR-replica cluster, repoint clients, and verify secrets flow. Neither works if its first real execution is during the actual incident, so both must be drilled on a schedule until they are muscle memory.