Audit devices and on-call signals

Who read what, and what pages you.

A hospital pharmacy logs every opening of the controlled-substance cabinet: who opened it, which drawer, what time, and whether anything left the shelf. Nobody treats that log as an insult to the staff. It is the difference between thinking Dana pulled that vial and knowing. A Vault cluster with no audit device is that cabinet with no log, except the drawers hold your database passwords, your payment provider keys, and the credentials that build your cloud.

Audit devices are Vault's ledger. Every request that arrives and every response that leaves gets written as one line of JSON (JavaScript Object Notation, a plain-text format that people and machines can both read), with sensitive values replaced by a one-way fingerprint. Turn a device on before you onboard a second team, because a log only tells you about the days it was running. There is no way to reconstruct last month.

Here is the failure you are buying insurance against. Someone lifts an AppRole secret ID out of a build job's environment variables. AppRole is Vault's login method for machines: the role ID acts as the username, the secret ID acts as the password, and the pair buys a token. With that token they can read every path the role's policy allows, quietly, from anywhere that can reach port 8200. On a cluster with no audit device the read is invisible. Six weeks later a customer key turns up on a paste site and you face the two questions every incident asks. What did they touch, and what do we rotate? Without a log the only honest answer is everything, and rotating everything at company scale is a self-inflicted two-week outage.

The second half of this lesson is the part people meet the hard way. Once you enable a device, Vault will refuse to run a request it cannot record. That behavior is deliberate and correct for a secrets platform, and it is also a brand new way for Vault to be unavailable. Most of the work below is about making that trade a decision you took on a Tuesday afternoon instead of a discovery you made at 3 a.m.

What Vault Actually Writes Down

Start with the file device, which appends one JSON object per line to a path on disk. It is the most predictable of the three device types because its only failure mode is the filesystem, and filesystems are something you already know how to watch.

vault audit enable file \
  file_path=/var/log/vault/audit.log \
  mode=0640
 
vault audit list -detailed

Success! Enabled the file audit device at: file/
 
Path     Type    Description    Replication    Options
----     ----    -----------    -----------    -------
file/    file    n/a            replicated     file_path=/var/log/vault/audit.log mode=0640

Two details in that output matter later. The device landed at the path file/, which defaults to the device type, so a second file device needs an explicit -path= the same way secret engine mounts do. And mode=0640 widens the default of 0600. That default means only the file's owner can read it, and the owner is whatever user the Vault process runs as, usually vault, not root. If a log shipper runs under its own account, it needs group read, and 0640 plus a shared group gives it that without making your audit trail world readable.

Now generate an event and look at what landed.

vault kv get secret/lab/demo > /dev/null
 
sudo tail -n 4 /var/log/vault/audit.log \
  | jq -c '{type, path: .request.path, op: .request.operation, who: .auth.display_name, err: (.error // "")}'

{"type":"request","path":"sys/internal/ui/mounts/secret/lab/demo","op":"read","who":"approle","err":""}
{"type":"response","path":"sys/internal/ui/mounts/secret/lab/demo","op":"read","who":"approle","err":""}
{"type":"request","path":"secret/data/lab/demo","op":"read","who":"approle","err":""}
{"type":"response","path":"secret/data/lab/demo","op":"read","who":"approle","err":""}

One vault kv get produced four lines. The first pair is the KV v2 preflight, where KV v2 means version 2 of the key/value secrets engine, the one that keeps old versions of a secret. Before it can rewrite your path to secret/data/..., the CLI (command-line interface) asks Vault which version the secret/ mount runs. So expect your log volume to be roughly double the request count you had in mind.

The pairing itself is worth slowing down on, because the order of operations decides what you can search for. Vault checks your token and your policy first. Then it writes the request entry. Only once that entry is safely on at least one device does it hand the call to the secrets engine, and the response entry follows when the engine is done. A blocked call therefore never reaches an engine, so it never produces a response line. What you get instead is a lone request entry with permission denied sitting in the top-level error field. Filter with select((.error // "") != "") and you have every attempt Vault refused, which is exactly the view you want when somebody is probing the edges of their access.

Look at the identity block on a successful line and you will see why the log is safe to ship somewhere else.

sudo tail -n 1 /var/log/vault/audit.log | jq '.auth | {display_name, policies, accessor, client_token}'

{
  "display_name": "approle",
  "policies": [
    "app-read",
    "default"
  ],
  "accessor": "hmac-sha256:5a2f9c81b0e4d773a6c2f118d94e0b3c7ad61f28e0c9b4a7513d6f8029ce41bd",
  "client_token": "hmac-sha256:9e14a6d02b7fc35819ee0d4a6c73b1f5082ad9e6741c3b0f57ab2e9d63108c74"
}

A rubber stamp turns a name into a fixed scramble: the same name always stamps the same way, and nobody can read the name back out of the ink. That hmac-sha256: prefix marks exactly that kind of stamp, an HMAC (hash-based message authentication code), and the ink is a random salt Vault generates for each audit device and keeps inside its encrypted storage. Two things follow. Nobody can reverse a log line back into a token or a secret value, so the log can live in a search system your whole security team touches. And the same token stamps differently on every device, because every device has its own salt, which trips up investigators who search device A's log with a fingerprint computed from device B.

Why A Full Disk Takes Vault Down

A bank teller cannot hand you cash when the ledger system is down, and that is a feature rather than a bug. Vault treats its audit devices the same way. The rule is that at least one enabled device must persist the entry. Two devices with one broken is a degraded system that keeps serving. Every device broken is a Vault that answers vault status cheerfully while failing every read, because the request entry has to land before the call is routed anywhere.

The same rule applies on the way out, with a nastier edge. If no device can record the response, the client gets an error even though the write already committed to storage. Your caller sees failure, your data says success, and the two only agree again after somebody looks. Healthy audit storage is how you avoid that argument.

The asymmetry catches people. Zero audit devices means no requirement at all, so an unaudited cluster is perfectly available and perfectly blind. Enabling your first device is the moment you trade blindness for a dependency. Prove to yourself what that looks like in a lab before production shows you.

# a 20 MiB scratch filesystem, so it fills in seconds
sudo truncate -s 20M /var/tmp/auditfs.img
sudo mkfs.ext4 -q /var/tmp/auditfs.img
sudo mkdir -p /var/log/vault-small
sudo mount -o loop /var/tmp/auditfs.img /var/log/vault-small
 
# make smallfs/ the only enabled device on this lab node
vault audit enable -path=smallfs file file_path=/var/log/vault-small/audit.log
vault audit disable file/
 
sudo dd if=/dev/zero of=/var/log/vault-small/ballast bs=1M count=32 2>/dev/null
 
vault kv get secret/lab/demo

Success! Enabled the file audit device at: smallfs/
Success! Disabled audit device (if it was enabled) at: file/
 
Error making API request.
 
URL: GET https://vault-1.internal:8200/v1/sys/internal/ui/mounts/secret/lab/demo
Code: 500. Errors:
 
* internal error

The client sees a generic 500, on purpose, because telling an anonymous caller that your audit pipeline is broken is an invitation. The real story is on the server.

sudo journalctl -u vault -n 2 --no-pager -o cat

[ERROR] audit: backend failed to log request: backend=smallfs/ error="write /var/log/vault-small/audit.log: no space left on device"
[ERROR] core: failed to audit request: path=sys/internal/ui/mounts/secret/lab/demo error="no audit backend succeeded in logging the request"

A full 20 MiB partition took your entire secrets platform offline, and the read never touched storage at all. That is the correct behavior, and it is also why audit disk is now production-critical storage. Size it for peak request rate times retention, remembering the KV v2 preflight doubles your line count, alert on free space, and never share the volume with anything chatty.

Getting The Log Off The Box

A security camera whose only recording sits in the lobby it films is decoration. An audit log that never leaves the Vault node is the same thing: the first move of anyone with root on that host is to edit or delete it, and you will never know they were there. Every part of the pipeline below exists to make the log outlive the machine that produced it.

Rotation comes first, because the file device rotates nothing on its own. It does close and reopen its file when the Vault process receives SIGHUP (the hangup signal, a long-standing Unix convention meaning reload your files). Rename the file without sending that signal and Vault keeps writing to the same inode, which is the file's real identity on disk. The name moved; the open handle did not. Your shiny new audit.log stays at zero bytes while the disk fills with a file ls no longer shows.

/var/log/vault/audit.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
  create 0640 vault vault
  sharedscripts
  postrotate
    /bin/systemctl kill -s SIGHUP vault.service
  endscript
}

create 0640 vault vault makes logrotate build the replacement file with the owner and permissions your shipper needs. Once logrotate owns file creation like that, set mode=0000 on the audit device, which is the documented way to tell Vault to leave file permissions alone instead of reapplying its own every time it reopens.

With rotation handled, point a shipper at the file (Vector, Fluent Bit, Filebeat, whichever your platform already runs) and send it to object storage in a different cloud account with object lock turned on. Object lock is the write-once-read-many setting: once an object is written, nothing modifies or deletes it until its retention clock expires, including the account root. Pick a window longer than your worst realistic detection lag. Ninety days is a floor, a year is what auditors tend to ask for, and a year is what you will wish you had when a breach turns out to be eight months old.

Vault also offers socket and syslog devices that push entries over the network. They look tidier than tailing a file and they carry a sharper failure mode, because a stalled collector can stall Vault itself, and the socket device gives up on a write after write_timeout, which defaults to two seconds per entry. The file device fails in a way you can measure with df, the command that reports free disk space. That predictability is why file plus a shipper is the common production shape, with a socket device added as the redundant second only when you have a collector you trust.

One more fix before the log is useful. Behind a load balancer, every entry records the balancer's address in request.remote_address, so all your traffic appears to come from one IP (internet protocol) address. Set x_forwarded_for_authorized_addrs on the listener so Vault trusts the forwarding header from your balancer subnet, then tell the audit subsystem to record that header without fingerprinting it. The sys/config/auditing/ paths are root-protected, so the token doing this needs the sudo capability on them, not only update.

vault write sys/config/auditing/request-headers/x-forwarded-for hmac=false
vault read  sys/config/auditing/request-headers/x-forwarded-for

Success! Data written to: sys/config/auditing/request-headers/x-forwarded-for
 
Key                Value
---                -----
x-forwarded-for    map[hmac:false]

From then on the header shows up under request.headers on every entry, and a client address stops being a guess.

Reading The Log Like An Investigator

The moment you need this log, the question is never abstract. A token leaked, which one is it in these forty million lines, and what else did it read? Fingerprinting protects the log at rest, and it also means you cannot grep for the token you are holding. Vault gives you the missing half: hand it the value and it returns the fingerprint for a named device. Your policy needs update on sys/audit-hash/<device>, so put it in your incident-response role rather than the everyday operator role, and test it on a quiet day.

vault write sys/audit-hash/file input="hvs.CAESIHqk8Zt3rM2yV0pQ..."

Key     Value
---     -----
hash    hmac-sha256:8f2c1b6e5d0a9c37b41e6f2a8d55c9013ab7e4f6c28d90ab13e5f7c4d6082a19

Your hash will differ from anyone else's for the same input, because the salt belongs to the device. Compute it against file/ to search file/'s log, compute it again against a second device to search that one. Then filter on request.client_token rather than auth.client_token. Both carry the fingerprint on a normal call, but when Vault cannot resolve a token at all it writes the entry with no auth block, so a search on the auth field silently misses the attempts made with an expired or forged credential.

sudo jq -r 'select(.request.client_token == "hmac-sha256:8f2c1b6e5d0a9c37b41e6f2a8d55c9013ab7e4f6c28d90ab13e5f7c4d6082a19")
            | select(.type == "request")
            | [.time, .request.operation, .request.path, (.error // "-")] | @tsv' \
  /var/log/vault/audit.log | head -6

2026-07-26T22:41:03.118Z  read  secret/data/prod/db        -
2026-07-26T22:41:03.402Z  read  secret/data/prod/stripe    -
2026-07-26T22:41:11.907Z  list  secret/metadata/prod       -
2026-07-26T22:43:55.244Z  read  sys/policies/acl/app-read  permission denied
2026-07-26T22:44:02.661Z  list  sys/auth                   permission denied
2026-07-26T22:44:09.330Z  list  sys/mounts                 permission denied

Selecting only request entries gives you one line per call, allowed or blocked, which is the timeline you want. The first two lines look like an application doing its job. The rest do not. Applications know the paths they need and read them on a schedule. They do not list a directory, try to read their own policy document, then enumerate authentication methods and mounts inside a minute. That shape is a human with a stolen credential drawing a map, and the denials are the loudest part of it. A list of sys/auth or sys/mounts from a token whose display name is a machine role should open a ticket every single time.

Two device options tune this work. hmac_accessor=false writes token accessors in the clear, so a responder can go from a log line straight to vault token revoke -accessor <accessor> without a lookup dance. An accessor cannot be used as a token, so the exposure is modest, but anyone holding the log can then revoke tokens and read their metadata. That is a real decision rather than a default to flip on autopilot. The other is elide_list_responses=true, which replaces the body of a LIST response with a count. Without it, one vault list against an identity store holding 40,000 entities writes 40,000 names into your log on every call, and a single misbehaving script fills the disk you now know how to fear. Elision keeps the fact that someone listed the path and drops the payload.

Build saved searches for the paths that only ever appear on a bad day: writes under sys/policies/acl/, anything under sys/rekey/ or sys/generate-root/, sys/step-down, any read of sys/raw/, and any request whose auth.policies array contains root. Add one more that is easy to miss. Enabling or disabling an audit device runs against sys/audit/, which is root-protected, so only a sudo-capable token can do it. When an attacker deletes your last device, Vault writes the request entry first, carries out the delete, and then has nowhere to put the response. The signature is a successful delete against sys/audit/<name> with no response line, followed by a log file that stops growing. Alert on both halves, because a rule like no audit line seen in ten minutes also catches the case where the delete entry never reached your collector either.

Signals That Should Wake A Human

The audit log tells you what people did. Telemetry tells you whether Vault is in a state where anything works at all. Turn on the Prometheus endpoint, then scrape each node directly instead of going through the load balancer, because whether this node is sealed is a per-node question and a balancer will happily hide the sealed one behind two healthy ones.

telemetry {
  prometheus_retention_time = "24h"
  disable_hostname          = true
}
 
listener "tcp" {
  address = "0.0.0.0:8200"
  # ...tls settings...
 
  telemetry {
    # only with the metrics port firewalled to your scrapers
    unauthenticated_metrics_access = true
  }
}

curl -s --cacert /etc/vault.d/tls/ca.crt \
  'https://vault-1.internal:8200/v1/sys/metrics?format=prometheus' \
  | grep -E '^vault_(core_unsealed|autopilot_failure_tolerance|audit_log_request_failure)'

vault_core_unsealed{cluster="vault-prod"} 1
vault_autopilot_failure_tolerance{cluster="vault-prod"} 1
vault_audit_log_request_failure{cluster="vault-prod"} 0

Four signals belong on the phone at any hour. vault_core_unsealed dropping to 0 means that node serves nothing, and if auto-unseal did not bring it back, something is wrong with your key management service or the network path to it. Treat a failed scrape as the same page, since a node that cannot answer at all is not a healthy node. vault_audit_log_request_failure moving off zero means you are either failing requests already or one disk away from it. vault_autopilot_failure_tolerance, reported by the active node, tells you how many more nodes the Raft cluster can lose and still hold a majority, so a 0 there turns your next routine reboot into an outage. Free space under ten percent on an audit volume is that same alarm arriving early enough to fix calmly.

Two cautions on the metrics themselves. Counters like the audit failure one may not appear in a scrape at all until the first time they fire on some versions, so an alert shaped like vault_audit_log_request_failure > 0 can look green because the series does not exist yet. Write it as increase(vault_audit_log_request_failure[10m]) > 0 instead. And metric names have drifted between Vault releases, so confirm yours against /v1/sys/metrics on the version you actually run rather than importing a dashboard from a blog post.

A second tier deserves a ticket the same day rather than a page. A sustained rise in denied requests on a path that never used to deny is either a policy change that quietly broke an application or somebody testing the fence. Token creation running several times above baseline is usually a client that stopped caching its token, and occasionally credential harvesting. Lease count climbing without matching workload growth means something is minting and never revoking, which turns your next restart into a very slow one.

None of those metrics can tell you your shipper died, your collector filled up, or your index quietly stopped accepting writes. The check that covers the whole chain is a canary: a cron job that reads one dedicated path every minute, plus an alert that fires when that path has not appeared in your search system within five minutes. It exercises Vault, the audit device, the file, the rotation, the shipper, the network, and the index in one shot. It is the only test in this lesson that fails when your monitoring is the broken thing.

Try This

Ten minutes in a lab gets you both halves: a denial recorded, and the deletion of the recorder recording itself. Use vault read on the raw path rather than vault kv get, so the bad token fails on the path you are watching instead of on the KV preflight.

vault audit enable -path=lab file file_path=/tmp/vault-lab-audit.log mode=0640
 
VAULT_TOKEN=hvs.notarealtoken vault read secret/data/lab/demo || true
 
jq -c 'select(.request.path=="secret/data/lab/demo") | {type, err: (.error // ""), auth: (.auth // "none")}' \
  /tmp/vault-lab-audit.log

Success! Enabled the file audit device at: lab/
 
Error reading secret/data/lab/demo: Error making API request.
 
URL: GET https://vault-1.internal:8200/v1/secret/data/lab/demo
Code: 403. Errors:
 
* permission denied
 
{"type":"request","err":"permission denied","auth":"none"}

One line, not two, and no identity attached to it, because Vault never resolved that token. The hashed value still sits under request.client_token, which is why your investigator queries filter on that field. Now delete the device and watch it sign its own exit.

vault audit disable lab/
tail -n 1 /tmp/vault-lab-audit.log | jq -c '{type, op: .request.operation, path: .request.path, err: (.error // "")}'

Success! Disabled audit device (if it was enabled) at: lab/
 
{"type":"request","op":"delete","path":"sys/audit/lab","err":""}

The last line that file will ever hold is the order to stop writing, with no error on it and no response behind it. Go build the alert for that shape before you need it. The next lesson covers rotation and break-glass, where these same signals decide when somebody reaches for the sealed envelope.

Takeaway

The trap worth remembering here: two devices, two volumes, and an alert that depends on neither. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related