Vault Agent and secret injection

Templates, renewals, and keeping apps dumb.

Your payments service reads /app/config/app.env when it boots, and reads it again whenever something sends it a SIGHUP (signal hang-up, the old Unix way of telling a running program to reload its configuration without restarting it). It has worked that way for four years. Nobody has added a Vault client library to it, and nobody is going to. Yet the database password inside that file is different today than it was yesterday, and the service never noticed the swap.

Banks do not send every cashier down to the vault. A courier with a badge goes, draws what the tills need, and refills each drawer before it runs dry. The cashier opens the same drawer all day and never learns the combination. Vault Agent is that courier: a small process running beside your application that proves its identity using something the platform already handed it, fetches what the application needs, writes it where the application already looks, and goes back for fresh material before the old material expires.

The alternative is a Vault SDK (software development kit, the vendor's client library) compiled into every service. Twelve teams then own twelve copies of the same retry logic, twelve opinions about what to do when a renewal fails at 4 a.m., and twelve places where somebody plants the very first credential that lets the app reach Vault at all. That credential has a name, secret zero, and every design trips over it. Agent moves the whole problem into one binary the platform team owns.

The Three Jobs Agent Does

Agent does three separable things. Naming them separately matters, because they fail separately and you debug them in a fixed order.

First, auto-auth. Agent logs in to Vault on the workload's behalf using an identity the platform already issued: the service account token Kubernetes mounts into every pod (a JWT, or JSON Web Token, a short signed statement of who this pod is), an AppRole pair on a virtual machine (a role ID that names the app plus a secret ID that proves it), or an instance identity document on a cloud host. You never hand Agent a Vault token. It earns one, and earns a new one when the old one runs out of road.

Second, sinks. A sink is where Agent drops the token it earned so something else can pick it up. The file sink writes the raw token to a path, mode 0640 by default, meaning the owner can read and write it, the group can read it, and everyone else is locked out. Be honest about that file: it is a live Vault token carrying your application's full policy, and anyone who reads it can do everything the app can, from anywhere, until it expires. Set wrap_ttl on the sink and what lands on disk is a response-wrapping token instead, a sealed envelope that opens exactly once. Your app unwraps it to get the real token. If a thief unwraps it first, your app's unwrap fails loudly, so you learn you were robbed rather than never finding out.

Third, templates. Agent renders files using Consul Template syntax, the same Go-based templating language HashiCorp's standalone consul-template binary uses, then optionally runs a command so the application picks up the change. This is the part your app actually touches, and where nearly every production bug lives. Agent can also sit in front of Vault as a caching proxy, so local processes talk to http://127.0.0.1:8200 without holding a token of their own, though Vault 1.14 split that job into a command of its own, vault proxy, which is the same binary you already have rather than a separate download. New work belongs there.

# HCL: HashiCorp Configuration Language, the format Vault reads
pid_file = "/run/vault-agent/pid"
 
vault {
  address = "https://vault.internal:8200"
  ca_cert = "/etc/vault.d/tls/internal-ca.crt"
  retry {
    num_retries = 5
  }
}
 
auto_auth {
  method "approle" {
    mount_path = "auth/approle"
    config = {
      role_id_file_path   = "/etc/vault.d/role_id"
      secret_id_file_path = "/etc/vault.d/secret_id"
      # default true: the file is read once, then deleted
      remove_secret_id_file_after_reading = true
    }
  }
 
  sink "file" {
    config = {
      path = "/run/vault-agent/token"
      mode = 0640
    }
  }
}
 
template_config {
  # fail closed: exit rather than run blind once retries are exhausted
  exit_on_retry_failure         = true
  # how often non-leased secrets (KV v2) are re-read; default is 5m
  static_secret_render_interval = "1m"
}
 
template {
  source               = "/etc/vault.d/templates/app.env.ctmpl"
  destination          = "/app/config/app.env"
  perms                = "0640"
  error_on_missing_key = true
  sandbox_path         = "/app/config"
 
  exec {
    command = ["/usr/bin/systemctl", "reload", "myapp"]
    timeout = "30s"
  }
}

The lines in that file that look like housekeeping are the ones doing the security work. remove_secret_id_file_after_reading defaults to true, so Agent reads the AppRole secret ID off disk and deletes the file immediately; a disk image taken an hour later has nothing to steal. sandbox_path refuses to render anywhere outside /app/config, so a typo in a destination cannot overwrite something in /etc. And exit_on_retry_failure is the fail-closed switch: if Vault is unreachable or the read is denied, Agent exits rather than carrying on with stale data, and your systemd restart policy turns that into a loud crash loop instead of quiet drift.

One naming trap. The exec block nested inside template replaced the older command = "systemctl reload myapp" spelling, which still works and still fills most blog posts. A separate top-level exec block also exists and does something entirely different; you meet it near the end of this lesson. Same word, two jobs, different nesting.

sudo -u vault vault agent -config=/etc/vault.d/agent.hcl

==> Vault Agent started! Log data will stream in below:
 
==> Vault Agent configuration:
 
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.17.6
 
2026-07-27T09:14:02.118Z [INFO]  agent.sink.file: creating file sink
2026-07-27T09:14:02.118Z [INFO]  agent.sink.file: file sink configured: path=/run/vault-agent/token mode=-rw-r-----
2026-07-27T09:14:02.119Z [INFO]  agent.template.server: starting template server
2026-07-27T09:14:02.119Z [INFO]  agent.auth.handler: starting auth handler
2026-07-27T09:14:02.119Z [INFO]  agent.auth.handler: authenticating
2026-07-27T09:14:02.181Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2026-07-27T09:14:02.181Z [INFO]  agent.auth.handler: starting renewal process
2026-07-27T09:14:02.182Z [INFO]  agent.sink.file: token written: path=/run/vault-agent/token
2026-07-27T09:14:02.244Z [INFO]  (runner) creating watcher
2026-07-27T09:14:02.301Z [INFO]  (runner) rendered "/etc/vault.d/templates/app.env.ctmpl" => "/app/config/app.env"

Read that log top to bottom, because its order is your debugging order. If you never reach authentication successful, stop reading your templates; the fault is the role, the mount path, or plain network reachability. If you reach it and never see rendered, the fault is the template or the policy sitting behind the path it asks for.

The Template Is Where The Mistakes Live

vault kv get secret/payments/db on the command line and secret "secret/data/payments/db" inside a template point at the same secret. The kv command quietly hides a data/ segment that KV v2 (key-value secrets engine, version 2, the one that keeps a version history) inserts into the real API (application programming interface) path. Templates speak that raw path, so you type data/ yourself, and you reach through a second .Data.data in the response body. Get either half wrong and Vault does not shout. You get an empty string where a password belongs.

{{- with secret "secret/data/payments/db" -}}
DB_USER={{ .Data.data.username }}
DB_PASSWORD={{ .Data.data.password }}
{{ end -}}
{{- with secret "database/creds/payments-rw" -}}
PG_USER={{ .Data.username }}
PG_PASSWORD={{ .Data.password }}
{{ end -}}

The two halves of that file are shaped differently on purpose. KV v2 nests your keys one level deeper, so it reads .Data.data.password. A dynamic database credential comes back flat, so it reads .Data.password. Copying the first shape onto the second is the most common template bug in the wild, and by default it fails without a sound.

That silence is a setting, not a law of nature. error_on_missing_key defaults to false, so a key the template cannot find renders as the literal text <no value>. Your application then starts up with the password <no value>, fails at the first query, or worse, connects to something that treats a blank credential as anonymous access. Turn it on. A template that refuses to render is a page you can act on in two minutes. A config file full of <no value> is an incident three hours later that four people misread as a database problem.

Agent writes to a temporary file in the destination's own directory and renames it into place, so nothing ever reads a half-written secret. That rename is also why single-file bind mounts break. Mount /app/config/app.env into a container as a file and Docker binds the inode, the filesystem's internal handle for that exact file. The rename swaps in a brand new inode, the host file updates, and the container happily reads the original bytes forever. Mount the directory, not the file. Kubernetes subPath volume mounts carry the same disease and stop updating after the first write.

ls -l /app/config/app.env
grep -c PASSWORD /app/config/app.env
grep -c '<no value>' /app/config/app.env

-rw-r----- 1 vault app 127 Jul 27 09:14 /app/config/app.env
2
0

Ownership is the half of file permissions people forget. Mode 0640 only helps if the group is right, and the group comes from the process that created the file, so set Group=app in the systemd unit and let Agent create files the app's user can read and nobody else can. Verify with ls -l. A file that is mode 0640 but owned root:root is a file your application cannot open, and you tend to discover that during a restart at the worst hour of the night. A world-readable secret file recreates the exact problem Vault was bought to solve.

Renewal Is Not Rotation

Agent's token has a lifetime, called a TTL (time to live), and vault token lookup against the sink file is the fastest way to see exactly what that token can do and how long it has left. Run it first whenever an application that worked yesterday starts collecting 403s (HTTP status 403, Forbidden, which Vault returns as permission denied).

VAULT_TOKEN=$(sudo cat /run/vault-agent/token) vault token lookup

Key                  Value
---                  -----
accessor             8QLmR2vTn9pXkC4bZsHw3aYd
creation_time        1753606442
creation_ttl         1h
display_name         approle
entity_id            4f1c9e2a-7b3d-4a55-9c0e-2d8f6b1a0c73
expire_time          2026-07-27T10:14:02.118Z
explicit_max_ttl     0s
id                   hvs.CAESIJ9r2xQmT4...
issue_time           2026-07-27T09:14:02.118Z
meta                 map[role_name:payments-agent]
num_uses             0
orphan               true
path                 auth/approle/login
policies             [default payments-read]
renewable            true
ttl                  59m41s
type                 service

Most of that output is bookkeeping. policies [default payments-read] is the real answer to "what can this application do", regardless of what the role was meant to grant. renewable true with ttl 59m41s means Agent will extend the token well before the hour is out, and it staggers those renewals slightly so a fleet does not all knock on the same second. num_uses 0 means unlimited uses, which is right for a long-running service and wrong for a one-shot CI job. And explicit_max_ttl 0s reads like "forever" but is nothing of the sort: the auth mount's own token_max_ttl and Vault's system maximum lease, 32 days out of the box, still cap it. Once a token hits that ceiling, renewal stops extending anything, so Agent throws the token away and authenticates from scratch. That is why the underlying identity has to stay valid for the whole life of the process, not only for its first minute.

Renewal covers leases. A lease is Vault's timed claim on a credential it created: it records what was issued, when that credential dies, and how Vault can pull it back early. Renewal does not cover static secrets, and that distinction confuses almost everyone exactly once. A dynamic database credential arrives with a lease, and Agent renews it until the lease hits its own maximum, then asks for a brand new credential and re-renders the file. A KV v2 secret has no lease, so there is nothing to renew; Agent re-reads it on a timer instead, static_secret_render_interval, five minutes by default. Change a KV value, watch the file for thirty seconds, and you will conclude Agent is broken. It is doing precisely what you configured.

Some leases cannot be renewed at all, either because the secrets engine forbids it or because the lease already reached its maximum. For those, Agent waits until lease_renewal_threshold of the lease has elapsed, 0.9 by default, then fetches a fresh secret and re-renders. Lower that number when your reload path is slow. At 0.9 of a five minute lease you have thirty seconds to write the file, signal the application, and let it finish opening new connections before the old credential dies underneath it.

Renewal traffic adds up faster than teams expect. A thousand Agents on one-hour tokens produce a thousand renewals an hour, and if they all started during the same rollout they bunch into the same minute forever afterwards. Agent's built-in stagger softens that without curing it. Vary token TTLs across roles instead of giving every workload the same round number, and go looking at Vault's request-rate metrics after a large deploy rather than after an outage.

The Application Still Has One Job

Every mechanism above delivers a fresh file. Whether the running process notices is your application's problem. A service that reads its configuration once at startup and caches it in memory will hold a revoked password for as long as it runs, while the file on disk looks perfect to anyone who checks. You have three honest options: re-read on SIGHUP and let the template's exec block send the signal, restart the unit on change and accept the blip, or open a fresh connection per request so the credential is read at the moment of use. Pick one and test it, because "the file changed" and "the application changed behaviour" are two different claims, and only the second one protects you.

Test it from the far side. Rotate the credential, wait for the render, then ask the database which usernames are actually holding connections right now.

# run this on the database host, not on the app host
psql -h db.internal -U postgres -Atc \
  "select usename, count(*) from pg_stat_activity where usename like 'v-%' group by 1;"

v-approle-payments-rw-Qk8T2mZ9-1753606442|12
v-approle-payments-rw-Lp4X9nB2-1753610042|0

Twelve live connections under the credential Vault issued an hour ago, zero under the one Agent rendered ten minutes ago. The file rotated. The application did not. That is a reload bug in your service rather than a Vault problem, and no amount of Agent configuration papers over it. Skip this check and rotation becomes a story you tell auditors instead of a control you operate.

Kubernetes: Pick One Delivery Belt

Kubernetes hands you three ways to move the same secret, and platform teams get into trouble by running all three with slightly different policies behind each. The Agent Injector is a mutating admission webhook, a piece of cluster software that edits pod definitions on their way in: annotate a pod and the cluster rewrites it before scheduling, adding an init container that renders the secret once plus a sidecar that keeps it fresh. The Secrets Store CSI (Container Storage Interface) driver mounts secrets as a volume, through the same plumbing that mounts disks. The Vault Secrets Operator, and the wider External Secrets Operator, read from Vault and write ordinary Kubernetes Secret objects that pods consume the normal way.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payments-api"
        vault.hashicorp.com/auth-path: "auth/kubernetes"
        vault.hashicorp.com/agent-inject-secret-app.env: "secret/data/payments/db"
        vault.hashicorp.com/agent-inject-perms-app.env: "0400"
        # 0400 means owner-only, so the agent must run as the app's UID
        vault.hashicorp.com/agent-run-as-user: "1000"
        # always supply your own template: the default dump format
        # is not the shape your application parses
        vault.hashicorp.com/agent-inject-template-app.env: |
          {{- with secret "secret/data/payments/db" -}}
          DB_USER={{ .Data.data.username }}
          DB_PASSWORD={{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: payments-api
      securityContext:
        runAsUser: 1000
      containers:
        - name: api
          image: registry.internal/payments-api:1.9.3

POD=payments-api-6c9f7d5b84-lk2vq
kubectl get pod $POD -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}{"\n"}'
kubectl exec $POD -c api -- cat /vault/secrets/app.env
kubectl exec $POD -c vault-agent -- sh -c 'mount | grep /vault/secrets'

vault-agent-init
api vault-agent
DB_USER=payments
DB_PASSWORD=s3cr3t-from-kv
tmpfs on /vault/secrets type tmpfs (rw,relatime,size=8102340k,inode64)

That output proves the whole chain in one screen. The webhook added an init container plus a sidecar, so the secret exists before your container runs its first line of code and stays fresh afterwards. The file sits at /vault/secrets/app.env in the shape your application expects. And the volume behind it is an emptyDir with medium: Memory, which the kernel presents as tmpfs, so a node disk snapshot carries nothing away. Watch the UID (user ID, the number Linux uses to decide who owns a file). Agent defaults to UID 100, and a 0400 file it creates is unreadable by an app container running as anything else, which is why agent-run-as-user matches 1000 above. Setting agent-pre-populate-only: "true" drops the sidecar and keeps only the init container: cheaper, and the secret then never refreshes for the life of the pod.

Pick one belt per platform and make it boring. Three partial integrations buy you three policy models, three sets of failure modes at 3 a.m., and one team who quietly parked a static password in a ConfigMap because none of the three fit their runtime.

When You Do Not Want A File At All

Some applications read only environment variables, and you cannot change them. Agent has a mode built for that, called process supervisor mode: Agent starts your application as a child process and hands it the secrets as environment variables at launch. No file is written anywhere. When a secret changes, Agent stops the child and starts it again with the new values.

vault {
  address = "https://vault.internal:8200"
  ca_cert = "/etc/vault.d/tls/internal-ca.crt"
}
 
auto_auth {
  method "approle" {
    mount_path = "auth/approle"
    config = {
      role_id_file_path   = "/etc/vault.d/role_id"
      secret_id_file_path = "/etc/vault.d/secret_id"
    }
  }
  # no file sink: nothing else on this host needs the token
}
 
# top-level exec: Agent supervises the application itself
exec {
  command                   = ["/usr/local/bin/payments-api", "--listen=:8080"]
  restart_on_secret_changes = "always"   # this is the default
  restart_stop_signal       = "SIGTERM"  # the polite "please shut down" signal
}
 
env_template "DB_PASSWORD" {
  contents             = "{{ with secret \"secret/data/payments/db\" }}{{ .Data.data.password }}{{ end }}"
  error_on_missing_key = true
}

vault agent -config=/etc/vault.d/agent-exec.hcl

2026-07-27T09:22:10.443Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2026-07-27T09:22:10.501Z [INFO]  (runner) creating watcher
2026-07-27T09:22:10.612Z [INFO]  agent.exec.server: started process: pid=41822
2026-07-27T09:27:10.744Z [INFO]  agent.exec.server: detected new secrets, restarting process
2026-07-27T09:27:10.745Z [INFO]  agent.exec.server: stopping process: pid=41822 signal=SIGTERM
2026-07-27T09:27:11.902Z [INFO]  agent.exec.server: started process: pid=41977

That is the top-level exec block, the one whose name collides with the exec nested inside template. The trade-off runs in both directions. Environment variables never touch a disk, which is the whole point, but on Linux they sit in /proc/<process id>/environ readable by the same user and by root, they are inherited by every child process your app spawns, and crash reporters love attaching the entire environment to a stack trace. restart_on_secret_changes = "always" also means a rotated password restarts your process, so it had better start fast and drain its connections cleanly.

Prove The Control Works

A control you have never watched fail is a control you are guessing about. Break this one deliberately in staging and look hard at the shape of the alert, so the shape is familiar when it arrives uninvited at midnight.

# take the path away from the agent's policy
vault policy write payments-read - <<'EOF'
path "secret/data/payments/other" {
  capabilities = ["read"]
}
EOF
 
# wait for the next render, then look at the unit
journalctl -u vault-agent -n 8 --no-pager

Success! Uploaded policy: payments-read
 
Jul 27 09:31:04 app-01 vault[41802]: [ERROR] agent.template.server: template server error: error="template: :1:12:
Jul 27 09:31:04 app-01 vault[41802]:   executing "" at <secret "secret/data/payments/db">: error calling secret:
Jul 27 09:31:04 app-01 vault[41802]:   Error making API request.
Jul 27 09:31:04 app-01 vault[41802]:   URL: GET https://vault.internal:8200/v1/secret/data/payments/db
Jul 27 09:31:04 app-01 vault[41802]:   Code: 403. Errors:
Jul 27 09:31:04 app-01 vault[41802]:   * permission denied"
Jul 27 09:31:09 app-01 systemd[1]: vault-agent.service: Main process exited, code=exited, status=1/FAILURE
Jul 27 09:31:09 app-01 systemd[1]: vault-agent.service: Failed with result 'exit-code'.

That failure is more helpful than it looks. The 403 prints the exact URL it was denied, so you can tell a wrong path from a wrong policy at a glance. The break took effect on a token that was issued before you touched anything, because a Vault token stores policy names rather than a frozen copy of the rules, and every request looks the rules up fresh. And the unit exited, because exit_on_retry_failure was set. That choice carries a price: fail closed and a Vault outage or a policy typo takes your application down with it, fail open and the app keeps serving from the last file it rendered, which is kinder to uptime and unkind to everything rotation was meant to protect. Decide per workload rather than globally, and write the decision down next to the alert, because the engineer paged at 3 a.m. will otherwise assume you chose the other one.

Try this

Fifteen minutes against a dev server is enough to feel the timer behaviour that catches people out in production. Point a template at a KV path, change the value underneath it, and watch the clock rather than the file. The lab template holds one line, PASSWORD={{ .Data.data.password }}, reading secret/data/agentdemo.

vault kv put secret/agentdemo password=first
 
# agent-lab.hcl sets: template_config { static_secret_render_interval = "10s" }
vault agent -config=/etc/vault.d/agent-lab.hcl >/tmp/agent.log 2>&1 &
sleep 3 && cat /app/config/app.env
 
vault kv put secret/agentdemo password=second
sleep 5  && cat /app/config/app.env
sleep 10 && cat /app/config/app.env

== Secret Path ==
secret/data/agentdemo
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:40:02.118374Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
PASSWORD=first
 
== Secret Path ==
secret/data/agentdemo
 
======= Metadata =======
Key                Value
---                -----
created_time       2026-07-27T09:40:11.930221Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2
 
PASSWORD=first
PASSWORD=second

Five seconds after the write the file still says first; ten seconds after that it says second. Now delete the static_secret_render_interval line and run the whole thing again with the default in place. The file stays stale for five full minutes while every log line looks perfectly healthy, and you will recognise that symptom instantly the next time somebody swears Vault Agent has stopped working.

Takeaway

Agent's job ends when the file lands. Yours ends when the database agrees. In the lab you watched a rendered file sit five minutes behind a changed KV value, and in the rotation check you watched twelve connections stay open on a credential that had already been replaced; both of those look perfectly healthy in the Agent log. Run that pg_stat_activity query against your own service this week, and if the old credential is still holding the connections, the bug is in your reload path rather than anywhere in agent.hcl.

Related