Choosing a secrets manager

Vault, cloud-native, or SOPS — a decision tree.

Two offices, two ways of handling keys. In the first there is a locked drawer: the keys sit in a box, and everyone who needs one gets told the combination. In the second there is a staffed key desk. You show your badge, the clerk checks a list, hands you a key, writes the checkout in a logbook, and can have every lock re-cut overnight. A secrets manager is the key desk: a dedicated service that stores secrets encrypted, decides *per identity* who may read each one, records every read, and can rotate (replace) a secret from one central place. SOPS (short for Secrets OPerationS), the file encryption tool from the last lesson, is a well-run locked drawer. The encryption is real. There is no clerk and no logbook.

The leak a manager prevents is not the first one. Earlier lessons covered logs, ps output and git history. What a manager fixes is what comes after, when secrets sprawl. Once copies of one credential live in files and environment variables across twenty repositories, three questions stop having answers. Who holds this credential right now? When was it last used? How do you replace it without an outage? A secrets manager can answer all three, because there is exactly one authoritative copy, an audit log (a running record of every access, who read it and when), and a rotation path that changes the secret in one place instead of twenty.

What actually happens inside one

Nearly every manager pulls the same layered trick, the way a hotel keeps its master key locked in the safe and issues room keys from it. Each secret is locked with its own key, called a *data key*. The data keys are themselves locked by a *root key* that never leaves a KMS (key management service, a hardened service that will use its keys on your behalf but never hand them over) or an HSM (hardware security module, the same idea built as a tamper-resistant appliance). That arrangement has a name: envelope encryption. Nothing touches disk in plaintext, and the only door is an API (application programming interface, meaning a network request rather than a file you can copy). A read works the same way everywhere. The caller proves who it is, using an IAM role (identity and access management, the cloud's own record of who a workload is), a Kubernetes service account, or an OIDC token (OpenID Connect, a standard way one system proves identity to another). Then a policy is checked against the secret's path, the value is decrypted, and an audit event is written. Vault adds one idea on top. When it starts, its storage is sealed, meaning unreadable, until the root key is rebuilt from key shares or fetched from a cloud KMS. That second option is called auto-unseal. Vault can also mint dynamic secrets: it holds one admin login to, say, PostgreSQL, and creates a short-lived database account whenever an app asks for one. Most credentials then expire on their own instead of sitting around long enough to leak.

The three families

Encrypted files in git. SOPS with age or a cloud KMS key, or sealed-secrets running inside the cluster. There is no server to operate, it fits GitOps perfectly (GitOps being the practice of letting the git repository decide what runs in production), and you already know it from the last lesson. The limits come from the design, not from sloppiness. Values are static. Nothing audits reads, because git shows you who changed a file and never who decrypted it. Taking access away from someone who leaves means rotating keys, re-encrypting and redeploying.

Cloud-native managers. AWS Secrets Manager, Google Secret Manager, Azure Key Vault. The provider runs them, so availability is their pager and not yours, and access control rides on the IAM rules you already have. That last part is the feature that wins arguments: a pod or a virtual machine that already carries a platform identity can read its secrets with nothing stored on it at all. You pay per secret and per API call. The boundary is the obvious one. Each of them lives in one cloud.

# Create a secret (one JSON blob per logical secret is idiomatic)
aws secretsmanager create-secret \
  --name prod/payments/db \
  --secret-string '{"username":"app","password":"S3ttled-Horizon-91"}'
# -> {
# ->   "ARN": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/db-Ab12Cd",
# ->   "Name": "prod/payments/db",
# ->   "VersionId": "9f4c2e1a-77b0-4d2a-9c3e-5f8d1b6a0e42"
# -> }
 
# Read it back — this is the call your app makes at startup
aws secretsmanager get-secret-value \
  --secret-id prod/payments/db \
  --query SecretString --output text
# -> {"username":"app","password":"S3ttled-Horizon-91"}
 
# Rotation is an API, not a wiki page: a Lambda re-issues the credential on schedule
aws secretsmanager rotate-secret --secret-id prod/payments/db \
  --rotation-lambda-arn arn:aws:lambda:eu-west-1:111122223333:function:rotate-db \
  --rotation-rules '{"AutomaticallyAfterDays": 30}'

Vault, and OpenBao, the open-source fork that moved under the Linux Foundation after HashiCorp changed the license in 2023. OpenBao speaks the same API; you type bao where you used to type vault. This family works across clouds and on-prem, and it carries the deepest feature set: database credentials created on demand, an internal certificate authority (the service that issues the TLS certificates your systems use to trust each other), encryption as a service. The bill arrives in operations. You run the cluster, the storage, the unsealing, the upgrades and an on-call rotation for a service that everything else depends on.

# Dev mode: in-memory, auto-unsealed, root token printed. Learning ONLY — never production.
vault server -dev
# -> Root Token: hvs.6fT9wKxq...
 
export VAULT_ADDR=http://127.0.0.1:8200
vault kv put -mount=secret payments/db username=app password=S3ttled-Horizon-91
# -> == Secret Path ==
# -> secret/data/payments/db
# -> version    1
 
vault kv get -mount=secret -field=password payments/db
# -> S3ttled-Horizon-91
 
# The feature the other families lack: credentials that don't exist until requested
vault read database/creds/payments-app
# -> Key                Value
# -> lease_id           database/creds/payments-app/rGx4tYq8zW
# -> lease_duration     1h
# -> username           v-kubern-payments-app-x7Qb2M
# -> password           A1a-9wLpXnE3vRtC
# a fresh DB account, valid one hour, revoked automatically at expiry

The decision tree

Ask the questions in order. *Do you have fewer than a few dozen static secrets and a GitOps workflow?* Encrypted files in git are enough, and you settled that one last lesson. *Is everything in one cloud, with workloads that already carry IAM identities?* Use that cloud's manager. It is the smallest operational load, audit and rotation ship with it, and the policy language is one your team already reviews. *Are you multi-cloud or on-prem, or do you need database credentials that expire by themselves, or your own certificate authority?* Now you are in Vault and OpenBao territory, and only if you can staff it. An unstaffed Vault is worse than a well-run cloud manager. On Kubernetes, the External Secrets Operator (ESO, a controller that runs inside your cluster) keeps the decision reversible: it copies values out of whichever backend you picked into ordinary Kubernetes Secrets, so your manifests never learn which manager won.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-prod            # a ClusterSecretStore pointing at Secrets Manager
    kind: ClusterSecretStore
  target:
    name: payments-db         # the K8s Secret ESO creates and keeps in sync
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password
# kubectl get externalsecret payments-db
# -> NAME          STORE      REFRESH INTERVAL   STATUS         READY
# -> payments-db   aws-prod   1h                 SecretSynced   True

What breaks at scale

Cloud managers charge per secret and per call, roughly $0.40 per secret per month plus $0.05 per 10,000 API calls on AWS. A chatty fleet of services fetching on every request turns that into a real invoice and, sooner than you expect, throttled calls. Fetch once at startup and cache the value. The official client-side caching libraries and the Vault Agent sidecar exist for exactly this. SOPS scales badly in *people*: every joiner and every leaver means editing key lists and re-encrypting across repositories. Vault scales badly in *criticality*. It becomes tier-zero infrastructure, and while it is down nothing can start, so plan snapshots, disaster recovery and capacity before the first production secret lands. And no manager fixes hygiene downstream. An app that reads a secret and then logs it has leaked it, exactly as in lesson one.

Run every candidate past the same scorecard. Workloads authenticate with platform identity, not a stored token. Credentials are short-lived wherever the backend allows it. The audit log ships somewhere a human actually looks. Rotation has been *performed* at least once, not merely configured. There is a written break-glass path for the day the manager itself is down. Then take the cheapest rung of the ladder that passes all five. Moving up later is a data migration, not a rewrite, especially if ESO already sits between your cluster and the backend.

If your tree ended on the third rung, the next course, *Vault in production*, picks up right there: high-availability clusters on integrated Raft storage, auto-unseal, policy design, and dynamic secrets past the demo stage. If it ended on the first or second rung, you are finished, and finishing cheaply is a good outcome.

Say finance asks why you want Vault when "AWS Secrets Manager already exists." Answer with constraints, not enthusiasm. Multi-cloud, database credentials that expire on their own and an internal certificate authority are Vault's lane. Single-cloud reads that ride on IAM are the cloud manager's lane. Five static secrets in a GitOps repo are SOPS's lane. Buying the richest tool for the poorest problem buys you an on-call surface you cannot staff.

External Secrets Operator keeps Kubernetes manifests honest, because they point at a store rather than a vendor logo, and that is what makes a migration reversible. Keep one thing in mind, though. What ESO produces is an ordinary Kubernetes Secret, so everything you learned about etcd and RBAC (role-based access control, the rules for who may read which objects) still applies to the synced copies.

Try this

Have AWS credentials for a sandbox account? Create one secret and read it back. If not, start Vault in dev mode and push a value through kv put and kv get. The goal is to feel the API as the single copy, with nothing on disk for anyone to grep.

# Option A — AWS sandbox
aws secretsmanager create-secret --name tmp/sf-demo --secret-string '{"password":"ThrowAway-Only"}'
aws secretsmanager get-secret-value --secret-id tmp/sf-demo --query SecretString --output text
# Option B — Vault dev
vault server -dev > /tmp/vault-dev.log 2>&1 &
sleep 1; export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=$(awk "/Root Token/ {print \$3}" /tmp/vault-dev.log)
vault kv put -mount=secret demo/db password=ThrowAway-Only
vault kv get -mount=secret -field=password demo/db

{"password":"ThrowAway-Only"}
# or
ThrowAway-Only
# one authoritative read path — now ask how your app authenticates without a long-lived token

Takeaway

Climb the ladder only as far as your problem pushes you: encrypted files in git, then a cloud manager, then Vault or OpenBao. And authenticate with platform identity, or you will have rebuilt the same sprawl in the shape of secret zero.

Next, score whatever you run today on three things. Has rotation actually been performed, or only configured? Does anyone read the audit log? What is the plan for the hour the manager is down? Move up a rung when a real pain shows up, and not a day earlier.

Related