Cloud secret managers, briefly

AWS/GCP/Azure managers without the deep dive.

A safe-deposit box at a bank is a boring steel drawer. The steel is not the thing keeping strangers out of it. What keeps them out is the clerk who checks your face against a signature card before the vault door swings open, plus the ledger that records every visit. Cloud secret managers work the same way. AWS Secrets Manager, Google Secret Manager and Azure Key Vault all encrypt what they hold, and that part is close to uninteresting. The part that decides whether your database password stays yours is the identity check at the counter and the log of who walked in.

These are managed secret stores. You hand the provider a string, it keeps that string encrypted, and it hands the string back to callers who pass an identity check. AWS ships two of them: Secrets Manager, and Parameter Store, which lives inside SSM (Systems Manager, the AWS operations toolbox). Google has Secret Manager, Azure has Key Vault. Each one does the same three things: it encrypts what it stores, it checks who is asking before every read, and it logs the request. The encryption uses the provider's KMS (Key Management Service, the cloud's own hardware-backed key locker). The check uses the provider's IAM (Identity and Access Management, the rules engine that decides which identity may call which operation). The log records API calls, and an API (application programming interface) is just the door one program uses to talk to another.

This lesson is a map of that shape rather than a tour of every feature. You will run the AWS path end to end, see the same two ideas expressed in Google Cloud and Azure, and then spend most of your attention on the three places teams actually get hurt: the identity your caller presents, the route the value travels to reach your running code, and what stays readable after you rotate.

What the Manager Actually Buys You

Before a manager, one production database password lives in nine places: a developer laptop, a .env file, a continuous integration variable, three Kubernetes Secrets in three namespaces, a Terraform state file, a wiki page, and a Slack thread from 2023. Rotating it means finding all nine, and you will miss two. After a manager, there is one authoritative copy and eight references to it. That change alone is most of the value, and it lands before any encryption feature does.

Three concrete things you get. First, encryption at rest with a key you can point at: the stored bytes are ciphertext, and the key that decrypts them carries its own separate access policy, so "who may read this secret" and "who may use this key" become two independent locks. Second, an access decision on every single read, evaluated against the calling identity rather than against a network location or a password everybody on the team already knows. Third, a record of every call, which turns "someone must have leaked it" from a hallway argument into a query you can run.

Three things you do not get, and should stop expecting. There is no uniformity across clouds: operation names, identity models and audit formats differ enough that a tidy multi-cloud wrapper quietly becomes a small internal product you now maintain. There is no dynamic-secret engine of the kind HashiCorp Vault offers, where the store mints a brand-new database user per request and deletes it an hour later. AWS rotation swaps a static value on a schedule, which is a weaker guarantee wearing similar clothes. And there is no policy language as expressive as Rego (the language Open Policy Agent uses to write rules); you get IAM conditions, which handle "this role, this resource, this tag" well and handle anything shaped like a sentence badly.

The AWS Path, Start to Finish

Every access decision starts with a question the command line can answer for you: who am I right now? The answer comes from STS (Security Token Service, the part of AWS that hands out temporary credentials and knows who is holding them). Get in the habit of asking before you debug anything else.

# Which identity is this shell using?
aws sts get-caller-identity

{
    "UserId": "AROA4EXAMPLEID:app-deploy",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/app-deploy/app-deploy"
}

# Lab account, throwaway value. Note the value is on the command line here
# purely to keep the example readable. In real life pass a file, so the
# password never lands in your shell history:
#   --secret-string file://payments-db.json
aws secretsmanager create-secret \
  --name prod/payments/db \
  --description "Payments service Postgres credentials" \
  --secret-string '{"username":"payments","password":"Rk8-tQ2v-Lm4x-Zp71"}'

{
    "ARN": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/db-a1B2c3",
    "Name": "prod/payments/db",
    "VersionId": "8f3d5c2a-91b4-4e77-9a10-7c6be2d0f145"
}

Look closely at that ARN (Amazon Resource Name, the globally unique postal address of an AWS object). The name you chose was prod/payments/db, but the ARN ends prod/payments/db-a1B2c3. AWS appends a hyphen and six random characters, and it picks fresh ones if you delete the secret and create it again. Any IAM policy naming the exact ARN you copied today will silently stop matching tomorrow, which is why tired people give up and write Resource: "*". Write the resource as prod/payments/db-?????? instead. Each ? matches exactly one character, so six of them match the suffix and nothing longer. The lazier-looking prod/payments/db-* would also match a completely different secret named prod/payments/db-backup, whose ARN might be prod/payments/db-backup-x9Y2z1.

# Read it back the way an application would
aws secretsmanager get-secret-value \
  --secret-id prod/payments/db \
  --query SecretString --output text

{"username":"payments","password":"Rk8-tQ2v-Lm4x-Zp71"}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOnlyThisOneSecret",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/db-??????"
    },
    {
      "Sid": "DecryptOnlyViaSecretsManager",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/3f1c9a77-4b02-4d3e-91aa-0e6c5b7d2f10",
      "Condition": {
        "StringEquals": { "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com" }
      }
    }
  ]
}

That second statement is the one people forget. If the secret uses the default aws/secretsmanager key you can leave it out, because that AWS-managed key's own policy already permits principals in the same account to use it through Secrets Manager. The moment you move production onto a customer-managed key, which is what you need for cross-account sharing, your own key policy and your own rotation schedule, kms:Decrypt becomes mandatory on the caller side. Its absence produces the most common ticket in this whole area: "I definitely granted GetSecretValue and it still says access denied." The kms:ViaService condition then stops that same role borrowing the key to decrypt some other ciphertext an attacker drops in a bucket.

Watching the Deny Actually Happen

A control you have never watched fail is a control you are guessing about. Grant the read to one role, then prove a second role bounces off it.

# Become a role that was never granted this secret. The round brackets matter:
# everything inside them runs in a subshell, so the borrowed credentials live
# for three lines and die at the closing bracket. Nothing touches disk either.
(
  creds=$(aws sts assume-role \
    --role-arn arn:aws:iam::111122223333:role/reports-reader \
    --role-session-name deny-test \
    --query Credentials --output json)
 
  export AWS_ACCESS_KEY_ID=$(echo "$creds" | jq -r .AccessKeyId)
  export AWS_SECRET_ACCESS_KEY=$(echo "$creds" | jq -r .SecretAccessKey)
  export AWS_SESSION_TOKEN=$(echo "$creds" | jq -r .SessionToken)
 
  aws secretsmanager get-secret-value --secret-id prod/payments/db
)
 
# Outside the brackets you are yourself again. Check, do not assume.
aws sts get-caller-identity --query Arn --output text

An error occurred (AccessDeniedException) when calling the GetSecretValue operation:
User: arn:aws:sts::111122223333:assumed-role/reports-reader/deny-test is not authorized
to perform: secretsmanager:GetSecretValue on resource:
arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/db-a1B2c3
because no identity-based policy allows the secretsmanager:GetSecretValue action
# EXPECTED - this is the control working
 
arn:aws:sts::111122223333:assumed-role/app-deploy/app-deploy
# SUCCESS - the borrowed credentials are gone, so the next command is yours

Read the last line of that error every time. AWS tells you which side is missing. "No identity-based policy allows" points at the caller's role. "No resource-based policy allows" points at the secret's own policy. A message naming kms:Decrypt points at the key. Three different fixes, and guessing between them is how Principal: "*" ends up in a resource policy at 11pm and stays there for two years. The brackets around that assume-role matter almost as much as the error does. Export those three variables into your ordinary shell and every command you run afterwards is the deliberately powerless test role, including the CloudTrail query below, which would answer with a denial you then waste an hour misreading.

# Did both the success and the denial get recorded? Wait about fifteen minutes
# before running this. CloudTrail is not live, and an empty array straight after
# the deny means 'not indexed yet', not 'your audit trail is broken'.
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --max-results 2 \
  --query 'Events[].{time:EventTime,who:Username,event:EventName}'

[
    {
        "time": "2026-07-27T09:44:03+00:00",
        "who": "deny-test",
        "event": "GetSecretValue"
    },
    {
        "time": "2026-07-27T09:41:12+00:00",
        "who": "app-deploy",
        "event": "GetSecretValue"
    }
]

Both calls are there, the allowed one and the refused one, newest first. CloudTrail (the AWS service that keeps a running log of API calls) records Secrets Manager reads with no configuration from you, and it keeps roughly the last ninety days searchable without a trail of your own. Enjoy that, because it is the exception rather than the rule.

The Same Two Ideas in GCP and Azure

# Google splits the secret (a named container) from its versions (the values)
gcloud secrets create payments-db --replication-policy=automatic
printf '%s' 'Rk8-tQ2v-Lm4x-Zp71' | gcloud secrets versions add payments-db --data-file=-
 
# Grant one service account read access to exactly this one secret
gcloud secrets add-iam-policy-binding payments-db \
  --member="serviceAccount:payments@my-project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"
 
gcloud secrets versions access latest --secret=payments-db

Created secret [payments-db].
Created version [1] of the secret [payments-db].
Updated IAM policy for secret [payments-db].
bindings:
- members:
  - serviceAccount:payments@my-project.iam.gserviceaccount.com
  role: roles/secretmanager.secretAccessor
etag: BwYb1kQ2mXo=
version: 1
Rk8-tQ2v-Lm4x-Zp71

Note printf '%s' rather than echo. echo adds a newline character on the end, that newline becomes part of the stored secret, and you then spend a cheerful afternoon wondering why a password that looks identical in every log refuses to authenticate. Note also where the binding is attached: to the individual secret. Granting roles/secretmanager.secretAccessor at project level instead hands the caller every secret in the project, which is the Google-shaped version of Resource: "*" and roughly as common.

# Azure: Key Vault stores secrets, keys and certificates; this is the secret plane
az keyvault secret set \
  --vault-name kv-payments-prod \
  --name payments-db \
  --value 'Rk8-tQ2v-Lm4x-Zp71' \
  --query id -o tsv
 
# On a vault created with --enable-rbac-authorization, access to the data is an
# Azure role assignment (RBAC, role-based access control) rather than a legacy
# per-vault access policy
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee 8f1e2c7a-0b43-4d92-9a6f-51c3e0d7b284 \
  --scope "/subscriptions/$SUB/resourceGroups/rg-payments/providers/Microsoft.KeyVault/vaults/kv-payments-prod"
 
az keyvault secret show --vault-name kv-payments-prod --name payments-db --query value -o tsv

https://kv-payments-prod.vault.azure.net/secrets/payments-db/3c9f0b41d7a24e6f9d0c2b7a5e13f882
Rk8-tQ2v-Lm4x-Zp71

Same story, three dialects. A named container holds an immutable value, a policy attached to an identity decides who may fetch it, and the platform encrypts the bytes on disk. What differs day to day is granularity. Azure's cleanest access boundary is the vault itself, so the usual pattern is one vault per application per environment. Google and AWS scope down to the individual secret. Pick your naming prefixes early, something like prod/payments/db and staging/payments/db, because policies key off those paths and renaming a secret later means editing every policy that mentions it.

Who You Are When You Ask

Back to the clerk at the counter. The whole scheme falls over if your application's identity card is itself a laminated badge somebody printed once and left in an unlocked drawer. That is exactly what happens when a team creates an IAM user, generates a long-lived access key pair, and stores both in a Kubernetes Secret so the app can call GetSecretValue. The manager now holds ten secrets, the cluster holds an eleventh, and the eleventh opens the other ten. It never expires, it shows up in env output and crash dumps like any other environment variable, and rotating it is a manual chore nobody puts on a calendar.

The fix is platform identity: the cloud already knows which workload is running and will mint short-lived credentials on the spot. On EKS (Elastic Kubernetes Service, the managed Kubernetes offering) this is IRSA, IAM Roles for Service Accounts. The Kubernetes API server issues a signed OIDC token (OpenID Connect, a standard way for one system to vouch for an identity), the kubelet drops that token into the pod's filesystem and refreshes it before it expires, and the AWS SDK (software development kit, the AWS client library your code imports) swaps the token for temporary keys through sts:AssumeRoleWithWebIdentity. Those keys last about an hour and the SDK renews them for you. EKS Pod Identity is a newer flavour of the same trick with less OIDC plumbing to set up. On plain virtual machines (EC2, Elastic Compute Cloud) or ECS (Elastic Container Service) it is the instance or task role served by the metadata service, and you should enforce IMDSv2 (Instance Metadata Service version 2), because the older version is why a single request-forgery bug in a web application used to turn straight into stolen cloud credentials. Google calls its version Workload Identity Federation for GKE (Google Kubernetes Engine), Azure calls it managed identity. One idea: no stored key anywhere.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments
  namespace: payments
  annotations:
    # The only "credential" in this manifest is the NAME of a role
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/payments-secret-reader
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels: { app: payments-api }
  template:
    metadata:
      labels: { app: payments-api }
    spec:
      serviceAccountName: payments
      containers:
        - name: api
          image: ghcr.io/acme/payments-api:1.9.3
          env:
            - name: AWS_REGION
              value: eu-west-1
            - name: DB_SECRET_ID
              value: prod/payments/db   # a pointer is safe to commit; a value is not

# Prove the pod has an identity without holding a key
kubectl exec -n payments deploy/payments-api -- aws sts get-caller-identity
kubectl exec -n payments deploy/payments-api -- env | grep -c AWS_SECRET_ACCESS_KEY

{
    "UserId": "AROAV3EXAMPLEID:botocore-session-1785312061",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/payments-secret-reader/botocore-session-1785312061"
}
0
# SUCCESS - a real AWS identity, and zero long-lived keys in the environment

Getting the Value Into the Process

The manager solved storage. Delivery is still entirely your problem, and delivery is where the old failures walk back in. Three routes are common in Kubernetes, and they differ mainly in how many durable copies of the password they leave lying around afterwards.

Calling the SDK from your own code creates the fewest copies: the value lives in process memory and nowhere else. Cache it in the process rather than fetching it per request, because AWS bills roughly $0.05 per 10,000 API calls and every provider throttles you eventually. Mounting it with the Secrets Store CSI Driver (Container Storage Interface, the standard plug-in system Kubernetes uses for volumes) puts the value in a file inside a tmpfs volume, a filesystem that lives in memory and is never written to the node's disk, and it never reaches etcd (the cluster's own database, where Kubernetes keeps everything it knows) unless you explicitly ask it to. Syncing it with External Secrets Operator copies the value into an ordinary Kubernetes Secret, which is convenient and easy to manage from git, and which lands the value in etcd, where everything from the Kubernetes Secrets lesson applies again: base64 is an encoding, a reversible way of writing bytes as text, not encryption of any kind, so anyone holding get secrets in that namespace reads the password, and encryption at rest for etcd is a separate switch you have to go and throw.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payments-db
  namespace: payments
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/payments/db"
        objectType: "secretsmanager"
        # jmesPath pulls one field out of the stored JSON document and
        # writes it to its own file, so the app never sees the whole blob
        jmesPath:
          - path: "password"
            objectAlias: "db_password"
 
  # Uncomment this and the value ALSO becomes a durable Kubernetes Secret
  # stored in etcd (and the driver must be installed with syncSecret enabled).
  # Leave it commented and the value only ever exists as a file in the pod's
  # tmpfs volume, gone the moment the pod dies.
  # secretObjects:
  #   - secretName: payments-db
  #     type: Opaque
  #     data:
  #       - objectName: db_password
  #         key: password

That commented block is the entire decision, in eight lines. Teams switch it on because their application reads DB_PASSWORD from the environment and nobody wants to touch the code, then wonder why adopting a cloud secret manager did not shrink their exposure. It did not, because the delivery path still ends in an environment variable: readable through /proc/<pid>/environ by anything running as the same user, present in a crash dump, and inherited by every child process the app spawns. Reading a file at /mnt/secrets/db_password costs four lines of code and removes two copies. That file does not appear on its own, though. The class above only says what to fetch; the pod spec has to say where it lands, and those two stanzas are the ones people leave out.

# The SecretProviderClass says WHAT to fetch. This says WHERE it lands.
# Without both stanzas below there is no file under /mnt/secrets at all.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  template:
    spec:
      serviceAccountName: payments
      containers:
        - name: api
          image: ghcr.io/acme/payments-api:1.9.3
          volumeMounts:
            - name: db-secret
              mountPath: /mnt/secrets   # + objectAlias db_password = the file path
              readOnly: true
      volumes:
        - name: db-secret
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: payments-db

Rotation, Versions, and What Stays Readable

Changing the locks on a building is half a job if the old key still hangs in the hallway on a hook labelled "previous". Cloud managers keep that hook on purpose. AWS Secrets Manager tracks versions with staging labels: AWSCURRENT is what a normal read returns, AWSPENDING is a new value being tested part-way through a rotation, and AWSPREVIOUS is whatever was current before the last one. A rotation function (a small Lambda you attach to the secret) walks four steps, createSecret, setSecret, testSecret and finishSecret, and only that last step moves the labels.

# Assumes a rotation function is already attached to this secret
aws secretsmanager rotate-secret --secret-id prod/payments/db --rotate-immediately > /dev/null
 
# The current value changed. What happened to the old one?
aws secretsmanager get-secret-value --secret-id prod/payments/db \
  --version-stage AWSPREVIOUS --query SecretString --output text

{"username":"payments","password":"Rk8-tQ2v-Lm4x-Zp71"}
# The OLD password, still readable by anyone holding GetSecretValue

That behaviour exists for a good reason: a consumer that cached the old value halfway through a deploy needs something to fall back on while it restarts. It also means "we rotated it" and "the leaked string is dead" are two separate claims, and people say the first while meaning the second. If you rotated because the value leaked, go and revoke it at the database, the API provider, or wherever the string is actually honoured. A secret manager stores strings. It has no power at all to invalidate a string some other system still accepts. Same lesson as purging git history, arriving from a different direction.

Rotation also breaks in a way calendars hide. The rotation function updates the store and the database, then reports success in green. Meanwhile the analytics job that read the password once at boot three weeks ago carries on with the old one, works fine for a while, then fails at 3am on a Sunday with an authentication error nobody connects to a rotation that went green weeks earlier. Force a rotation in staging, leave it running for a full deploy cycle, and watch which services quietly start erroring. A partial rotation you believe in is worse than an honest stale secret with a named owner.

The Honest Trade-offs

Cost shapes these designs more than security does, so say it out loud. At roughly $0.40 per secret per month, 500 secrets costs about $200 a month before a single API call. The predictable response is to cram twelve unrelated values into one JSON document (JavaScript Object Notation, the plain-text format those examples use), which welds their lifecycles together: rotating the SMTP password, the one your mail-sending service uses, now means writing a new version of an object that also holds the database password, and every consumer of either value is in the blast radius. Parameter Store standard parameters are free and often the right home for the boring 90%.

The manager also becomes a hard startup dependency. If your pods fetch secrets on boot and the regional endpoint is degraded, perfectly healthy nodes cannot start pods, and your outage is now somebody else's control plane. Decide in advance: cache to an encrypted file with a short expiry, replicate the secret to a second region, or accept out loud that a Secrets Manager outage is a payments outage. All three are defensible. Discovering during the outage that you never chose is not.

Deletion has teeth too. aws secretsmanager delete-secret schedules removal behind a recovery window of 7 to 30 days, defaulting to 30, and will not free the name until then unless you pass --force-delete-without-recovery. Azure Key Vault enables soft-delete by default with 90-day retention, so a "deleted" secret stays recoverable and its name stays reserved until somebody purges it. Treat that as a safety net when a colleague fat-fingers a cleanup script, and as a nasty surprise when a compliance rule says a value must be unrecoverable within 24 hours.

Cross-account reads need three things lined up at once: an IAM policy on the caller, a resource policy on the secret, and a customer-managed KMS key, because the default aws/secretsmanager key cannot be shared across accounts. Miss any one and the call fails closed, which is precisely the moment somebody reaches for a wildcard principal to make a demo work and never goes back to tighten it. Portability is real but limited: External Secrets Operator lets your manifests reference an abstract store instead of a vendor name, so migrating becomes a configuration change rather than a rewrite, while rotation functions, IAM conditions and audit queries stay firmly vendor-shaped.

Try this

Do the deny loop for real, on a lab account. Create a throwaway secret, write the reader policy with prod/payments/db-?????? in it, and read the value back as yourself. Then create a second secret named prod/payments/db-backup, change the policy's resource to prod/payments/db-*, and read the backup with the same role: the lazier wildcard just handed out a secret you never meant to grant. Put the six question marks back, run the assume-role deny from the subshell above, then go and make a coffee and come back to find both events sitting in CloudTrail.

Takeaway

The manager is the clerk and the ledger, not the vault. Encryption at rest is the part you never have to think about; the identity policy, the route the value takes into your process, and the revocation at the database are the parts that go wrong. Of everything here, the one thing that cannot be fixed after the fact is logging. Switch on Data Access audit logs in Google Cloud and attach a diagnostic setting to every Key Vault this week, because neither is retroactive and you will want them on the day somebody asks who read it.

Next: rotation habits that hold up without heroics, including dual-run windows, named owners, and how to tell a rotation that finished from one that only reported success.

Related