KMS encryption essentials

CMKs, key policies, and envelope encryption.

A bank vault runs on one rule the manager will not bend: the master keys never leave the building. Hand the teller something small and it goes straight into the vault. Need to lock up a crate that will never fit through the door? You ask for a fresh padlock and its key, snap the padlock shut on your own crate, then hand the padlock key back to be sealed inside. What you can never do is walk out holding a master key. That single rule is the whole design of KMS (Key Management Service, the AWS service that stores encryption keys and does the locking on your behalf), and nearly every strange thing about it falls out of that rule.

The thing living in that vault is a KMS key. Older documentation calls it a CMK (customer master key); AWS retired that name back in 2021, so a blog post still using it is telling you how old it is. Three kinds exist and only one is properly yours. A customer managed key is one you create: you write its policy, choose its rotation schedule, and can schedule it for deletion. An AWS managed key shows up on its own the first time you switch on encryption in a service, carries a name like aws/s3 (for Simple Storage Service, the object store) or aws/sqs (Simple Queue Service), rotates once a year whether you want it to or not, and has a policy you are not allowed to edit. An AWS owned key sits in an account AWS controls, is shared across many customers, costs nothing, and never appears in your key list at all. Everything below happens on the first kind, the only one whose rules you get to write.

The Vault Never Hands Over the Master Key

KMS will encrypt small things for you directly, through the kms:Encrypt call, and with a symmetric key it refuses anything larger than 4,096 bytes. That ceiling is not a temporary limitation waiting to be raised. It is a hint about how the service expects to be used. Your 40 GB (gigabyte) database export never goes near the vault. Instead you ask KMS for a brand new random key of its own making, called a data key, and it comes back to you twice: once in the clear so you can use it right now, and once wrapped inside your KMS key so you can store it somewhere ordinary. You encrypt the 40 GB on your own machine with the clear copy, wipe that copy, and file the wrapped copy next to the ciphertext. Those two copies of one key are what people mean by envelope encryption, and it is what every AWS service quietly does for you.

KEY=$(aws kms create-key \
  --description "billing exports data key wrapper" \
  --query KeyMetadata.KeyId --output text)
 
aws kms create-alias --alias-name alias/billing-exports --target-key-id "$KEY"
 
aws kms generate-data-key --key-id alias/billing-exports --key-spec AES_256

{
    "CiphertextBlob": "AQIDAHjRb0Zt1yQ9K2m7fVQ4Xg8pLd6ScVhT0nWqYzB1mE9K...Zk3d2Q==",
    "Plaintext": "8Kx1sJmQ0pVn5rTgWc7yZaE3hLbN9uDf4iRoP2vXsA0=",
    "KeyId": "arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7081-92a3-b4c5d6e7f809"
}

Read those first two fields as the same 32 bytes written down twice. Plaintext is live key material, base64-encoded only because JSON has no way to carry raw binary, and it is sitting in your terminal scrollback at this moment. Base64 is an encoding, not a lock; anyone who reads it can decode it in a second. CiphertextBlob is that identical material sealed under the key named in KeyId, which is why it is safe in a database column or a file on disk. The clear copy has to die the instant the encryption finishes. A process that keeps a plaintext data key alive in memory for hours has turned careful envelope encryption back into one object an attacker can steal. Here is the round trip in a shell.

aws kms generate-data-key --key-id alias/billing-exports --key-spec AES_256 \
  --query '[Plaintext,CiphertextBlob]' --output text > dk.txt
read -r DK WRAPPED < dk.txt
 
# use the clear copy locally, then throw it away
DK_HEX=$(printf '%s' "$DK" | base64 -d | xxd -p -c 64)
IV=$(openssl rand -hex 16)
openssl enc -aes-256-cbc -K "$DK_HEX" -iv "$IV" -in export.csv -out export.csv.enc
 
# keep the wrapped key and the IV forever, next to the ciphertext
printf '%s' "$WRAPPED" | base64 -d > export.csv.key
printf '%s' "$IV" > export.csv.iv
shred -u dk.txt
unset DK DK_HEX
ls -l export.csv.enc export.csv.iv export.csv.key

-rw-r--r-- 1 you you 4193920 Jul 27 10:14 export.csv.enc
-rw-r--r-- 1 you you      32 Jul 27 10:14 export.csv.iv
-rw-r--r-- 1 you you     184 Jul 27 10:14 export.csv.key

One hundred and eighty four bytes carry the entire security story of a four megabyte file. Lose export.csv.key and the data is gone for good. Leak it and very little happens, because a wrapped key is dead weight without a principal allowed to call kms:Decrypt on the key that wrapped it. That is exactly why those files sit there readable by every user on the box and nobody should panic. Now read the same snippet as an attacker would. The key material goes onto the command line in -K "$DK_HEX", where any local user running ps can read it while openssl runs. AES-CBC (Advanced Encryption Standard in cipher block chaining mode) has no tamper check, because openssl enc flatly refuses to do AES-GCM (Galois/Counter Mode, the variant that adds an authentication tag), so someone who flips bytes in the middle of your ciphertext gets garbage on decrypt instead of a loud failure. And if you forget the IV (initialization vector, the random starting value the cipher needs) the file is unreadable forever. Three real holes in eleven lines. Build envelopes with the AWS Encryption SDK (software development kit) or your language's crypto library, and keep shell for reading, not for sealing.

Switch on SSE-KMS (server-side encryption with KMS keys) on an S3 bucket, an EBS volume (Elastic Block Store, the disk you attach to a server), an SQS queue or a Secrets Manager secret, and AWS performs this exact dance for you on every read and write. The convenience is real. It is also why so many teams have encryption turned on everywhere and still cannot answer the only question that matters: who is allowed to call kms:Decrypt.

Two Rulebooks, and One Is Never Blank

A safe-deposit box answers to two separate rulebooks. The bank decides who is allowed into the vault room at all. The box itself carries a signature card naming who may open that particular box. Satisfy one and not the other and you go home empty-handed. KMS works the same way, with a twist that catches people out: the signature card is never blank. Every KMS key carries a key policy, it is mandatory, and it is the primary control. An IAM policy (Identity and Access Management, the service that decides what your users and roles may do) that grants kms:Decrypt counts for absolutely nothing unless that key's own policy has agreed to let IAM have a say.

aws kms get-key-policy --key-id alias/billing-exports \
  --policy-name default --query Policy --output text | jq .

{
  "Version": "2012-10-17",
  "Id": "key-default-1",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
      "Action": "kms:*",
      "Resource": "*"
    }
  ]
}

That statement looks terrifying and is widely misread. The word root there is not the root user, and the statement does not hand kms:* to everybody in the account. Read it as a delegation: the key agrees that account 111122223333's own IAM policies may decide who uses it. Nobody receives a single permission until some IAM policy says so. Resource: "*" is misleading in the same way. Inside a key policy it means this key and nothing else, because a key policy has no vocabulary for talking about any other key.

The exact order of evaluation is worth holding in your head, because it explains almost every KMS access ticket you will ever pick up. A request to use a key succeeds when three things line up at once. First, no explicit Deny anywhere: key policy, identity policy, permission boundary, session policy, or an SCP (service control policy, an organization-wide ceiling on what an account is allowed to do). A single explicit Deny in any of those beats every Allow ever written. Second, the key policy allows the call, either by naming the principal directly or by delegating to IAM the way the default policy does. Third, something on the identity side says yes: an IAM policy, or a grant, which can supply that yes on its own. Boundaries and SCPs never grant anything; they only subtract. Which is why "I added the IAM policy and it still fails" is usually a key policy that never delegated, and "it works in dev but not in prod" is usually an SCP nobody mentioned.

Now watch that harmless-looking delegation turn into an incident. Attach {"Effect":"Allow","Action":"kms:Decrypt","Resource":"*"} to the role a Lambda function runs as. Every key in the account carrying the default policy has already handed the decision to IAM, and IAM has now said yes to all of them. That one role can read the SSE-KMS objects in every bucket, every value in Secrets Manager, and every encrypted EBS snapshot the account owns. One wildcard on one line, and it survives code review over and over, because kms:Decrypt reads as narrow and harmless and read-only.

A key policy worth having splits the duties in two. Admins can manage the key and cannot read data with it. The application can use the key and cannot change it. Neither side can quietly grant itself the other half.

{
  "Version": "2012-10-17",
  "Id": "billing-exports",
  "Statement": [
    {
      "Sid": "AdminsManageTheKeyButCannotReadData",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/kms-admin" },
      "Action": [
        "kms:DescribeKey", "kms:GetKeyPolicy", "kms:PutKeyPolicy",
        "kms:EnableKeyRotation", "kms:DisableKeyRotation",
        "kms:TagResource", "kms:ListGrants", "kms:RevokeGrant",
        "kms:DisableKey", "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AppUsesTheKeyOnlyThroughS3",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/billing-exporter" },
      "Action": ["kms:Decrypt", "kms:GenerateDataKey*", "kms:DescribeKey"],
      "Resource": "*",
      "Condition": {
        "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" },
        "StringLike": {
          "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::billing-exports-prod/*"
        }
      }
    }
  ]
}

kms:ViaService is the sharpest condition key in KMS and one of the least used. It demands that the request arrive through a named service acting on the principal's behalf, so a stolen credential cannot point aws kms decrypt at some blob the attacker found in an old backup. Be honest about where the fence ends, though. That same stolen credential can still call s3:GetObject, and S3 will happily decrypt the object and hand back the plaintext. ViaService narrows the tool, not the outcome, and it is no substitute for scoping the S3 permission with the same care.

Crossing an account boundary needs both rulebooks at once, with no shortcuts. The key policy must name the external account or a principal inside it, and that account's own IAM policy must separately allow the action on your key's ARN. Neither half does anything alone. When a cross-account decrypt "should obviously work" and does not, you will find exactly one of the two halves in place roughly every time.

Encryption Context, the Part Everyone Skips

Write the recipient's name across the flap of a tamper-evident envelope in permanent ink, and agree with the courier that the seal only breaks if the name still matches. Anybody can read the name. Nobody can change it without wrecking the envelope. That is encryption context, known in cryptography as AAD (additional authenticated data): plain key-value pairs, not secret, not encrypted, but bound so tightly to the ciphertext that decryption fails unless you present them again exactly, right down to letter case. They also land in your CloudTrail logs, which turns an anonymous Decrypt line into one that names the tenant and the object.

aws kms encrypt --key-id alias/billing-exports \
  --plaintext fileb://token.txt \
  --encryption-context tenant=acme,purpose=invoice \
  --query CiphertextBlob --output text | base64 -d > token.enc
 
# same key, same ciphertext, one word changed in the context
aws kms decrypt --ciphertext-blob fileb://token.enc --key-id alias/billing-exports \
  --encryption-context tenant=globex,purpose=invoice \
  --query Plaintext --output text || true
 
# and now with the exact context it was sealed under
aws kms decrypt --ciphertext-blob fileb://token.enc --key-id alias/billing-exports \
  --encryption-context tenant=acme,purpose=invoice \
  --query Plaintext --output text | base64 -d

An error occurred (InvalidCiphertextException) when calling the Decrypt operation: 
sk_live_9f2c41ab7d0e

Look closely at that first line. The message after the colon is empty, and that is not a formatting mistake in this lesson. KMS deliberately tells you nothing, because explaining why a ciphertext was rejected would help an attacker probe it. Correct behaviour, and infuriating at three in the morning. Learn the shape of it, because a wrong or missing encryption context is far and away the most common cause. Notice the --key-id on both decrypt calls too. For a symmetric key that flag is optional, since the key identifier travels inside the blob, and leaving it off means whoever handed you that blob gets to choose which of your keys does the work. Name the key you expect and let it fail loudly when the blob disagrees.

This is also the fix for the classic confused deputy problem. A multi-tenant service holds one role and one key, and decrypts whatever blob a caller passes in. With no context bound to the data, tenant B submits tenant A's blob and reads it, because from where KMS is standing the same permitted role asked the same permitted question about a valid ciphertext. Bind tenant= into the context at encrypt time, condition the permission on it, and tenant B's session cannot open tenant A's data even though the role and the key never changed.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DecryptOnlyThisSessionsTenant",
    "Effect": "Allow",
    "Action": "kms:Decrypt",
    "Resource": "arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7081-92a3-b4c5d6e7f809",
    "Condition": {
      "StringEquals": {
        "kms:EncryptionContext:tenant": "${aws:PrincipalTag/tenant}"
      }
    }
  }]
}

${aws:PrincipalTag/tenant} reads the session tag set at the moment the role was assumed, so one policy document covers ten thousand tenants while the permission that actually applies narrows to a single one per session. Tag the keys themselves the same way, with something like owner and data-class, then condition on aws:ResourceTag/data-class to keep general-purpose roles away from anything marked restricted. Untagged keys are how you end up with orphans: a key nobody can name, protecting data nobody can identify.

Rotation, Grants, and Things That Should Expire

Rotation is a locksmith swapping the cylinder in your front door while keeping every old cylinder in a labelled drawer. New locking uses the new cylinder; anything locked with an old one still opens. Turn on automatic rotation and KMS generates fresh backing material on a schedule and starts using it for new encrypt calls. The key ID does not change. The ARN does not change. The alias does not change. Your application never notices anything happened. Every earlier piece of material stays inside the key permanently, which is why a blob written three years ago still decrypts and why nothing on your disks ever needs rewriting.

Be clear-eyed about what rotation actually buys you. It caps how much data any single piece of material has ever protected, which genuinely matters for long-lived archives and for the auditor who asks. It does nothing at all against the threat most people picture when they enable it. A leaked credential still calls kms:Decrypt, and KMS quietly reaches into the drawer and picks the right old cylinder for the attacker. If you need old ciphertext to stop opening, you re-encrypt the data or you retire the key, and there is no third option. The default period is 365 days, you can set anything from 90 to 2,560 days, and you can force a rotation on demand. Symmetric encryption keys with KMS-generated material rotate automatically; asymmetric keys and HMAC keys do not.

aws kms enable-key-rotation --key-id alias/billing-exports --rotation-period-in-days 180
aws kms get-key-rotation-status --key-id alias/billing-exports

{
    "KeyRotationEnabled": true,
    "KeyId": "arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7081-92a3-b4c5d6e7f809",
    "RotationPeriodInDays": 180,
    "NextRotationDate": "2027-01-23T10:14:31.622000+00:00"
}

Grants are the day pass at the front desk rather than a name added to the building's permanent access list. A grant is a separate little record attached to the key, letting one principal run specific KMS operations, optionally fenced by encryption context, without anybody editing the key policy. AWS services create them constantly on your behalf: attach an encrypted EBS volume to an instance and EC2 (Elastic Compute Cloud, the virtual server service) creates a grant so the hypervisor can unwrap that volume's key. Grants are eventually consistent, so create-grant hands back a GrantToken you can pass into the very next call rather than sitting in a retry loop. An admin kills one with revoke-grant; the holder hands one back with retire-grant. The half that matters for you: a grant is an access path that reading the key policy will never reveal.

aws kms list-grants --key-id alias/legacy-app-2019 \
  --query 'Grants[].[GranteePrincipal,join(`,`,Operations),CreationDate]' \
  --output text

arn:aws:sts::111122223333:assumed-role/aws:ec2-infrastructure/i-04b9c2f1a7d38e5c0	Decrypt	2025-11-04T09:12:44+00:00
arn:aws:iam::111122223333:role/batch-export-2024	Decrypt,GenerateDataKey	2024-02-17T03:00:11+00:00
arn:aws:iam::444455556666:role/vendor-migrator	Decrypt,ReEncryptFrom	2024-06-02T14:40:09+00:00

Line one is ordinary and you leave it alone: that is the EC2 infrastructure identity for a running instance, and without it the encrypted volume will not mount. Lines two and three are your finding. The batch-export-2024 role belongs to a nightly job that was decommissioned over a year ago, and nobody ever called retire-grant. The last line hands kms:Decrypt to a role in account 444455556666, a vendor whose migration finished in 2024, and that account appears nowhere in today's key policy, so a reviewer reading only the policy would tell you with total confidence that the vendor has no access. Add GrantId to that query when you are ready to revoke, and run list-grants in the same audit that reads your key policies rather than as a separate task you will get to later.

Proving the Control Works

Reading a policy proves nothing about the system you actually run. Trying the action and watching it fail proves something. The cheapest honest test in KMS is to switch the key off and confirm that the thing you believed depended on it really does stop.

aws kms disable-key --key-id alias/billing-exports
aws s3 cp s3://billing-exports-prod/2026-07/summary.csv - | head -2
aws kms describe-key --key-id alias/billing-exports \
  --query 'KeyMetadata.[KeyState,Enabled]' --output text

download failed: s3://billing-exports-prod/2026-07/summary.csv to - An error occurred
(KMS.DisabledException) when calling the GetObject operation:
arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7081-92a3-b4c5d6e7f809 is disabled.
(Service: Kms, Status Code: 400, Request ID: 8c2f41d9-77b0-4e2a-9a13-6d0c1b7e5f22)
Disabled	False

The exception name is the fastest diagnostic tool in KMS, so learn the five you will meet. AccessDeniedException means a policy said no, and the culprit is a key policy, an IAM policy, an SCP, or a grant that never existed. DisabledException, which S3 passes through as KMS.DisabledException, means the key is switched off, which is an availability incident rather than a permissions one and needs a completely different fix. KMSInvalidStateException means the key is pending deletion or waiting on imported material. IncorrectKeyException means you named a key that did not wrap this particular ciphertext. InvalidCiphertextException means the encryption context does not match or the blob has been altered. Reading the name before you read anything else saves an hour of policy archaeology per incident.

Detection comes from CloudTrail, which records KMS calls as management events and captures them in any trail by default. The encryption context rides along inside requestParameters, and that single field is what separates a useful audit log from a wall of identical Decrypt lines.

aws kms enable-key --key-id alias/billing-exports
 
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time "$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --query 'Events[].CloudTrailEvent' --output json \
  | jq -r '.[] | fromjson
      | [.eventTime, (.userIdentity.arn // "-"), (.errorCode // "ok"),
         (.requestParameters.encryptionContext // {} | tostring)] | @tsv'

2026-07-27T09:58:12Z  arn:aws:sts::111122223333:assumed-role/billing-exporter/i-04b9  ok  {"aws:s3:arn":"arn:aws:s3:::billing-exports-prod/2026-07/summary.csv"}
2026-07-27T10:02:41Z  arn:aws:sts::111122223333:assumed-role/ci-deploy/gh-run-8891   AccessDenied  {"tenant":"acme"}
2026-07-27T10:02:43Z  arn:aws:sts::111122223333:assumed-role/ci-deploy/gh-run-8891   AccessDenied  {}
2026-07-27T10:02:44Z  arn:aws:sts::111122223333:assumed-role/ci-deploy/gh-run-8891   AccessDenied  {}

Line one is the application doing its job, and the context names the exact object it read. The next three lines are a build role trying to decrypt billing data, failing, then immediately retrying with no context at all, which is what a compromised CI (continuous integration) pipeline looks like while it probes for whatever might work. There is your alert, and it is not subtle: group KMS events by principal, fire when errorCode starts with AccessDenied more than a handful of times in a minute, and fire separately on any successful Decrypt from a principal that has never touched that key before. Teams who exclude KMS events from their trail to shave the CloudTrail bill are deleting precisely this signal, and they usually find out during the incident review.

The Trade-Offs Nobody Puts on the Slide

One key for the whole account is cheap and means a single compromised role reads everything you own. A key per sensitive workload runs about a dollar a month each plus roughly three dollars per million cryptographic requests, and it turns the payroll key and the marketing key into separate decisions with separate audit trails and separate blast radius. That is the right call nearly every time. It stops being right somewhere around a key per customer, where every read becomes a billable KMS call, you start bumping into the per-region request rate quota on busy paths, and the default ceiling of 100,000 customer managed keys per region per account stops being a theoretical number. Keep development and production keys apart while you are at it, because shared decrypt across environments is how a staging compromise becomes a production data leak.

S3 Bucket Keys are the escape hatch for request cost, and what they cost you is resolution. On a busy bucket, SSE-KMS means one GenerateDataKey per write and one Decrypt per read, which shows up on the bill and can push you into KMS request throttling. A bucket key derives a short-lived bucket-level key and cuts those calls by up to ninety-nine percent. The price is that the encryption context becomes the bucket ARN instead of the object ARN, so your CloudTrail no longer gives you a line per object. If your detection story was "alert when someone reads an unusual object," you have traded that story for a smaller invoice. Decide which one you actually need on that bucket.

Multi-Region keys are a primary key plus replicas elsewhere, sharing the same backing material and the same key ID, which you can spot because it starts with mrk-. Ciphertext written in Ireland opens in Ohio with no re-encryption step at all, which is a genuine answer for disaster recovery and for cross-region replication. It also means two regions can now decrypt your data, each replica carries its own separately editable key policy and its own grants, and a mistake in either region exposes the data in both. Use them where you truly must read data during a regional failure, not as a default because they sound safer.

The sweep that catches most of this trouble fits in a single loop. Skip the AWS managed keys, since you cannot change them anyway, and print only what is yours to fix.

for k in $(aws kms list-keys --query 'Keys[].KeyId' --output text); do
  mgr=$(aws kms describe-key --key-id "$k" --query 'KeyMetadata.KeyManager' --output text)
  [ "$mgr" = "AWS" ] && continue
  rot=$(aws kms get-key-rotation-status --key-id "$k" \
        --query KeyRotationEnabled --output text 2>/dev/null || echo n/a)
  al=$(aws kms list-aliases --key-id "$k" --query 'Aliases[0].AliasName' --output text)
  gr=$(aws kms list-grants --key-id "$k" --query 'length(Grants)' --output text)
  printf '%-26s rotation=%-6s grants=%-4s %s\n' "$al" "$rot" "$gr" "$k"
done

alias/billing-exports      rotation=true   grants=1    1a2b3c4d-5e6f-7081-92a3-b4c5d6e7f809
alias/legacy-app-2019      rotation=false  grants=3    9f8e7d6c-5b4a-3928-1706-f5e4d3c2b1a0
alias/jwt-signing          rotation=n/a    grants=0    3e4f5061-7283-94a5-b6c7-d8e9f0a1b2c3
None                       rotation=false  grants=0    c7d8e9f0-1a2b-3c4d-5e6f-708192a3b4c5

Three details in four lines. legacy-app-2019 has rotation switched off and three grants, one of which you already read hands decrypt rights to a vendor in another account. The n/a on jwt-signing is the loop being honest: get-key-rotation-status throws UnsupportedOperationException on an asymmetric key, because automatic rotation does not apply there, and a script without that 2>/dev/null would have died in the middle of your audit. The last row is the one that should hold your attention. No alias, no grants, rotation off, and nothing in this output can tell you whether it protects a forgotten test bucket or the only readable copy of a seven-year compliance archive. Do not delete it on that evidence. Disable it, wait a full cycle, and let CloudTrail tell you whether anything in your estate screams. Next up is S3 misconfiguration, where a bucket can carry a flawless key policy and still serve its objects to the entire internet.

Try this

Run aws kms create-alias --alias-name alias/billing-exports --target-key-id "$KEY" on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the Two Shapes of a Global Decrypt Button. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related