You confirm an active AWS access key is sitting in a public commit. What do you do first?

Set the key to Inactive in IAM, then start scoping how it was used. Revoking at the issuer is the only step that makes every stolen copy useless at once, and Inactive preserves the creation date, last-used data, and owning user that you need for scoping.

You rotate the password inside the Kubernetes Secret payments-db. The payments-worker pod reads it through envFrom. What is actually true?

Nothing changes until the container restarts, and the old value stays readable by anyone with get access on the Secret, since the data field is base64 encoded rather than encrypted. Environment variables are copied once at container start and frozen until a restart, so you must roll the deployment. And base64 is an alphabet swap that one command reverses, so read access on the Secret means possession of the plaintext.

You deactivated the leaked key twenty minutes ago. CloudTrail now shows CreateUser and CreateAccessKey calls made with that key at 04:11, before you revoked it. What is the next move?

Find and remove the IAM user the attacker created, along with its access keys, login profile, and attached policies, then re-check the log for anything those created in turn. The attacker established persistence. Containment is not complete until every identity and permission the key brought into existence is gone, and the new identity's own activity has been swept.

Courses Secrets management foundationsWhen a secret leaks

When a secret leaks

Revoke first, then clean history and notify.

Beginner25 min · lesson 13 of 13

A leaked secret behaves like a copied house key, not a broken window. Nothing is smashed, nothing looks damaged, and whoever holds the copy can walk in at any hour looking exactly like you. So your first move is the one you would make in the physical world: change the lock. You do not begin by working out who photographed your keyring, and you certainly do not begin by rewriting git history.

Speed matters here because the other side is a machine. Bots watch the public commit feed on GitHub and read new pushes within minutes. Researchers who deliberately plant real-looking AWS (Amazon Web Services) keys in public repositories have recorded the first unauthorized attempt to use them inside that same window, sometimes under a minute. Every minute the credential still works is a minute somebody can spend inside your account. The order runs contain, then assess, then eradicate, then recover, then learn. Contain sits first because it is the only step that stops the bleeding. Treat what follows as a runbook: print it, keep it where the on-call engineer can reach it, and rehearse it while nothing is burning.

The First Fifteen Minutes

Containment means killing the credential at the system that issued it, never at the place you found it. It is the difference between phoning your bank about a stolen card and shredding the copy you found on the pavement. Deleting the file from GitHub changes nothing. Closing the Slack thread changes nothing. The only authority that can make those forty characters stop working is the issuer: your cloud provider's IAM (Identity and Access Management, the service that decides who is allowed to do what), your IdP (identity provider, the login system behind single sign-on), the database itself, or the admin console of the SaaS (software as a service) vendor that minted the token.

Start by proving what you are actually holding. An AWS access key ID that begins with AKIA is a long-lived key attached to a user, and it keeps working until somebody turns it off. One that begins with ASIA is a temporary session credential that expires on its own, usually within hours, which changes your urgency but not your response. AWS will tell you which account owns a key even when you have no permissions in that account, which is handy when a stranger emails you a screenshot.

terminal

# Whose key is this? Answered with your own credentials,
# no access to the target account required.
aws sts get-access-key-info --access-key-id AKIAIOSFODNN7EXAMPLE

output

{
    "Account": "123456789012"
}

That is your own account number, so the key is yours to kill. Look it up, then turn it off. IAM is eventually consistent, so the change propagates globally in seconds rather than the same instant everywhere, but the attacker's next call is the one that starts failing.

terminal

aws iam list-access-keys --user-name svc-deploy

output

{
    "AccessKeyMetadata": [
        {
            "UserName": "svc-deploy",
            "AccessKeyId": "AKIAIOSFODNN7EXAMPLE",
            "Status": "Active",
            "CreateDate": "2026-02-11T09:14:22+00:00"
        }
    ]
}

terminal

aws iam update-access-key \
  --user-name svc-deploy \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive
 
aws iam list-access-keys --user-name svc-deploy \
  --query 'AccessKeyMetadata[].[AccessKeyId,Status]' --output text

output

AKIAIOSFODNN7EXAMPLE	Inactive

Deactivate before you delete

Deleting the key erases the metadata you are about to need: the creation date, the last-used timestamp, and the user it belonged to. Inactive refuses every request with the same finality and keeps the paper trail intact. Delete it once your scoping is finished, not while it is still your best source of evidence.

A ticket that says "revoked" is a claim. Proving it takes one command. Feed the leaked values into a throwaway shell and ask AWS's STS (Security Token Service, the part of AWS that answers the question "who am I?") to identify you.

terminal

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
aws sts get-caller-identity

output

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity
operation: The security token included in the request is invalid.

That error is the verification step. A control is proven when the credential fails from the attacker's side, not when somebody ticks a box in a spreadsheet. Run the same check for every other kind of secret you revoke: a dead GitHub token should return 401 (unauthorized), a rotated database password should be refused at login, a disabled webhook should stop delivering.

terminal

# GitHub PAT (personal access token): revoke at github.com/settings/tokens,
# then confirm from outside that it no longer authenticates.
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer ghp_REDACTEDEXAMPLETOKENVALUE000000000000" \
  https://api.github.com/user

output

Then shut the doors that would hand the secret straight back out. CI (continuous integration, the system that builds and tests your code automatically) stores variables and re-injects the old value into every future build, so remove them early. Otherwise a routine pipeline run helpfully restores the incident an hour after you closed it.

terminal

gh secret delete AWS_SECRET_ACCESS_KEY --repo acme/payments-api
gh secret list --repo acme/payments-api

output

✓ Deleted Actions secret AWS_SECRET_ACCESS_KEY from acme/payments-api
NAME                 UPDATED
AWS_ACCESS_KEY_ID    about 5 months ago
SENTRY_DSN           about 2 months ago

Kubernetes deserves its own warning, because the word Secret is doing a lot of unearned work. The values inside one are base64 encoded, which is an alphabet swap in the same family as writing a phone number in Morse code. There is no key, nothing is locked, and no attacker is slowed down by a single second. One command turns it back into the original text.

terminal

kubectl get secret payments-db -n payments \
  -o jsonpath='{.data.password}' | base64 -d; echo

output

pr0d-db-pa55w0rd-in-the-clear

So anybody who can run that command in the namespace already holds the real password. On a default cluster the value also sits in etcd (the database that stores all cluster state) as plaintext on disk, unless an administrator has switched on encryption at rest. Several managed platforms turn that on for you, which is worth checking rather than assuming. During an incident this reframes your question from who saw the file to who has read access to that namespace, which is usually a much longer list.

Before you change the value, list every workload that consumes it, so you know what is about to break and what needs restarting. These queries use jq (a command-line filter for JSON data) to read the pod specifications.

terminal

# pods that mount the Secret as a volume
kubectl get pods -A -o json | jq -r '.items[]
  | select(any(.spec.volumes[]?; .secret.secretName == "payments-db"))
  | "\(.metadata.namespace)/\(.metadata.name) volume"'
 
# pods that pull the whole Secret in as environment variables
kubectl get pods -A -o json | jq -r '.items[]
  | select(any(.spec.containers[].envFrom[]?; .secretRef.name == "payments-db"))
  | "\(.metadata.namespace)/\(.metadata.name) envFrom"'

output

payments/payments-api-6c9f7b8d54-2xkqz volume
payments/payments-api-6c9f7b8d54-r7m4p volume
payments/payments-worker-7d5b96c8f9-jn2wt envFrom

Two queries, because one shape hides from the other. A third shape, valueFrom.secretKeyRef, pulls a single key out of the Secret and needs its own search. The distinction has teeth during an incident. A Secret mounted as a volume behaves like a noticeboard: the kubelet refreshes the file inside the running container within roughly a minute of the change. A Secret read through envFrom behaves like a photocopy taken on your first day, copied once when the container starts and frozen in memory until the process restarts. Two exceptions catch people out. A volume mounted with subPath never refreshes, and a Secret marked immutable never changes by design. After you rotate the value, roll the deployments. A new Secret does not reach a running container on its own.

The first hour after a leak

1Revoke at the issuer

IAM, IdP, CI, database

2Prove it fails

call the API with the dead key

3Scope the blast

last-used, CloudTrail, 90-day limit

4Hunt persistence

users, keys, rules the key created

5Purge the copies

git, images, registries, pastes

6Notify and fix

IR lead, timeline, scanner rule

Assume public exposure even for private repos until proven otherwise.

Work Out What The Key Touched

Containment buys you the time to answer the question everyone will ask next: what did they do with it? This is the door-entry log, and you read it before you tidy anything, because tidying destroys evidence. AWS keeps a summary of the most recent use of every access key.

terminal

aws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE

output

{
    "UserName": "svc-deploy",
    "AccessKeyLastUsed": {
        "LastUsedDate": "2026-07-27T04:12:00+00:00",
        "ServiceName": "iam",
        "Region": "us-east-1"
    }
}

Read that carefully, because the region field will mislead you here. IAM and STS are global services, and AWS records their activity in us-east-1 regardless of where on the planet the caller was sitting. An attacker working from a laptop three continents away still produces us-east-1 against an IAM call, so treating that value as a geographic anomaly is a mistake people make in real incidents. The two fields carrying genuine signal are the service name and the timestamp. A deploy key whose entire job is pushing files to S3 (Simple Storage Service, the AWS file store) has no business calling IAM at all, and 04:12 is not when your pipeline runs. That pairing turns a suspicion into a confirmed compromise. Handle the opposite result with the same care: a null LastUsedDate, with ServiceName and Region both reading N/A, means IAM holds no record of use since it began tracking this in April 2015, which is weaker than proof that nobody touched it.

For the actual sequence of events, query CloudTrail, the AWS service that records API (application programming interface) calls, filtering on the leaked key ID.

terminal

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-07-25T00:00:00Z \
  --max-results 3 \
  --query 'Events[].[EventTime,EventName,Username]' --output text

output

2026-07-27T04:12:03+00:00	CreateAccessKey	svc-deploy
2026-07-27T04:11:58+00:00	CreateUser	svc-deploy
2026-07-26T22:03:11+00:00	GetCallerIdentity	svc-deploy

Those top two lines should stop you cold. The intruder did what intruders do and made a second way in. Deactivating the leaked key has no effect whatsoever on an IAM user that the leaked key created, or on the fresh access key it handed that user, or on any policy it attached along the way. Containment is unfinished until you have found and removed everything the credential brought into existence, then re-read the log for anything those new identities created in turn. This is why you read the logs before you rewrite history, and why the runbook says assess after contain rather than skipping straight to cleanup. One more detail on where to look: because IAM is global, these events land in the us-east-1 event history even if every other resource you own lives in Frankfurt.

Three limits decide whether an empty result means anything. The searchable event history covers ninety days, so an older leak can return nothing at all. It returns management events, the calls that change or describe configuration, and not data events such as S3 object reads, Lambda function runs, or DynamoDB item access, so a key used purely to download customer files can look completely idle unless somebody had already configured a trail that records data events. And new events take roughly fifteen minutes to become searchable, which matters enormously in the first hour. Absence of evidence in lookup-events is not evidence of absence.

Preserve what you find before anything gets cleaned. Export the events to a file, note the commit hash and the exact push time, screenshot the Slack message with its timestamp visible, and record the five facts you will be asked for later: when the secret was created, when it was exposed, when somebody else first used it, when you revoked it, and what it could reach.

Only Now, Clean Up The Copies

With the credential dead and the blast radius mapped, cleanup is finally safe to start. Git is a photocopier that keeps every draft. Deleting a file removes it from the current checkout and leaves the value sitting in history forever, readable by anyone with a clone. The -S flag, known as the pickaxe, finds commits where the number of occurrences of a string changed, which is how you locate the moment it went in.

terminal

git log --all --oneline -S 'AKIAIOSFODNN7EXAMPLE'
git show 9f3c1ab:deploy/upload.sh | grep AKIA

output

9f3c1ab Add staging deploy script
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE

To rewrite that out, git filter-repo replaces the string across every commit. It refuses to run on a working repository, a safety feature that most people meet for the first time as an error message.

terminal

printf '%s\n' 'AKIAIOSFODNN7EXAMPLE==>AWS_KEY_REMOVED' > /tmp/leaked.txt
git filter-repo --replace-text /tmp/leaked.txt

output

Aborting: Refusing to destructively overwrite repo history since
this does not look like a fresh clone.
  (expected freshly packed repo)
Please operate on a fresh clone instead.  If you want to proceed
anyway, use --force.

Do what it asks. Clone fresh, rewrite there, then add the remote back by hand, because filter-repo deliberately deletes every configured remote so that nobody force pushes out of muscle memory. Push branches and tags separately, since the rewrite renamed the objects behind both.

terminal

git clone git@github.com:acme/payments-api.git /tmp/purge
cd /tmp/purge
git filter-repo --replace-text /tmp/leaked.txt
git remote add origin git@github.com:acme/payments-api.git
git push --force --all
git push --force --tags

output

Cloning into '/tmp/purge'...
remote: Enumerating objects: 3184, done.
Receiving objects: 100% (3184/3184), 1.84 MiB | 6.10 MiB/s, done.
Parsed 412 commits
New history written in 1.42 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at 4d1e77a Bump chart version
Completely finished after 4.11 seconds.
Enumerating objects: 3184, done.
Writing objects: 100% (3184/3184), done.
 + a71b0c2...4d1e77a main -> main (forced update)

Now the part nobody enjoys. On GitHub the old commit stays reachable by its hash at github.com/acme/payments-api/commit/9f3c1ab after your force push. Worse, every repository in a fork network shares one object store, so a commit that survives in any fork stays reachable from all of them, and your rewrite never touches those. For a repository that was public, open a support ticket asking GitHub to remove the cached view, and tell every person who cloned to re-clone rather than pull, because a pull onto the old history will cheerfully push the deleted objects straight back up.

Rewriting history is cleanup, not containment

Anyone who cloned, forked, or mirrored the repository before your rewrite still holds the original objects, and so does your CI cache and last night's backup snapshot. filter-repo is hygiene. Rotation is the control, because it invalidates every copy everywhere at the same instant, including the ones you will never find.

Container images carry the same permanence with even less visibility. A value passed in with --build-arg and referenced in a RUN line gets baked into the image's own metadata, where anyone who can pull the image can read it back. Environment variables baked in with ENV show up the same way under docker inspect.

terminal

docker history --no-trunc payments-api:1.4.2 | grep -o 'AKIA[A-Z0-9]\{16\}'

output

AKIAIOSFODNN7EXAMPLE

Rebuild without it, push a new tag, and delete the old tag and its digest from the registry. Then accept the limit: every host that already pulled that image still holds the layer in its local cache, and you cannot reach into those. The Slack paste, the Jira comment, and the wiki page work the same way. You can delete the message. You cannot un-index it, un-export it, or erase it from somebody's phone notification history. Rotation is what makes all of those copies worthless at once.

Tell The Right People, Then Remove The Cause

Bring your IR (incident response) lead in at the start, not once the argument about history rewriting is settled. Depending on what the secret protected, notifying customers or a regulator may be a contractual or legal obligation with a clock attached, and that is not a decision to make alone at midnight with a terminal open. Hand over the timeline you have been keeping and let the people whose job it is decide who gets told and when.

If the key was used for cryptocurrency mining or bulk spam, the cloud provider's abuse team becomes genuinely useful. They can act on the sending infrastructure, help with the bill you did not authorize, and share their own view of the timeline. Keep those contact details in the same document as the revoke steps. Somebody paged at three in the morning should be reading a phone number, not inventing a process.

Then remove the reason it happened. An incident that ends with the word "rotated" and nothing else will repeat with a different key next quarter. The cheapest preventive control is a scanner rule that makes this exact shape impossible to commit again.

.gitleaks.toml

title = "acme baseline"
 
[extend]
useDefault = true
 
[[rules]]
id = "acme-aws-access-key"
description = "AWS access key ID committed to the repo"
regex = '''\b(AKIA|ASIA|ABIA|ACCA)[A-Z0-9]{16}\b'''
# Keywords MUST be lowercase. Gitleaks lowercases the text it scans
# before the pre-filter runs, so an uppercase keyword never matches
# and the rule silently finds nothing.
keywords = ["akia", "asia", "abia", "acca"]
 
[[allowlists]]
description = "the documented AWS example key, safe to keep in docs"
regexes = ['''AKIAIOSFODNN7EXAMPLE''']

Wire the same config into the pipeline so a pull request cannot merge past it. A non-zero exit code fails the build, and --redact keeps the secret out of the CI log you are about to publish to everyone with read access. Use the current subcommands: gitleaks git scans history, gitleaks dir scans files on disk, and the old gitleaks detect was deprecated back in version 8.19.

terminal

gitleaks git --redact --log-opts='--all' .
echo "exit code: $?"

output

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks
 
Finding:     export AWS_ACCESS_KEY_ID=REDACTED
Secret:      REDACTED
RuleID:      acme-aws-access-key
Entropy:     3.523561
File:        deploy/upload.sh
Line:        4
Commit:      9f3c1ab2d7e4f8a1b03c95d6e2f7a4b8c1d0e3f6
Author:      Dana Okafor
Email:       dana@acme.io
Date:        2026-02-11T09:14:22Z
Fingerprint: 9f3c1ab2d7e4f8a1b03c95d6e2f7a4b8c1d0e3f6:deploy/upload.sh:acme-aws-access-key:4
 
12:41PM INF 412 commits scanned.
12:41PM INF scanned ~1.84 MB (1.84 MB) in 812ms
12:41PM WRN leaks found: 1
exit code: 1

Honeytokens are the other half of the answer, and they work like the dye pack a bank slips into a bundle of cash. A canary key is a credential that guards nothing and exists only to tell you when somebody tries it. Plant one in a private repository and another in a CI log. If either fires, you have learned something no scanner could tell you: that the repository or the log was read by someone with no business reading it. Treat that alarm as a reason to rotate the real secrets sitting next to it, immediately, before anybody asks you for further evidence.

Close the incident with a tracked action that has an owner and a date. A scanner rule, a pipeline condition that blocks the pattern, a change of ownership for the account, or moving that credential into a secrets manager so the next version is short-lived. A ticket recording only that you rotated the key is an unfinished ticket.

Practice Before You Need It

Run this as a twenty-minute tabletop with the team. Take one scenario, a leaked AWS access key in a public repository, and write the first five actions with a named owner and a time target against each. Fifteen minutes to deactivation is a fair bar for a team that has done it once.

terminal

printf '%s\n' \
  '1. Deactivate the access key           owner: on-call    target: 15 min' \
  '2. Prove it fails via sts              owner: on-call    target: 20 min' \
  '3. CloudTrail lookup, export evidence  owner: security   target: 45 min' \
  '4. Delete CI variables, rotate deps    owner: platform   target: 2 h' \
  '5. Incident channel and timeline       owner: IR lead    target: ongoing'

output

1. Deactivate the access key           owner: on-call    target: 15 min
2. Prove it fails via sts              owner: on-call    target: 20 min
3. CloudTrail lookup, export evidence  owner: security   target: 45 min
4. Delete CI variables, rotate deps    owner: platform   target: 2 h
5. Incident channel and timeline       owner: IR lead    target: ongoing

The exercise nearly always exposes the same two gaps. Nobody is certain who holds permission to revoke in production, and nobody knows where the audit logs actually live. Both are far cheaper to discover on a Tuesday afternoon than during the real thing.

When "rotate everything this key could reach" turns into a week of work, static credentials have outgrown you. The Vault production and advanced secrets courses cover credentials that get generated per use and expire on their own, which shrinks most of this runbook down to revoking a lease and going back to bed.

Try this

Run aws sts get-access-key-info --access-key-id AKIAIOSFODNN7EXAMPLE on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: deactivate before you delete. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Choosing a secrets managerFinish · self-test

Why production Vault existsVault from dev to production The secrets threat model & secret-zeroAdvanced secrets management

When a secret leaks

The First Fifteen Minutes

Work Out What The Key Touched

Only Now, Clean Up The Copies

Tell The Right People, Then Remove The Cause

Practice Before You Need It

Try this

Takeaway

Related