How secrets leak

Logs, ps, crash dumps, child processes, repos.

A secret is any string that opens a door: a database password, an API key (application programming interface key, the string one program shows to prove it is allowed to call another), a cloud access token, a TLS private key (transport layer security, the key that proves your site really is your site). A house key is metal. Hand it over and you no longer have it. A secret is text. Copying it costs nothing, takes no time, and leaves no mark, and every system that touches it can keep its own copy without telling you. Most leaks are not a break-in. Nobody picks a lock. Someone pastes a key somewhere convenient, the copy sticks around, and one day the wrong person reads it.

What counts as a secret

Three words get used loosely, so pin them down. A credential is anything you show to prove who you are, like a badge at a reception desk. A secret is a credential that only works while nobody else knows it: passwords, API keys, signing keys, database connection strings. A token is a secret that a machine handed out, usually a random-looking string such as ghp_... (GitHub) or AKIA... (an AWS access key ID, the visible half of an Amazon Web Services key pair), and that prefix is deliberate so scanners can spot it on sight. Nearly all of these are bearer credentials. The word comes from a cinema ticket or a bearer bond: whoever *bears* (holds and presents) it gets in, and nobody asks a second question. Your production database cannot tell your deploy script apart from a stranger on another continent typing the same 40 characters.

That one property is the whole threat model. A leaked secret needs no exploit, no software bug, no malware. The attacker signs in as you, and every firewall rule and detection rule you built waves the session through, because the activity looks like ordinary work. Only one fix ends it: rotation, meaning you issue a new secret and revoke the old one, so every copy anyone took stops working at the same moment.

Why leaks beat exploits

The numbers here are not a scare story. GitGuardian scans public GitHub every year, and its count for 2024 was roughly 24 million secrets newly committed in that year alone. It also reports that a large share of leaked secrets are still valid long after they were exposed. Finding them takes no human effort at all. Researchers run honeytoken studies, planting deliberately fake AWS keys in public repositories and watching who bites, and the first unauthorized use attempt usually arrives within minutes of git push. GitHub's own secret scanning tells providers fast. Attacker automation is racing the same clock, and it wins outright on any secret format the scanners were never taught.

Real incidents follow the same shape. Uber's 2022 breach got worse when the attacker found PowerShell scripts sitting on an internal file share with admin credentials for the company's privileged-access system typed straight into them. The 2021 Codecov supply-chain attack quietly copied environment variables out of thousands of CI pipelines (CI is continuous integration, the automated system that builds and tests your code on every push), because CI is the one place where every team's secrets end up together. Neither attack needed a single bug in the victim's own software.

Where the copies land

Source code is the biggest leak surface by a wide margin. git commit is close to permanent. Delete the file tomorrow and the value is still in the history, in every clone on every laptop, and in every fork (the next lesson takes that apart properly). The vector people underrate is container images. A Dockerfile ARG (a value passed in at build time) or ENV (an environment variable baked into the image) looks like scaffolding that disappears once the build finishes. It doesn't:

# Bake a token in with a build arg — feels temporary, isn't
docker build -t payments-api:1.4.2 \
  --build-arg NPM_TOKEN=npm_4kX9mQvTz7Jw2LhPbN6cRd8s .
 
# Every layer's metadata ships with the image
docker history payments-api:1.4.2 --no-trunc | grep NPM_TOKEN
# -> ARG NPM_TOKEN                                                        0B
# -> RUN |1 NPM_TOKEN=npm_4kX9mQvTz7Jw2LhPbN6cRd8s /bin/sh -c npm ci # buildkit   211MB

Anyone who can pull that image reads the token with one command. At plenty of companies that means every engineer, every CI runner, and whoever takes over either of them. Files fare no better. COPY .env . writes the file into a layer, and it stays there even when a later instruction deletes it, the way a deleted paragraph still sits in a document's version history. The proper fix is a BuildKit secret mount (RUN --mount=type=secret,id=npm_token ...), BuildKit being the modern Docker build engine. It hands the value to that one build step and never writes it into a layer.

Now look at the machine under your fingers. Your shell keeps a diary, and the operating system publishes a live list of every running command. Both outlive the moment:

# Paste a token into a "one-off" command...
curl -s -H "Authorization: Bearer ghp_wK9xT2mQ8vL4jR7nB3cY5dF1aZ6sE0hU" \
  https://api.github.com/user
 
# ...and it outlives the request
tail -n 1 ~/.bash_history
# curl -s -H "Authorization: Bearer ghp_wK9xT2mQ8vL4jR7nB3cY5dF1aZ6sE0hU" https://api.github.com/user
 
# Long-running processes expose their arguments to every user on the host
ps -eo pid,args | grep [p]g_dump
#  4172 /usr/bin/pg_dump --dbname=postgresql://app:S3cretPw@db:5432/prod

The output of ps (process status, the command that lists what is running) is readable by every user on that host, arguments included. So pass secrets in through environment variables or files, never on the command line. Logs come next: an app that prints its config at startup, a CI job running with set -x (which echoes every command it runs, arguments and all), a crash reporter that ships the whole environment along with the stack trace. Then the human layer. A key pasted into Slack, a Jira ticket or a wiki page gets indexed, which makes it searchable by anyone in the company, and by anyone who phishes one account.

Find them before attackers do

Attackers scan around the clock, so the only stance that holds up is running the same class of scanner yourself, earlier, where a finding is cheap to fix. gitleaks is the open-source tool most teams start with:

# Scan the working tree; -v prints each finding inline
# (v8.19+ syntax — `gitleaks git .` walks full history too)
gitleaks dir . -v --report-path gitleaks-report.json
 
# Finding:     AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# Secret:      AKIAIOSFODNN7EXAMPLE
# RuleID:      aws-access-token
# Entropy:     3.684184
# File:        deploy/staging.env
# Line:        3
# Fingerprint: deploy/staging.env:aws-access-token:3
#
# 4:12PM INF scanned ~842 KB (862341 bytes) in 214ms
# 4:12PM WRN leaks found: 1

gitleaks dir reads what is in the folder right now. gitleaks git walks the repository's entire commit history, which is where the unpleasant surprises live. Its cousin trufflehog adds a step that changes your morning: it *verifies* a finding by calling the provider's API to ask whether that credential still works. Any repository with a few years behind it turns up piles of long-dead keys, and verification tells you which ones can still hurt you. Run a scanner in three places. A pre-commit hook is the cheapest, because the secret never leaves the laptop. CI catches whatever the hook missed, on every push. A scheduled sweep across the whole organization catches the repositories nobody remembers owning.

Shrink the blast radius

Prevention will fail eventually, so design for the leak that gets through. Prefer credentials with a short life over ones that live forever. Cloud OIDC federation (OpenID Connect, a standard that lets one system vouch for an identity to another) lets a CI job trade a signed identity token for cloud credentials that expire in an hour or less, so there is no long-lived key lying around to leak. Scope every secret down to what it actually needs. A leaked read-only token is an incident you work through on Tuesday. A leaked admin token is a company-wide emergency. And keep an inventory, because you cannot rotate a secret nobody remembers creating.

One vector earns a lesson of its own. Git was built never to lose data, which is exactly why it never loses your secrets either. Delete the file, force-push over it, even delete the repository from the server, and the value can still be pulled back out of a clone, a fork or a reflog. Next up: how secrets survive in git history, how to prove whether one ever entered yours, and what real cleanup actually costs.

Here is a Tuesday that really happens. A contractor forks a private repo into their personal account, and for twenty minutes that fork is public. Scanners do not wait for your change window. By the time anyone notices, the AWS key in deploy/staging.env is already in someone else's paste buffer. Your firewall never sees a breach packet. What it sees is a normal AssumeRole call from a region you rarely use. That is why leak hygiene counts as production security work, and not a chore the security team handles alone.

When you triage findings, sort them two ways: alive or already dead, and how much power each one carries. A verified admin cloud key outranks a year-old CI token that expired months ago. Verification of the TruffleHog kind saves weekends. A pattern-only dump of four hundred hits buries the one key that still opens the door. Write a short runbook while nothing is on fire: who rotates IAM (identity and access management, the AWS service that decides who can do what), who regenerates GitHub Apps, who updates the secret store, and who talks to customers if the key could have reached their data.

Speed decides how the story ends. Everything above exists to shorten one window: fewer places a copy can land, a scanner that catches the paste before it leaves the laptop, and credentials short-lived enough that a stolen copy expires before anyone bothers to try it. None of that is polish. It is the difference between an awkward thread in your team channel and a phone call with your customers.

Try this

Do this in a throwaway folder, never on a real project. You plant a fake key, scan for it, then watch a token copy itself into your shell history. The point is to see the leak surfaces with your own eyes, not to chase a real incident.

mkdir -p /tmp/sf-leak && cd /tmp/sf-leak
printf 'AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n' > staging.env
gitleaks dir . -v --no-banner 2>&1 | head -20
# also: paste a fake token into a command and inspect history
echo 'curl -H "Authorization: Bearer ghp_exampleOnlyNotReal000000000000000" https://example.com' >> ~/.bash_history
tail -n 1 ~/.bash_history

Finding:     AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
RuleID:      aws-access-token
File:        staging.env
WRN leaks found: 1
 
curl -H "Authorization: Bearer ghp_exampleOnlyNotReal000000000000000" https://example.com
# shell history kept the bearer token — same class of copy as git and CI logs

Takeaway

Carry one rule out of this lesson: to any system holding the door, whoever has the string is you. So when a secret slips, the first question is never how bad it looks. It is which places kept a copy, and the first action is always rotation.

Next: take one real secret and trace every place it could have landed in the past month, then put a pre-commit scanner on your laptop so the next paste never gets as far as a push.

Related