Detecting secrets before merge

gitleaks, pre-commit, and CI gates that stick.

A smoke detector does not put out fires. It makes noise early enough that you still can. Secret scanning plays that role for credentials: a program reads your files and your commit history looking for strings that look like keys, and it tells you before one of them reaches somewhere permanent. The failure mode is the same too. A detector wired to nothing, beeping in an empty basement, is decoration. Most secret scanning programmes fall apart at the wiring, not at the detecting.

Three kinds of tool cover the ground. gitleaks is one small binary that matches your files and your git history against more than 150 built-in rules. TruffleHog matches too, then goes a step further and asks the provider whether the credential still works; it ships detectors for over 800 kinds of secret. And the hosting platform scans on its own: GitHub secret scanning watches for hundreds of token formats contributed by partner companies, and GitLab's pipeline secret detection runs gitleaks for you inside the job. None of them catch every custom token your company invented last year. All of them catch the boring disasters that still dominate incident reports.

What the scanner is actually matching

There are two ways to recognise a credential inside a wall of text, and they behave very differently. The first is shape. Providers deliberately stamp a prefix on their tokens so that scanners can spot them: ghp_ for a GitHub personal access token (a password-equivalent string a developer generates so scripts can act as them), AKIA for an AWS (Amazon Web Services) access key ID, xoxb- for a Slack bot token. Matching those is a regex (regular expression, a compact pattern language for text), and it costs almost nothing to run.

The second is entropy, which sounds exotic and is not. A bag of assorted sweets is more mixed than a bag of all lemon ones, and Shannon entropy is the number that measures exactly that: how evenly the characters of a string are spread, counted in bits per character. hunter2 leans on seven ordinary letters and scores 2.81. A 40-character machine-generated token spreads across many different characters and scores far higher, with a ceiling of 5.32 for a string that length, which is the base-2 logarithm of 40 and is what you get when all forty characters differ. gitleaks uses the score as a filter: a rule fires only when the candidate looks random enough to have come from a machine rather than a keyboard.

gitleaks combines both, and its catch-all rule is stricter than most people assume. The GitHub token rule is the regex ghp_[0-9a-zA-Z]{36} with an entropy floor of 3. The generic rule, generic-api-key, does not flag every high-entropy string it sees. It needs three things at once: a keyword from a fixed list (access, api, auth, key, credential, creds, passwd, password, secret, token) sitting next to the value, a value of at least ten characters, and an entropy of at least 3.5. Here is what that means on a file with three suspicious-looking lines.

RELEASE_BOT=ghp_EXAMPLEexampleEXAMPLEexampleEXAMPLE1
db.password=hunter2
build.id=7f3c9a12-4b8e-4f21-9d0c-2a6b5e8f1c47

# 'dir' scans files on disk, with no git history involved.
# (older spelling, deprecated in v8.19.0 but still working:
#  gitleaks detect --source /tmp/leak-demo --no-git)
gitleaks dir /tmp/leak-demo --no-banner --verbose
echo "exit code: $?"

Finding:     ghp_EXAMPLEexampleEXAMPLEexampleEXAMPLE1
Secret:      ghp_EXAMPLEexampleEXAMPLEexampleEXAMPLE1
RuleID:      github-pat
Entropy:     3.820950
File:        /tmp/leak-demo/app.env
Line:        1
Fingerprint: /tmp/leak-demo/app.env:github-pat:1
 
4:04PM INF scanned ~119 bytes (119 B) in 4.02ms
4:04PM WRN leaks found: 1
exit code: 1

One finding out of three lines, and both misses teach you something. db.password=hunter2 is a real database password and gitleaks said nothing about it. The keyword is right there, but the generic rule's regex refuses to capture a value shorter than ten characters, so hunter2 never even reaches the entropy check, and at 2.81 bits it would have failed that too against a 3.5 threshold. The build ID is the opposite trap. A UUID (universally unique identifier, the random-looking 7f3c9a12-... string) is scrambled enough to clear any entropy bar, and gitleaks passed over it only because none of those keywords sits beside it. Loosen the keyword requirement and every UUID, git commit hash, and certificate blob in your repository becomes a finding. That trade is the entire tuning problem, visible in three lines.

Where you put the gate decides whether it works

A scanner that produces a report is a scanner nobody reads by the third week. A sign asking people not to enter is a report. A turnstile is a gate: a place where the answer "secret found" stops the work from moving forward. You have four candidate places to put one. Two of them run on a developer's own laptop or not at all, and two run on servers you control, inside CI (continuous integration, the service that runs automated checks on every proposed change). They are nowhere near equal in strength.

The ordering matters because the cost of a miss grows as you move right. Catching a key in a staged diff costs a developer thirty seconds. Catching it after merge costs you a rotation, a history rewrite, and an awkward conversation with whoever owns the credential. Catching it after somebody else's bot found it costs an incident. So the leftmost layer should be fast and friendly, and the rightmost layers should be impossible to skip.

Layer one: the hook on your laptop

A git hook is a script that git runs at a fixed moment, in this case immediately before a commit is recorded. The usual way to install one is pre-commit, a small hook manager that reads a YAML config file (YAML being a plain-text format for settings, indentation instead of brackets), downloads the tools that file names, and wires them into your repository.

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1              # pin a real released tag, then bump it deliberately
    hooks:
      - id: gitleaks

pre-commit install
git add app.env
git commit -m "add deploy config"

pre-commit installed at .git/hooks/pre-commit
 
Detect hardcoded secrets.................................................Failed
- hook id: gitleaks
- exit code: 1
 
Finding:     REDACTED
Secret:      REDACTED
RuleID:      github-pat
Entropy:     3.820950
File:        app.env
Line:        1
Fingerprint: app.env:github-pat:1
 
4:07PM WRN leaks found: 1

Behind that four-line config, the hook runs gitleaks git --pre-commit --redact --staged --verbose. Two flags carry the weight. --staged means it reads the diff you are about to commit rather than every file on disk, which is why it finishes in milliseconds even on a large repository. --redact replaces the secret with the word REDACTED in the output, and that matters more than it looks.

Now the honest part. That hook lives in .git/hooks, a directory git never versions and never syncs. It exists only in the clones where somebody installed it, and any developer can walk straight around it with one flag.

git commit -m "add deploy config" --no-verify
git log --oneline -1

[main 4a1c2f9] add deploy config
 1 file changed, 3 insertions(+)
4a1c2f9 add deploy config
# the hook never ran, and nothing anywhere records that it was skipped

There is no server-side enforcement in that mechanism at all. pre-commit even ships its own escape hatch, SKIP=gitleaks git commit -m "...", for the times a check is genuinely wrong. Treat the hook as what it is: a fast, friendly save from someone's paste buffer, not a control you can point at during an audit. You can raise the floor by shipping hooks through a template repository or by setting core.hooksPath to a managed directory in a shared git config. Both help. Neither survives a developer who has decided to push.

Layers two and three: the parts nobody can skip

The next gate sits on the platform, on the push itself. GitHub calls it push protection: the server matches incoming commits against its own patterns and rejects the push outright, so the secret never lands in a branch anyone can fetch. Since March 2024 it has been on by default for new public repositories owned by personal accounts. Everything else, including new public repositories owned by organisations and every repository that already existed, has to be switched on deliberately, and a repository admin can switch it back off afterwards. Go and check yours rather than assuming.

Push protection is the strongest control in this lesson, because it is the only one that runs before the data is stored anywhere. A developer can still override it, but the override is a deliberate click with a stated reason, and that reason lands in an audit log a security team can read. Compare that with --no-verify, which leaves no trace whatsoever.

The third gate is the CI job, and here the gate is an exit code. gitleaks returns 0 when it finds nothing, 1 when it finds a leak or hits an error, and 126 when you hand it a flag it does not recognise. A non-zero exit fails the job, and a failed required check greys out the merge button.

name: secret-scan
on:
  pull_request:
  push:
  schedule:
    - cron: "0 4 * * *"        # nightly full-history sweep
 
jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 0        # WITHOUT THIS you scan exactly one commit
      - uses: gitleaks/gitleaks-action@v3
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # license key required for org-owned repos, free for personal accounts
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}

One line in that workflow does the heavy lifting, and it is the line people delete when they are tidying up: fetch-depth: 0. By default actions/checkout performs a shallow clone with a depth of 1, meaning the runner holds exactly one commit. gitleaks will happily scan it, pass, and print a green check that proves almost nothing. You can reproduce the blind spot in two commands.

# what the CI runner gets by default
git clone --depth 1 https://github.com/acme/payments.git shallow
gitleaks git shallow --no-banner --verbose
 
# what it gets with fetch-depth: 0
git clone https://github.com/acme/payments.git full
gitleaks git full --no-banner --redact --verbose

4:11PM INF 1 commits scanned.
4:11PM INF scan completed in 24.6ms
4:11PM INF no leaks found
 
Finding:     REDACTED
Secret:      REDACTED
RuleID:      aws-access-token
Entropy:     3.784184
File:        terraform/prod/main.tf
Line:        42
Commit:      9c1f0a3d2b7e4f5a6c8d9e0b1a2c3d4e5f60718a
Author:      Priya Raman
Email:       priya@corp.example
Date:        2025-11-14T09:22:07Z
Fingerprint: 9c1f0a3d2b7e4f5a6c8d9e0b1a2c3d4e5f60718a:terraform/prod/main.tf:aws-access-token:42
 
4:12PM INF 1284 commits scanned.
4:12PM INF scan completed in 6.44s
4:12PM WRN leaks found: 1

Same repository, same rules, two different answers, because the first run only ever saw one commit out of 1284. If you spot 1 commits scanned in a CI log, your gate is theatre. Fix the checkout depth, then make the job a required status check in branch protection, because a failing job that nobody marked as mandatory is still a report with extra steps. Full history on a large monorepo (one giant repository holding many projects) is slow, so a sensible split is this: scan the pull request diff on every change, and run the full sweep nightly and on release tags.

Baselines and allowlists you can defend

Switching this on for an existing repository produces the same first experience for everybody: dozens of findings, most of them test fixtures, and a powerful urge to add --exit-code 0 and walk away. There are two honest instruments to reach for instead, and they do different jobs.

A baseline is a snapshot of what you already know about, the way a home survey records every crack before you move in so you can tell later which ones are new. Record today's findings once, then scan against that file so only new findings can fail the build.

# 1. record what is already in the repo, without failing anything yet
gitleaks git . --report-path .gitleaks-baseline.json --exit-code 0 --no-banner
 
# 2. from here on, only NEW findings fail
gitleaks git . --baseline-path .gitleaks-baseline.json --redact --no-banner
echo "exit code: $?"

4:20PM INF 1284 commits scanned.
4:20PM INF scan completed in 6.42s
4:20PM WRN leaks found: 37
 
4:21PM INF 1284 commits scanned.
4:21PM INF scan completed in 6.51s
4:21PM INF no leaks found
exit code: 0

The identity that makes this work is the Fingerprint line: commit, file path, rule ID, and start line joined together, stable from one scan to the next. A baseline is a promise to deal with those 37 findings later, never a decision that they are fine. Every entry is an unrotated credential until somebody proves otherwise, so keep the file in git where its size shows up in every diff, and put a date on when it has to reach zero.

An allowlist is a different claim: this specific pattern or path is permanently acceptable. Test fixtures are the legitimate case. A test that exercises your token parser needs a token-shaped string, and that string should be an obvious fake. While you are in the config file, which uses TOML (another plain-text settings format, key equals value under square-bracket sections), this is also where you teach gitleaks about the token formats your own company issues, which no built-in rule knows.

[extend]
useDefault = true            # keep every built-in rule, add ours on top
 
[[allowlists]]
description = "test fixtures: a fake token only counts as fake if it says so"
condition   = "AND"          # default is OR, which would exempt the whole path
paths       = ['''(^|/)testdata/''', '''_test\.go$''']
stopwords   = ["example", "dummy", "notreal"]
 
[[rules]]
id          = "acme-internal-token"
description = "ACME internal service token"
regex       = '''acme_(?:prod|stg)_[a-f0-9]{32}'''
keywords    = ["acme_prod_", "acme_stg_"]
entropy     = 3.0

useDefault = true keeps the built-in rules rather than replacing them, which is the mistake that quietly disables scanning for most teams who write their first config. condition = "AND" is the line almost everyone leaves out, and leaving it out inverts the meaning: an allowlist with several criteria defaults to OR, so the path patterns alone would excuse every finding under testdata/ whether or not the value was ever marked fake. stopwords are substrings that void a finding, and they are checked against the extracted secret rather than the whole line, which is exactly why writing EXAMPLE inside every fixture value is worth the ugliness. For a single finding you want to silence in place, put a gitleaks:allow comment on that line, where at least a reviewer will see it.

Verified beats clever

A regex can tell you a string is shaped like a key. It cannot tell you whether the key still turns in the lock. TruffleHog walks up and tries it: for each candidate it calls the provider's API (application programming interface, the machine-facing front door of a service) and attempts to authenticate. A GitHub-shaped token gets a request to GitHub. An AWS-shaped key gets an AWS call.

# verification: authenticate each candidate against the real provider
trufflehog git file://. --results=verified --fail --no-update
echo "exit code: $?"

Found verified result
Detector Type: AWS
Decoder Type: PLAIN
Raw result: AKIA3PMEXAMPLE7NQZW2
Line: 42
Commit: 9c1f0a3d2b7e4f5a6c8d9e0b1a2c3d4e5f60718a
File: terraform/prod/main.tf
Email: priya@corp.example
Repository: file://.
Timestamp: 2025-11-14 09:22:07 +0000
exit code: 183

--results=verified prints nothing unless the credential actually worked (older builds spell the same idea --only-verified), and --fail turns any result into exit code 183 so the pipeline stops. A verified hit ends every argument you were about to have, because nobody debates whether a key that authenticated ten seconds ago is real. That Decoder Type: PLAIN line is worth noticing too: TruffleHog also runs base64 and UTF-16 decoders over content, so it finds keys that a plain prefix rule slides straight past. Two costs ride along. Verification sends candidate secrets from your build machine to third-party APIs, which some organisations will not accept on principle. And a credential your own company issues has no verifier anywhere, so internal tokens fall back to shape and entropy like everything else. "Not verified" means "not checkable", never "not a secret".

Running two scanners is not duplicated work. One scanner missing a live key is common. Two independent scanners missing the same live key is rare. When they disagree, believe the hit until somebody proves it dead. Here is a policy that survives contact with real developers: fail the build on prefix rules, where false positives are unusual, and let entropy-only rules warn while you tune them.

What none of this catches

Scanners read text and match patterns, so anything that hides the pattern hides the secret. A key stored base64-encoded in a config file is invisible to a prefix rule, because ghp_ no longer appears anywhere in the text. Be precise about what that encoding is, because the confusion behind it causes real incidents: base64 is an alphabet swap, not encryption. There is no key, no password, and nothing to break, and base64 -d reverses it in one command. That is exactly what the data: block of a Kubernetes Secret contains. The values there are base64 so that binary content can ride inside a text file, and anyone who can read the Secret can read the credential in full. A token split across two string concatenations in code is two harmless fragments. An internal credential shaped like svc-payments-prod-a91f has no prefix any tool recognises and not enough randomness to trip an entropy threshold. Secrets inside compiled binaries, container image layers, notebook outputs, and lockfiles sail past a source scan. And a scanner watching your main repository sees nothing at all when the leak happens in a fork, a submodule (a second repository checked out inside the first), or somebody's personal account.

The bigger limit is what a finding actually is: information, not remediation. A key found in commit 9c1f0a3 from eight months ago has been sitting in every clone, every fork, and every CI cache since the day it was pushed. Rewriting history does not recall those copies. So the order never changes. Revoke and rotate the credential first, confirm the old one now fails, then purge the history, then read the access logs covering the exposure window. Teams that swap the first two steps spend an afternoon on a rewrite while the key is still live.

Measure the programme with three numbers. How many findings the gates blocked this week, which tells you the gates are wired to something. Median time from a confirmed true positive to a completed rotation, which is the only number an attacker cares about. And the combined size of your allowlist and baseline, which should trend downward. If the third number only ever grows, the programme is decaying quietly while the dashboard stays green. One habit is worth more than any of them: hand every new engineer a fake key during onboarding and let them watch the CI gate fail safely. People who have seen the control work respect it. People who have only been warned about it learn --no-verify.

Try this

Prove to yourself that deleting the file does not remove the secret. This needs a throwaway repository and about a minute.

mkdir -p /tmp/leak-demo-git && cd /tmp/leak-demo-git && git init -q
printf 'RELEASE_BOT=ghp_EXAMPLEexampleEXAMPLEexampleEXAMPLE1\n' > app.env
git add app.env && git commit -q -m "add deploy config"
rm app.env && git commit -q -am "remove credentials"
ls app.env                       # gone from the working tree
gitleaks git . --no-banner --redact --verbose

ls: cannot access 'app.env': No such file or directory
 
Finding:     REDACTED
Secret:      REDACTED
RuleID:      github-pat
Entropy:     3.820950
File:        app.env
Line:        1
Commit:      3b7a1c05f2e94d8a6b0c1d2e3f4a5b6c7d8e9f01
Author:      Sam Ortiz
Email:       sam@example.com
Date:        2026-07-27T10:12:44Z
Fingerprint: 3b7a1c05f2e94d8a6b0c1d2e3f4a5b6c7d8e9f01:app.env:github-pat:1
 
2:12PM INF 2 commits scanned.
2:12PM INF scan completed in 8.11ms
2:12PM WRN leaks found: 1

Two commits scanned, one finding, and the file no longer exists in your working tree. That is the whole argument for gating before the commit instead of cleaning up afterwards. Scanning is hygiene, though, and not a place to store credentials. The next lesson takes the other road: SOPS and sealed-secrets, two tools that encrypt the values themselves so a secret can live in git on purpose, unreadable to the scanner and to whoever clones the repository next.

Takeaway

The trap worth remembering here: your scanner prints the secret it found. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related