CI logs, artifacts, and tickets

Pipelines that print, upload, or paste secrets.

A CI runner (continuous integration: the machine that rebuilds and retests your code every time somebody pushes) is a temp worker you hire for four minutes. You hand over the keys to production, the work gets done, the worker vanishes. The paperwork is the problem. Everything that worker touched gets written down, the write-up outlives the worker by ninety days, it gets attached to tickets, and on a public repository anybody with a GitHub account can read it.

CI is the one room where every key in the company ends up together: cloud credentials, container registry passwords, npm (Node package manager) and PyPI (Python Package Index) publishing tokens, code-signing keys, database passwords for migration steps. That concentration is why attackers aim at build systems rather than laptops. The difference between CI and a laptop holding the same keys is that CI publishes. It writes a log, uploads files, posts to Slack, comments on pull requests. Four channels, all switched on by default, all retained.

The Log Filter Is Smarter Than You Expect And Still Beatable

Secret masking sounds like a wall. It works more like a bouncer holding a photograph. When you store a value in GitHub Actions secrets, or tick the masked box on a GitLab variable, the runner writes that value onto a list, then scans every line of output for it and rewrites any match to *** before the line reaches the log. The filter has no concept of what a token is. It recognises a string it was shown.

GitHub's runner is shown more than one photograph. Alongside the raw bytes it registers encoded spellings of the same value: base64 (a way of rewriting arbitrary bytes as ordinary letters and digits so they survive transport), two shifted base64 variants that cover the value sitting at an awkward byte offset inside a longer encoded blob, JSON string escaping, URI escaping, XML escaping, command-line escaping and a pair of PowerShell forms. Eleven encoders in total. GitLab does none of that. It replaces the stored bytes and stops. You can model both in a shell and watch where each one gives up.

#!/usr/bin/env bash
SECRET='ghp_R2eXaMpLe0000000000000000000000abcd'
B64=$(printf %s "$SECRET" | base64 -w0)
 
gitlab() { sed "s/$SECRET/[MASKED]/g"; }                  # stored bytes only
github() { sed -e "s/$SECRET/***/g" -e "s/$B64/***/g"; }  # bytes + base64 spelling
 
try() {
  echo "== $1"
  echo "   gitlab: $(printf %s "$2" | gitlab)"
  echo "   github: $(printf %s "$2" | github)"
}
 
try "raw value"    "$SECRET"
try "base64 once"  "$B64"
try "base64 twice" "$(printf %s "$B64" | base64 -w0)"
try "split in two" "${SECRET:0:20} ${SECRET:20}"

== raw value
   gitlab: [MASKED]
   github: ***
== base64 once
   gitlab: Z2hwX1IyZVhhTXBMZTAwMDAwMDAwMDAwMDAwMDAwMDAwMDBhYmNk
   github: ***
== base64 twice
   gitlab: WjJod1gxSXlaVmhoVFhCTVpUQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNREJoWW1Oaw==
   github: WjJod1gxSXlaVmhoVFhCTVpUQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNREJoWW1Oaw==
== split in two
   gitlab: ghp_R2eXaMpLe0000000 000000000000000abcd
   github: ghp_R2eXaMpLe0000000 000000000000000abcd

Read the middle two blocks carefully, because the popular advice gets this backwards. One round of base64 walks past GitLab and gets caught by GitHub. Two rounds walk past both, because the second pass produces bytes that appear on nobody's list. Splitting the value across a space beats every entry on either platform, since the filter is matching a contiguous run of characters and there is no longer one to find. None of this is a flaw you can configure away. A list of known spellings can only ever cover the transformations somebody thought of.

That gap has been used in anger. In March 2025 somebody took over tj-actions/changed-files, an action used by more than twenty thousand repositories, and repointed its version tags at malicious code, tracked as CVE-2025-30066 (Common Vulnerabilities and Exposures, the public catalogue of known security flaws). The payload pulled a memory-dumping script from a public Gist, scraped credentials out of the runner process, and printed them into the workflow log double base64-encoded. The second round of encoding is the entire trick. One round would have matched the runner's registered base64 spelling and come out as ***. On public repositories those logs were world-readable for roughly fifteen hours, and the harvested material included AWS access keys, GitHub personal access tokens, npm publishing tokens and private RSA keys.

Platforms also refuse to mask values they cannot handle. GitLab requires a masked variable to sit on a single line with no spaces, run to eight characters or longer, and not share a name with an existing variable, and when variable expansion is switched on it accepts only alphanumerics plus a short list of punctuation. A PEM-format private key (Privacy Enhanced Mail, the text block starting -----BEGIN PRIVATE KEY-----) fails the single-line rule outright and cannot be masked at all. The usual workaround is to base64-encode the key so it becomes one line. That satisfies the storage rule and does nothing for the decoded form your script writes to disk thirty seconds later.

Two more things decide how bad a printed secret gets. Who can read the log: on a public repository that is the entire internet, and on a private one it is everyone with repository read access, which is nearly always a much larger group than the people cleared to see the secret itself. And for how long: GitHub keeps run logs and artifacts for ninety days by default, and organizations routinely extend that on private repositories, so a value printed in March is still sitting in a searchable log in June. Set retention to the shortest window your incident response genuinely needs.

Debug Tracing Turns Every Command Into A Log Line

set -x is a court stenographer for your shell. Switch it on and the shell prints each command to standard error before running it, fully expanded. The expansion is what hurts you. Variables get substituted before the line is printed, so the transcript records values where you wrote names.

#!/usr/bin/env bash
TOKEN='ghp_R2eXaMpLe0000000000000000000000abcd'
set -x
curl -s -o /dev/null -H "Authorization: Bearer $TOKEN" https://api.example.com/health

+ curl -s -o /dev/null -H 'Authorization: Bearer ghp_R2eXaMpLe0000000000000000000000abcd' https://api.example.com/health

On a real runner, if TOKEN came from the platform's secret store, that line arrives as Bearer *** and you get away with it. The trouble starts one step later. Pipelines trade credentials constantly: assume an AWS role and get a session token back, log into Vault and get a client token, call an OAuth endpoint and get an access token. None of those values are on the masker's list, so the trace prints them whole.

Reaching for a flag that keeps the secret off the command line helps less than people expect. docker login --password-stdin, curl --netrc-file and gh auth login --with-token all read the credential from standard input or a file, which keeps it out of the process listing that any other process on the runner can read with ps. It does nothing about the trace, because the command that produces the value gets traced too.

REGISTRY_TOKEN='ghp_R2eXaMpLe0000000000000000000000abcd'
set -x
printf %s "$REGISTRY_TOKEN" | docker login ghcr.io -u octo-deploy --password-stdin
set +x

+ printf %s ghp_R2eXaMpLe0000000000000000000000abcd
+ docker login ghcr.io -u octo-deploy --password-stdin
Login Succeeded
+ set +x

The docker login line is clean. The printf line above it hands the token over. Move set +x above the block instead of below it, keep tracing switched off around anything that authenticates, and when you do need a trace to debug a failing deploy, wrap the three noisy commands and leave the credential handling outside the traced region.

Notification integrations copy the same text somewhere else again. A Slack failure message that quotes the failing command carries whatever sat on that command line into a channel with its own membership and its own retention. Matrix builds multiply the count: one careless echo in a job that runs across five operating systems and three language versions publishes the value fifteen times, into fifteen separate log files that all need cleaning.

Artifacts Are Downloads Not Scratch Space

An artifact is a box left on the loading dock. It is not scratch space that evaporates when the job ends. It is a file anyone with read access to the repository can download, and on GitHub it sits there for ninety days by default. The classic mistake is one line of YAML (YAML Ain't Markup Language, the indented format CI configuration is written in). A test goes flaky, somebody wants the whole workspace for debugging, and path: . goes into the upload step. Here is what that box turns out to hold.

unzip -l workspace-dump.zip

Archive:  workspace-dump.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      112  2026-07-27 16:39   .env
       62  2026-07-27 16:39   .npmrc
      131  2026-07-27 16:39   terraform.tfstate
       15  2026-07-27 16:39   dist/app.js
      149  2026-07-27 16:39   reports/junit.xml
---------                     -------
      469                     5 files

unzip -qq workspace-dump.zip -d x
grep -rIn -E 'sk_live_[A-Za-z0-9]+|ghp_[A-Za-z0-9]{20,}|npm_[A-Za-z0-9]{20,}|"password"' x/

x/.env:2:STRIPE_KEY=sk_live_51ExampleExampleExample
x/.npmrc:1://registry.npmjs.org/:_authToken=npm_4kX9mQvTz7Jw2LhPbN6cRd8s
x/reports/junit.xml:2:  <failure message="auth failed: header=Bearer ghp_R2eXaMpLe0000000000000000000000abcd"/>
x/terraform.tfstate:4:    {"type":"aws_db_instance","instances":[{"attributes":{"password":"S3cr3tPassw0rd"}}]}

Four findings out of a 469-byte archive, and a fifth the pattern missed. Line 1 of that .env reads DATABASE_URL=postgres://app:S3cr3tPassw0rd@db.internal:5432/payments, and the regular expression above had no rule for a password buried inside a connection string. That is the honest limit of pattern-based scanning: gitleaks and trufflehog find what their rules describe, which covers the well-known token prefixes and misses your in-house formats until somebody writes rules for them.

Current versions of actions/upload-artifact skip hidden files by default, because include-hidden-files is false, and hidden means any file beginning with a dot or any file inside a folder beginning with a dot. That spares you the .env, the .npmrc and everything under .git. Read the listing again though. terraform.tfstate carries no leading dot, and Terraform writes resource attributes such as database passwords into state in cleartext. Neither is reports/junit.xml hidden, and there the test framework helpfully recorded the failing request's Authorization header. GitLab job artifacts and Jenkins archiveArtifacts apply no hidden-file default at all, so the same workspace upload there ships the dotfiles too.

Shortening retention costs you something real. Drop to three days and the artifact you needed for the flaky test that fires once a month is gone. Zero retention is not the answer either. Use a short window for routine build output, keep a longer one for release artifacts you actually ship, restrict who can download them the way you restrict who can read the secret store, and run a scanner in front of the upload step so nothing sensitive gets stored in the first place.

Fork Pull Requests And Inherited Secrets

A pull request from a fork is code from a stranger. GitHub handles the obvious case for you: a workflow triggered by on: pull_request from a fork receives no repository secrets and a read-only GITHUB_TOKEN. So attackers aim at the trigger that behaves differently.

pull_request_target runs the workflow definition from your base branch, with your secrets, in the context of your repository. It exists so maintainers can label, greet and triage outside contributions safely. It breaks the moment that workflow checks out the pull request head and runs something from it: a build script, a test suite, an npm ci that fires lifecycle hooks. A stranger's code is now executing next to your production keys. GitHub's Security Lab named the pattern a pwn request back in 2021.

# The classic pwn request: base-repo secrets and attacker-controlled code in one job.
# Since 20 July 2026 actions/checkout refuses this and fails the step.
name: e2e
on: pull_request_target          # your secrets, your base branch
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
        with:
          ref: ${{ github.event.pull_request.head.sha }}   # the stranger's code
      - run: npm ci && npm run test:e2e                    # runs their scripts
        env:
          STRIPE_KEY: ${{ secrets.STRIPE_KEY }}            # with your keys

That guardrail is recent and its exact shape matters. actions/checkout version 7, backported to every supported major version except v1, refuses to fetch fork pull request code under pull_request_target, and under workflow_run when the run it followed was a pull request event. It trips when repository: resolves to the fork, when ref: is refs/pull/<number>/head or the matching /merge reference, or when ref: resolves to the fork's head or merge commit. Same-repository pull requests are untouched. Workflows on floating tags such as @v4 picked the change up automatically on 20 July 2026, which is why some pipelines went red that week without anyone editing them.

The escape hatch is an input called allow-unsafe-pr-checkout, named that way on purpose so it stands out in a code review. Adding it to clear a red pipeline switches the guardrail off and hands you the original vulnerability back. The check also only watches actions/checkout. A git fetch origin pull/42/head written by hand, or a gh pr checkout, walks straight past it, so treat the block as a smoke alarm rather than a sprinkler system.

Split the work into two jobs instead. Let an on: pull_request workflow build and test the untrusted code with no secrets at all and upload its results as an artifact, then have a separate workflow_run workflow that holds the secrets read those results as data without executing anything from the fork. The rule underneath is easy to state and easy to break by accident: never run code you did not review in a job that holds credentials.

secrets: inherit is the same mistake wearing a suit. On a reusable workflow call it hands over every secret the calling workflow can see, rather than the two the job actually needs. If that reusable workflow lives in another repository, or a teammate adds a debugging environment dump to it next quarter, your whole set is in scope and no review request lands in your inbox. Name the secrets you pass, one line each.

Swap The Standing Key For A Four Minute One

A long-lived AWS access key sitting in CI variables is a house key taped under the mat. It works at three in the morning, it works from anywhere on earth, and it works for whoever finds it. OIDC (OpenID Connect: a standard way for one system to prove who it is to another without sharing a password) replaces that arrangement. The runner asks GitHub for a short-lived signed identity token describing the repository, the branch and the workflow. AWS checks the signature, compares the description against your rules, and hands back temporary credentials that expire when the job ends.

The security problem does not disappear. It moves. Guarding a key becomes writing one condition correctly.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": "repo:acme/payments-api:ref:refs/heads/main"
      }
    }
  }]
}

Both conditions carry weight. Delete the sub line and every repository on GitHub can assume that role, because every one of them gets a valid token from the same issuer. Loosen it to StringLike with repo:acme/* and any repository in your organization qualifies, including the one an intern created this morning. Pin the branch as shown and a pull request build cannot mint production credentials, whatever somebody puts in the workflow file.

aws sts get-caller-identity

{
    "UserId": "AROA3EXAMPLEID:GitHubActions",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/gha-deploy-payments/GitHubActions"
}

The words assumed-role in that ARN (Amazon Resource Name: the unique identifier AWS gives every resource) are what you verify. No user/ in the identity means no static access key exists in the account for anyone to steal, print or paste. The honest trade-off is that a credential problem became a policy-review problem, and a wildcard in a trust policy is quieter and more dangerous than an access key ID beginning AKIA sitting in a variable, because scanners look for the second one and nobody looks for the first.

Tickets, Chat, And Screenshots

The last channel has no configuration page. A deploy fails, somebody copies the log into a Jira ticket so the on-call engineer has context, and the token in that log now lives in a system with full-text search, its own retention policy, email notifications that carry the body outside the platform, mobile caches, nightly backups, and whatever integrations your company connected two years ago and forgot about. Slack behaves the same way, and so does the pull request comment where a bot pasted the failing output.

Deleting the message does not undo it. By the time you notice, the value has been indexed, mirrored into a search cache and delivered to every device that had the channel open. The right response to "I pasted a key in Slack" is identical to the response to "I pushed a key to GitHub": rotate the credential first, clean up second, and write down when the exposure started so you know which access logs to review.

Prove It On Your Own Pipeline

Two checks, twenty minutes, on a repository you already own. Start by auditing the workflow definitions for the four patterns in this lesson.

grep -rn -E 'set -x|secrets: inherit|pull_request_target|path: \.$' .github/workflows/

.github/workflows/build.yml:10:          path: .
.github/workflows/deploy.yml:8:          set -x
.github/workflows/e2e.yml:2:on: pull_request_target
.github/workflows/e2e.yml:6:    secrets: inherit

Every line there is a question rather than an automatic finding. Does that traced block touch a credential? Does the reusable workflow need all your secrets or two of them? Does the pull_request_target job check out the fork's code, and if actions/checkout is now refusing, has somebody quietly added allow-unsafe-pr-checkout to make the red go away? Does path: . sweep in files nobody outside the team should read? Write the answers down, because they change the next time somebody edits the file.

Then run the canary. Store a fake token as a genuine repository secret, print it four ways in a throwaway job, and read what your platform actually redacted.

name: mask-canary
on: workflow_dispatch          # manual only, never on push
jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - run: |
          echo "plain:  $FAKE"
          echo "b64:    $(printf %s "$FAKE" | base64 -w0)"
          echo "b64x2:  $(printf %s "$FAKE" | base64 -w0 | base64 -w0)"
          echo "split:  ${FAKE:0:20} ${FAKE:20}"
        env:
          FAKE: ${{ secrets.FAKE_CANARY }}   # stored value: ghp_notARealTokenOnlyACanary01

Compare that run against the four lines you produced locally. On GitHub the plain and b64 lines should both come back as ***. If b64x2 and split show the token, you have reproduced the tj-actions bypass on your own pipeline using a value that costs nothing to burn. Whatever survives redaction is exactly the shape a real secret will take on the day a deploy breaks at midnight and somebody adds a debug echo to find out why. Delete the canary secret and the workflow afterwards, then go and look at what your container images remember, because they keep a copy of every build argument you ever passed.

Try this

Run echo "== $1" on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the masker only knows what it was told. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related