Build-time secrets and how images leak them

ARG, ENV, history, cache exports and provenance, and the mount that avoids them.

A pull-request build log contains this line: #6 [build 3/4] RUN printf '//npm.example.test/:_authToken=%s\n' "npm_FAKE_lab_token_0000" > .npmrc. The Dockerfile never contained the token. It was passed with --build-arg, used only in a builder stage, and the final image is clean. The log is one of six places the value ended up anyway. You'll find all of them with a fake token, then run the build where the same credential ends up in none.

Use the main lab VM. The lesson files go in ~/lab/multistage: three Dockerfiles, a find-token.sh helper and a BuildKit config for the cache section. How stages, --target and RUN --mount work is the subject of "BuildKit builds: stages, cache and mounts" in Docker in depth. Here you'll take that syntax as known and ask a different question. Once a value enters a build, where does it go? The token npm_FAKE_lab_token_0000 isn't a credential for anything. Use a value like it whenever you test for leaks, never a real one.

A build argument in a builder stage

Dockerfile.arg is the pattern you'll find in many repositories. The builder stage takes the registry token as an ARG and writes an .npmrc. It fetches dependencies (a stand-in step here, so the lab needs no registry) and produces an artifact. The final stage copies only that artifact.

# Unsafe on purpose: shows where a token passed with --build-arg ends up.
FROM alpine:3.24 AS build
ARG NPM_TOKEN
WORKDIR /src
RUN printf '//npm.example.test/:_authToken=%s\n' "$NPM_TOKEN" > .npmrc
RUN test -s .npmrc && echo "dependencies fetched" && mkdir -p /out && echo "app v1" > /out/app.txt
 
FROM alpine:3.24
COPY --from=build /out/app.txt /app/app.txt
CMD ["cat", "/app/app.txt"]

#!/bin/sh
# find-token.sh IMAGE STRING: where does STRING appear in a local image?
img=$1 tok=$2
tmp=$(mktemp -d)
docker save "$img" | tar -x -C "$tmp"
echo "history lines: $(docker history --no-trunc --format '{{.CreatedBy}}' "$img" | grep -c -- "$tok")"
echo "config env:    $(docker image inspect --format '{{json .Config.Env}}' "$img" | grep -c -- "$tok")"
layers=0 other=0
for b in "$tmp"/blobs/sha256/*; do
  if gzip -t "$b" 2>/dev/null; then
    gzip -dc "$b" | grep -aq -- "$tok" && layers=$((layers + 1))
  else
    grep -aq -- "$tok" "$b" && other=$((other + 1))
  fi
done
echo "layer blobs:   $layers"
echo "json blobs:    $other"
rm -rf "$tmp"

find-token.sh checks the four places a local image can carry a string. Those are the history (docker history --no-trunc) and the environment in the image config. Then come the layer blobs and the other JSON blobs (config, manifests, attestations) of a docker save export. Build the image and check it:

Step #6 prints the RUN command with $NPM_TOKEN already replaced by its value. Every CI system that stores build logs now stores the token, readable by anyone who can open the job. The build also ended with a SecretsUsedInArgOrEnv warning, cut here. The ENV section below shows that check in full.

The final image itself is clean: zero hits in history, config and blobs. Multi-stage did its job for the image you ship. Everything that follows is about what else your build produced.

The builder stage is one --target away

Teams build the builder stage on its own all the time, to debug a failing build or to run tests in CI. Build it the way they'd do it:

The history has three hits. BuildKit records the ARG instruction with its value, and every RUN after it as RUN |1 NPM_TOKEN=... /bin/sh -c .... The |1 means one build argument was in scope, followed by its value. That's how BuildKit keeps the cache key correct. It's also why an ARG is visible to anyone who can read the image. The layer blob hit is the .npmrc file itself. The extract loop pulls it out of the saved image with nothing but tar. A file written into a layer stays in that layer (the reason is in "Layers and the build cache" in Docker for beginners). A later rm in another step wouldn't change this result.

Build records keep the arguments

BuildKit also keeps a record of every build: its status, logs, materials and the request that started it. You read them with docker buildx history:

The first line is the record the lookup found. BuildKit names a record after the context directory and the Dockerfile. So the awk pattern only matches when the files are in a directory called multistage. If you see NOT FOUND there, adjust the pattern to your directory name. The record holds the argument in clear text, together with the full log from the previous section. Anyone with access to the Docker daemon can read the records, and Docker Desktop shows them in its Builds view. BuildKit keeps them until its history limits drop old records. docker buildx history rm deletes them sooner.

ENV goes further

Dockerfile.env copies a token into the environment, a pattern often used to make a build value available at run time:

# Unsafe on purpose: ENV bakes the value into the image config.
FROM alpine:3.24
ARG API_TOKEN
ENV API_TOKEN=$API_TOKEN
RUN echo "configured"
CMD ["sh", "-c", "echo app started"]

Now the value is in the config of the final image (config env: 1) and in its history. It's also in every container started from it, where docker inspect, /proc/1/environ and every child process can read it. "Runtime secrets, done right" covers the run-time side. For the build side, run the checks on their own. --check evaluates the Dockerfile without building and exits non-zero when a rule fires:

SecretsUsedInArgOrEnv is one of BuildKit's build checks. It flags ARG and ENV names that look like credentials (token, password, secret, key and similar). Exit status 1 means you can use --check as a CI gate before the build step. A # check=error=true line at the top of a Dockerfile turns check warnings into build failures for every build of that file. The rule looks at names, never at values, so a credential in an innocently named argument passes it. It's a net for the obvious cases.

Provenance attestations with mode=max

On Docker 29 every BuildKit build attaches a provenance attestation to the image. "Image anatomy: index, manifest, config and layers" in Docker in depth shows where it sits in the index. The attestation travels with the image to every registry. Its detail depends on the mode: min, the default, or max, which adds the full build request. Push the same build both ways to a local registry:3:

The default attestation has no trace of the token. The max attestation has four. Two are the build arguments of the request (the request and its root). The other two are the environment of the two RUN operations in the build definition. This is the final image, the clean one from the first section, and anyone who can pull it can read its provenance. The last check shows the same attestation inside a local docker save export, so image tarballs carry it too.

Docker's docs state it directly: mode=max exposes the values of build arguments, and secret mounts are never included. Two settings make this common in practice. Supply-chain guidance often recommends mode=max because it records more about the build. And the docker/build-push-action GitHub Action adds mode=max provenance by default when the repository is public. A pipeline that passes a token with --build-arg in a public repository publishes that token with every image. "Pinning, SBOMs, provenance and scanning" covers what provenance is for. The fix is never to turn it down, but to keep secrets out of build arguments.

Exported build cache

CI runners share build cache by exporting it, to a directory or to a registry reference. The default docker driver can export cache as long as the containerd image store is in use. That's the default on fresh Docker 29 installs, this VM included. You'll use a docker-container builder anyway, because that's what most CI setups run, and it also works on hosts still on the legacy store. "Buildx builders, outputs, cache and Bake" in Docker in depth covers drivers and cache export. The builder reads buildkitd.toml, which pulls Docker Hub images through mirror.gcr.io and talks plain HTTP to the lab registry:

mode=min exports only the layers of the final result: four blobs, none with the token. mode=max exports the intermediate layers of every stage, so the builder's .npmrc layer is in it. The registry export is the same blob under a cache reference. The loop above read it with plain HTTP requests, as anyone with pull access to that repository can. mode=max is what makes exported cache useful for multi-stage builds. So treat a file written in a builder stage as readable by everyone who can pull the cache.

Secret and ssh mounts

Dockerfile.secret does the same work with the two mounts BuildKit provides for credentials. The npm configuration is mounted from a file for one RUN. The ssh mount forwards the client's ssh agent socket, so a git clone over ssh can authenticate while the key stays in the agent. The syntax and options (env=, required, mode, --ssh default) are in "BuildKit builds: stages, cache and mounts" in Docker in depth.

FROM alpine:3.24 AS build
WORKDIR /src
RUN --mount=type=ssh \
    test -S "$SSH_AUTH_SOCK" && echo "ssh agent socket at $SSH_AUTH_SOCK"
RUN --mount=type=secret,id=npmrc,target=/src/.npmrc,required=true \
    test -s .npmrc && echo "dependencies fetched" && mkdir -p /out && echo "app v1" > /out/app.txt
 
FROM alpine:3.24
COPY --from=build /out/app.txt /app/app.txt
CMD ["cat", "/app/app.txt"]

Create the inputs: the npm configuration with the same fake token, and a throwaway ssh key. Then build the builder stage, the worst case from before, with mode=max provenance and a push. Export a mode=max cache from the same Dockerfile too:

The build log shows the RUN lines with their mount flags and no value. The ssh step saw only a socket path inside the build container. Now search every place that leaked before, for the token and for a line of the private key file:

Zero everywhere: history, image config, layers, the attestation, the exported cache, the build record and its log. The record: line confirms that the last two counts come from the record of this build, not from an empty lookup. The second search string is the third line of the private key file, which holds key material. The max provenance records that a secret called npmrc and an ssh agent called default were used. That's useful for auditing, and it's all you get. The secret file is mounted on a tmpfs for the duration of the RUN. It isn't part of the layer that step produces, so no cache mode can export it.

One limit remains. A mount keeps the value out of the image, but the command that runs inside the step can still put it somewhere. RUN --mount=type=secret,id=npmrc,target=/src/.npmrc cp .npmrc /tmp/ writes it into the layer. A step that echoes the value prints it into the build log. Mounts protect what BuildKit records, not what your commands do.

Where each value ends up

Every cell is a result from this lab, and a dash is a case the lab didn't test. An ENV value is worse than all of these, since it sits in the config of the final image. If a real token has already gone through one of these channels, rotate it first. Deleting a tag, a cache reference or a build record afterwards limits further spread. It doesn't undo copies that runners, mirrors and pull caches already hold.

Clean up

Remove the builder (which deletes its cache and records), the registry, the images and this lesson's build records from the default builder:

Try this

On the main lab VM, copy Dockerfile.arg to a new file and rename NPM_TOKEN to something bland, such as BUILD_REF, everywhere in it. Run docker build --check on the original and then on the copy. Predict both exit codes before you look: the rule reads names, never values, so the fake token is just as exposed in the copy. Compare with the SecretsUsedInArgOrEnv warning that arg-build printed.

Takeaway

Pass build credentials only through --mount=type=secret or --mount=type=ssh. The fake token you sent with --build-arg showed up in the build log and in 3 history lines of the builder stage. It was also in the build record, and 4 times in the mode=max attestation, while the mount build counted zero everywhere. The trap in real pipelines is that the shipped image scans clean. So nobody checks the CI log, the exported cache or the provenance that went out with it.

Related