A Dockerfile copies `id_rsa` in one instruction and runs `rm -f /root/.ssh/id_rsa` in a later instruction. What has the rm actually done to the image?

Added a zero-byte whiteout marker that hides the file at runtime, while the original bytes stay in the earlier layer. The `.wh.id_rsa` entry hides it from anything running in the container, and gzip plus tar recover the original from the layer below.

You build with today's default builder and pass `--build-arg TOKEN=s3cr3t`. The Dockerfile declares `ARG TOKEN` and a RUN step that uses it. What does `docker history --no-trunc` show?

The RUN entry prefixed with `|1 TOKEN=s3cr3t`, recording the value in the image config. One build argument was in scope for that step, and the builder wrote down what it was set to.

A teammate runs `docker inspect api:2.1 --format '{{json .Config.Env}}'` on an image that has been in your registry for six months and gets back `["PATH=...","STRIPE_KEY=sk_live_4c2b..."]`. What is the correct first move?

Rotate the Stripe key immediately, then rebuild the image to read the credential at runtime. The key is a bearer credential that an unknown number of parties may already hold, so revoking it is the only step that actually stops it being used.

Courses Secrets management foundationsImages, build args, and layers

Images, build args, and layers

Secrets baked into layers you can still pull.

Beginner25 min · lesson 6 of 13

A container image arrives as a single file, so it feels like a single thing. Inside, it is a stack of transparent sheets laid one on top of another. Each build step draws on its own sheet: files added here, a file modified there, a file crossed out. Running the container means looking down through the whole stack at once and seeing the combined picture. No sheet below the top one is ever rubbed out. It gets covered over by the sheets above it.

Covering is where people get hurt. Removing a secret from an image works the way painting over a wall works. The room looks fine. The old colour is still sitting under the new one, and anyone holding a scraper has it back in under a minute. In this lesson you play the part of the person with the scraper, on your own images, so you can see exactly what ships when you think you cleaned up.

Who holds the scraper? Anyone who can pull the image from your registry (the server that stores images and hands out copies to whoever asks). That group is wider than most teams picture: every developer with read access, every CI runner (continuous integration, the automated service that builds and tests your code), every node in your cluster that keeps a local cache of images, and the entire internet if the repository is public. Pulling an image needs no exploit and no vulnerability. It produces one ordinary, boring log line that nobody reviews.

Why Deleting a File Does Not Remove It

Here is the pattern that shows up in real Dockerfiles. A build needs an SSH key (secure shell, the protocol Git uses to log in to a server over an encrypted connection) so it can clone a private repository. Someone copies the key in, uses it, and tidies up afterwards. The tidying feels responsible. It is the part that does nothing. Build this yourself and follow along, with a throwaway file standing in for a real key.

id_rsa

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAAFAKEDEMOKEY12345
-----END OPENSSH PRIVATE KEY-----

Dockerfile

FROM alpine:3.20.3
COPY id_rsa /root/.ssh/id_rsa
RUN rm -f /root/.ssh/id_rsa

Notice that the COPY and the rm are separate instructions, which means they land on separate sheets. Build it, then look inside a running container to check that the cleanup worked.

terminal

docker build -t layerdemo .
docker run --rm layerdemo ls -A /root/.ssh

output

(no output: /root/.ssh is empty)

The key is gone, as far as anything running in the container can tell. Now stop looking down through the stack and take the stack apart instead. The docker save command writes the image out as a plain tar archive (tape archive, the old Unix format that glues many files into one), and each filesystem layer inside it is a tar of its own, usually squashed down with gzip (the standard Unix compressor).

terminal

docker save layerdemo -o layerdemo.tar
mkdir unpacked && tar -xf layerdemo.tar -C unpacked
 
# walk every file in the archive: try it as gzip, fall back to plain tar
find unpacked -type f | while read -r blob; do
  hit=$( { gzip -dc "$blob" 2>/dev/null || cat "$blob"; } | tar -tvf - 2>/dev/null | grep id_rsa )
  [ -n "$hit" ] && printf '%s\n%s\n' "$blob" "$hit"
done

output

unpacked/blobs/sha256/a4169435584c1d2f8b0e77aa3c19d64b28f0e51a9c7d3b6e0f2a84c5d1e97b30
---------- 0/0               0 2026-07-27 16:40 root/.ssh/.wh.id_rsa
unpacked/blobs/sha256/f6ccc60b75f2ae913c47d0b82e5a1f946d80c3b715ae92f04c6b8d2107f3a95e
-rw-r--r-- 0/0             111 2026-07-27 16:40 root/.ssh/id_rsa

Two layers, two very different entries. The one in layer a416 is a whiteout marker: the .wh. prefix on the name, zero bytes long, every permission bit switched off. It is a sticky note that reads "pretend the file underneath is not here." That note is the entire mechanism behind your rm. One layer down, in f6cc, the real 111-byte key sits exactly where it was written, untouched. Pull it out with the same two tools.

terminal

layer=unpacked/blobs/sha256/f6ccc60b75f2ae913c47d0b82e5a1f946d80c3b715ae92f04c6b8d2107f3a95e
gzip -dc "$layer" | tar -xOf - root/.ssh/id_rsa

output

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAAFAKEDEMOKEY12345
-----END OPENSSH PRIVATE KEY-----

No specialist tooling was involved. Gzip and tar have shipped with every Unix system for decades, and the whole recovery took three commands. That is the honest cost of a secret that touched a layer: whoever pulls your image owns that credential, and no amount of cleanup further down the Dockerfile changes it.

Build Args Leak Without the File Ever Landing

An ARG (build argument, a value you hand to the builder on the command line) feels safer, because it never has to become a file at all. The value goes in, gets used, the build ends. Reading a password aloud in a meeting works the same way: the slip of paper you read from goes in the shredder, and the sentence still ends up in the typed minutes. The minutes here are the image history, the running record the builder keeps of how each step was run. This next build is deliberately tidier than the last one. It writes and deletes the file inside a single instruction, so nothing survives on a sheet of its own.

Dockerfile

FROM alpine:3.20.3
ARG TOKEN
RUN echo "$TOKEN" > /tmp/t && rm /tmp/t

The filesystem really is clean this time. Unpack the layers with the loop from the first section and you will find nothing, because the file was created and removed inside one instruction. Build it and read what the builder wrote down.

terminal

docker build --build-arg TOKEN=supersecret123 -t leakdemo .

output

 => [2/2] RUN echo "$TOKEN" > /tmp/t && rm /tmp/t              0.3s
 => exporting to image                                         0.1s
 
1 warning found (use docker --debug to expand):
 - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TOKEN") (line 2)

terminal

docker history leakdemo --no-trunc --format '{{.CreatedBy}}'

output

RUN |1 TOKEN=supersecret123 /bin/sh -c echo "$TOKEN" > /tmp/t && rm /tmp/t # buildkit
ARG TOKEN
CMD ["/bin/sh"]
ADD alpine-minirootfs-3.20.3-x86_64.tar.gz / # buildkit

Look at the RUN line. The |1 prefix means one build argument was in scope for that step, and the builder helpfully recorded what it was set to. Your token is now a permanent part of the image, written in plain text, in a place that has nothing to do with the filesystem you were so careful about. The ARG TOKEN entry above it records the name of the argument. Write a default into the Dockerfile instead, as in ARG TOKEN=fallback, and that default gets recorded as well, which is why a Dockerfile with a real value sitting in a default line is its own quiet disaster.

Plenty of older guidance claims this only happens with the legacy builder and that BuildKit (the engine that actually runs your builds today) fixed it. The output above is BuildKit, which has been the default since Docker Engine 23.0 in early 2023. It did not fix it. What BuildKit adds is that warning during the build, SecretsUsedInArgOrEnv, which is a genuinely useful signal as long as somebody reads the build log instead of scrolling past it on the way to the green tick.

This history does not live in any layer. It lives in the image config, a small JSON document (JavaScript Object Notation, a plain text data format) that sits alongside the layers. Registries serve that config as its own object, separate from the layer blobs, which means someone probing your registry fetches about a kilobyte of readable text and harvests every build argument you have ever passed. They never download the image.

ENV Is the Leak That Keeps Talking

An ARG at least stops mattering when the build ends. An ENV (environment variable, a named value the operating system hands to a program when it starts) instruction is different in kind, because it is a runtime setting written into the image config on purpose. Every container started from that image gets the variable handed to it, forever, along with every process running inside.

Dockerfile

FROM alpine:3.20.3
ENV API_KEY=sk_live_9f3ba21c7d

terminal

docker build -t envdemo .
docker inspect envdemo --format '{{json .Config.Env}}'

output

["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","API_KEY=sk_live_9f3ba21c7d"]

No container was started to get that. docker inspect reads the config, the same small JSON object a registry will hand to anyone holding pull access. From there the key spreads under its own steam: crash reporters attach the environment to error reports, debug endpoints print it back to whoever asks, and any process that can read /proc/self/environ inside the container picks it up. A dependency you never audited runs with the same environment your application does, and it does not need permission to look.

Where a secret hides inside an image

Filesystem layers

Copied files

still readable after a later rm

Whiteout markers .wh.*

hide the file, never delete it

Compressed tar blobs

open with gzip and tar, no special tools

Image config (a small JSON file)

history / created_by

records build args used by RUN

Config.Env

ENV reaches every container, forever

Served separately

harvested without pulling layers

Build context

.env, .pem, .git

COPY . . sweeps them all in

.dockerignore

the only thing that stops it

Running a container shows you only the top sheet. Anyone who can pull reads the whole stack plus the config.

Keeping the Secret Out in the First Place

BuildKit secret mounts fix the build argument problem properly. A mount is closer to holding a document up against a window for someone to read than to handing over a photocopy they keep. The file appears at /run/secrets/<id> for the length of one RUN instruction, then the window shuts, and nothing about it reaches a layer or the config. The required=true flag makes the build fail loudly if the secret was never supplied, which beats a build that quietly succeeds with an empty string.

Dockerfile

# syntax=docker/dockerfile:1.7
FROM alpine:3.20.3
RUN --mount=type=secret,id=tok,required=true test -s /run/secrets/tok && echo "secret readable at build time"

terminal

export TOKEN=supersecret123
docker build --secret id=tok,env=TOKEN -t safedemo .
docker history safedemo --no-trunc --format '{{.CreatedBy}}' | head -2

output

RUN /bin/sh -c test -s /run/secrets/tok && echo "secret readable at build time" # buildkit
CMD ["/bin/sh"]

The RUN line is recorded, the value is not, and there is no |1 prefix because no build argument was in scope. Sourcing from env=TOKEN rather than src=secretfile also keeps the secret off your disk, which matters on a shared build machine where the file would otherwise outlive the build and wait around for the next person who runs ls. Now check the claim against the artifact you would actually push, and compare it with the leaky image from earlier.

terminal

docker save safedemo -o safe.tar
docker save leakdemo -o leaky.tar
printf 'safe:  '; grep -ac supersecret123 safe.tar
printf 'leaky: '; grep -ac supersecret123 leaky.tar

output

safe:  0
leaky: 1

A clean grep is not proof of a clean image

Two traps sit inside the popular docker save ... | strings | grep check, and both hand you a false all-clear. First, strings ships as part of binutils and is missing from plenty of machines, including a stock Git Bash on Windows. When it is absent the pipeline writes its complaint to the error stream, prints nothing to the screen, and looks exactly like a pass. Use grep -a, which treats binary input as text and exists everywhere. Second, grepping the saved tar reliably finds secrets only in the image config, because that part is plain JSON. Layers are normally stored compressed, both inside the archive that recent Docker versions write and in every registry, so a secret sitting in a layer can score zero matches while the file is fully intact. That is exactly the case with the deleted-key image from the first section. Decompress every layer before you trust a negative result.

Multi-stage builds handle the other half of the problem. Do the messy work in a first stage, then start a fresh final stage and carry across only the finished artifact. The first stage's history and its arguments stay behind.

Dockerfile

FROM alpine:3.20.3 AS build
ARG NPM_TOKEN
RUN echo "$NPM_TOKEN" > /root/.npmrc && echo compiled > /out.txt
 
FROM alpine:3.20.3
COPY --from=build /out.txt /app/out.txt

terminal

docker build --build-arg NPM_TOKEN=npm_9xQvTz7Jw2 -t multidemo .
docker history multidemo --no-trunc --format '{{.CreatedBy}}'

output

COPY /out.txt /app/out.txt # buildkit
CMD ["/bin/sh"]
ADD alpine-minirootfs-3.20.3-x86_64.tar.gz / # buildkit

No trace of NPM_TOKEN (the credential that lets a build download private packages from the Node package registry), and searching the saved final image for the token value finds nothing either. Here is the honest limit of that result: it holds because exactly one file was copied across. Change that line to copy a directory, a home folder, or the whole filesystem root, and the .npmrc written in the builder stage travels with it, and you are back where you started. The protection comes from the narrowness of what you copy, not from the word FROM appearing twice. Exporting build cache to a registry can carry builder-stage history along too, so treat cache destinations as another place secrets surface.

COPY . . takes everything you forgot about

The build context is every file in the directory you point docker build at. A line as ordinary as COPY . . sweeps in .env, the whole .git folder with its full commit history, a stray id_rsa, a kubeconfig, and the .aws directory if somebody happened to leave one lying there. A .dockerignore file is the only thing standing in the way, and it deserves the same review attention as the Dockerfile itself. Start it with .git, .env*, *.pem, *.key and **/node_modules, then check what actually landed with docker run --rm your-image find / -name '.env*' -o -name 'id_rsa'. A flawless Dockerfile cannot rescue you from a dirty context.

Check What You Have Already Shipped

Everything above applies to the images sitting in your registry right now, plenty of them built by people who have since left. Start with the cheapest sweep, which reads the config of every image on a host and flags environment variables whose names look like credentials. It runs in seconds and needs no scanner installed.

terminal

for img in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep -v '<none>'); do
  docker inspect "$img" --format '{{range .Config.Env}}{{println .}}{{end}}' 2>/dev/null |
    grep -iE '^(.*_)?(KEY|TOKEN|SECRET|PASSWORD)=' | sed "s|^|$img  |"
done

output

envdemo:latest  API_KEY=sk_live_9f3ba21c7d

Then run docker history --no-trunc across your most-deployed tags and read the output for |N prefixes carrying long random-looking strings. For deleted-file leaks you need the layer walk from the first section, since neither history nor a plain grep will show them. Automate the boring parts in CI: Trivy scans image layers for secrets alongside its vulnerability work, TruffleHog can point straight at an image with trufflehog docker --image your-image and will try a live credential to see whether it still works, and gitleaks covers the git history that feeds the build. All of them match known credential shapes and high-entropy strings, so an internal token format nobody taught them about walks straight past. A quiet scan means the scanner recognised nothing, which is a much weaker statement than "there is nothing here."

When you do find one, fix it in the right order. Rotate the credential first: issue a replacement, then revoke the old value. A leaked secret is a bearer credential, which means whoever holds those forty characters is indistinguishable from you to the system accepting them. Only after the old value is dead do you rebuild the image. Deleting the tag from the registry is not remediation, because copies are already cached on cluster nodes, in CI runners, in local Docker daemons and possibly in somebody else's registry. The image you cannot recall stops mattering the moment the credential inside it stops working. Kubernetes Secrets, coming up next, look like they solve this at the platform level, and they protect rather less than the name suggests.

Try this

Run docker build -t layerdemo . on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a clean grep is not proof of a clean image. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

CI logs, artifacts, and ticketsKubernetes secrets, honestly

Why production Vault existsVault from dev to production The secrets threat model & secret-zeroAdvanced secrets management

Images, build args, and layers

Why Deleting a File Does Not Remove It

Build Args Leak Without the File Ever Landing

ENV Is the Leak That Keeps Talking

Keeping the Secret Out in the First Place

Check What You Have Already Shipped

Try this

Takeaway

Related