The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (2 files, 1 KB): imglayers.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/docker-hard/imglayers.tar.gz && tar -xzf imglayers.tar.gz, which creates ~/lab/imglayers/. SHA-256: af8905f73c3ac1b67c05eb55a4db15b463cb331a00263e541a423b41b4419bcd
Docker in depth assumes you know Docker fundamentals. You can run and inspect containers, write a Dockerfile, use volumes, networks and Compose, and push to a registry. Every command runs on the main lab VM, secopslog-docker. You build it from the lab kit as described in "Installing Docker Engine and running your first container" in Docker for beginners (./setup/create-lab.sh, then multipass shell secopslog-docker). A few lessons change the daemon or the host, so they use the second VM, secopslog-docker-sec. They say so in their first lines. They also give the commands to create that VM (./setup/create-lab.sh --profile sec) and open a shell in it (multipass shell secopslog-docker-sec). If you want a clean start, reset the main VM between lessons with /opt/secopslog-lab/setup/reset-lab.sh.
Run docker pull without -q and the output ends with a Digest: line. That digest isn't the hash of a file inside the image. It isn't the image ID of the old Docker either. It names a small JSON document, and everything else about the image hangs off that document by more digests. You'll walk that chain once, from the tag to the bytes. First you'll do it on alpine:3.24 from Docker Hub, then on an image you build and push to a registry of your own. The lesson files (a Dockerfile and a two-line script) go in ~/lab/imglayers.
A tag points at an index
docker buildx imagetools inspect reads an image straight from its registry without pulling it. With --raw it prints the exact JSON the registry serves. Save it to a file so the next commands can work on the same copy:
mediaType says this is an OCI image index. That's a list of manifests, each with a digest, a size and a platform. The first entry is the image for linux/amd64. The second has platform unknown/unknown and the annotation vnd.docker.reference.type: attestation-manifest. You don't run it. It's a record about the image named in vnd.docker.reference.digest, here the amd64 manifest. Here's the whole index, one line per entry:
There are eight platforms, each followed by its attestation manifest. When docker pull runs on this arm64 VM, it reads the index, picks the linux/arm64/v8 entry and downloads only that. The local store keeps the index and remembers the platforms it skipped:
Only linux/arm64/v8 has content. The other rows are known but cost nothing. On an amd64 machine the linux/amd64 row carries the data instead. The second command shows what Docker 29 means by an image ID. With the containerd image store (the default on fresh 29 installs), the ID is the digest of the index. It's the same digest the registry serves for the tag. .Descriptor says so plainly: media type index, that digest, and its size in bytes. Older hosts on the legacy graph-driver store use the digest of the config instead. "Inspecting, exporting and cleaning up images" shows both.
The manifest: one config, a list of layers
Now follow the entry for this machine's architecture. The command asks the daemon for its CPU architecture, so the same lines work on arm64 and amd64:
Besides schemaVersion, mediaType and optional annotations, an image manifest points at two kinds of content. config points at one JSON blob. It holds the runtime defaults (command, environment, user, working directory), the build history and the list of layer IDs. layers lists the filesystem layers in order, bottom first. Each one's a gzip-compressed tar archive with its digest and compressed size. alpine has a single layer of about 4 MB. Every arrow in this chain is a digest. The tag resolves to the index digest, the index lists manifest digests, and the manifest lists config and layer digests. Change one byte of a layer and its digest changes. Then the manifest changes, and so does the index. That's why a digest at the top pins everything below it.
From a tag to the bytes
Registry name
alpine:3.24
a tag: a movable pointer to one index digest
Index (application/vnd.oci.image.index.v1+json)
linux/amd64 manifest
one entry per platform
linux/arm64/v8 manifest
the entry this VM pulls
attestation manifests
unknown/unknown, one per platform, provenance about it
Image manifest
config blob
Cmd, Env, User, history, rootfs.diff_ids
layer blobs
tar+gzip, bottom first, addressed by compressed digest
On this host (containerd store)
content store
the compressed blobs as pulled
snapshots
the unpacked layers containers start from
Every link is a digest of the thing it points to, so the index digest pins the manifests, the config and every layer.
Build an image and read it back
Next comes an image whose contents you know. The Dockerfile mixes instructions that change files with instructions that only change metadata:
#0 building with "default" instance using docker driver
...
#9 exporting layers 0.1s done
#9 exporting manifest sha256:d514f512b6ca643d82e5bac882ea630ea2f9ea0d3da4fa4e6e1ddd34e413a5fb done
#9 exporting config sha256:420b18dc0147323c145916543de527a55a3e8161f53427a0a66039554968a24d 0.0s done
#9 exporting attestation manifest sha256:0ea4fcb2f258cba8518a3bddec0dde965fd96404a59f80a42eb4cdb0759c7d6d done
#9 exporting manifest list sha256:b1e0e1e5a3c1314d1f9bdf467d75f5c4d8e27e98cc096624a927dc6879a2762e done
#9 naming to docker.io/library/lab-anatomy:1 done
#9 unpacking to docker.io/library/lab-anatomy:1 0.0s done
#9 DONE 0.2s
The steps before the export are cut here (they ran on this first build and read CACHED on a rebuild). The export lines at the end are the chain from the previous section, produced locally. You'll see the image manifest, its config, an attestation manifest, and the manifest list (another name for an index) that ties them together. The last digest is the image ID. Even a single-platform build produces an index on Docker 29, because BuildKit attaches a minimal provenance attestation by default. The image runs as expected:
IMAGE CREATED CREATED BY SIZE COMMENT
b1e0e1e5a3c1 1 second ago CMD ["/app/greet.sh"] 0B buildkit.dockerfile.v0
<missing> 1 second ago USER app 0B buildkit.dockerfile.v0
<missing> 1 second ago LABEL org.opencontainers.image.title=lab-ana… 0B buildkit.dockerfile.v0
<missing> 1 second ago ENV GREETING=hello 0B buildkit.dockerfile.v0
<missing> 1 second ago COPY --chmod=0755 greet.sh /app/greet.sh # b… 12.3kB buildkit.dockerfile.v0
<missing> 1 second ago RUN /bin/sh -c adduser -D -H -u 10001 app # … 32.8kB buildkit.dockerfile.v0
<missing> 1 second ago WORKDIR /app 8.19kB buildkit.dockerfile.v0
<missing> 4 days ago CMD ["/bin/sh"] 0B buildkit.dockerfile.v0
<missing> 4 days ago ADD alpine-minirootfs-3.24.3-aarch64.tar.gz … 9.31MB buildkit.dockerfile.v0
History has nine rows: two from alpine's own build and seven from this Dockerfile. The image has four layers. First comes alpine's, then WORKDIR (it had to create /app). Then RUN adduser (it changed /etc/passwd, /etc/group and /etc/shadow) and COPY. ENV, LABEL, USER and CMD show 0B because they changed only the config. They add history entries and no layer at all, not even an empty one. <missing> in the IMAGE column only means that no separate image exists for that step. That's normal for anything built with BuildKit or pulled. Which of these steps reruns on the next build depends on the cache rules in "Layers and the build cache" in Docker for beginners.
Note what .RootFS.Layers lists. These aren't the layer digests from a manifest. They're diff IDs, the digests of the uncompressed tar streams. The next section shows you the difference with a real push.
What a registry stores
Start a registry:3 container on a lab network, published on the VM's loopback address only, and push the image to it:
Each line is one blob. You'll count the four layers, the provenance document and the two-byte empty config of the attestation manifest. The image config and the manifests go up after them without a progress line. The digest the push prints is the index digest, the same value as the local image ID. Now read back what the registry holds: the index, then the image manifest it points to:
The first layer's digest is identical to the single layer of alpine's arm64 manifest above. Same bytes, same digest. So a registry stores it once, and a host that already has alpine never downloads it again. That sharing is the whole reason layers exist as separate blobs. The config, read through --format '{{json .Image}}', shows the runtime defaults. It also shows the history, with empty_layer marking the metadata-only steps:
config, manifest says: sha256:420b18dc0147323c145916543de527a55a3e8161f53427a0a66039554968a24d
config, blob hashes to: sha256:420b18dc0147323c145916543de527a55a3e8161f53427a0a66039554968a24d
top layer, manifest says: sha256:cd5046d1a7400f1a649fc5262b395de6a1f4a80824a5ede654fab57814eb77ac
top layer, blob hashes to: sha256:cd5046d1a7400f1a649fc5262b395de6a1f4a80824a5ede654fab57814eb77ac
top layer, gunzipped: sha256:16bcd31679544e7f14a16d3cb0f62113a92f0dbb0bef7f143b104c67e3e621a9
The config blob hashes to its digest. The layer blob hashes to the digest in the manifest. The same layer, decompressed, hashes to the last entry of diff_ids. Registries and manifests deal in compressed blobs. The config and the unpacked snapshots on a host deal in uncompressed content. Recompress the same layer with different settings and it gets a new blob digest in the manifest, but it keeps its diff ID. That's why you can't compare a manifest's layer digests with docker image inspect output and expect a match.
The attestation manifest
The second entry of your pushed index is the provenance BuildKit attached:
It's an ordinary OCI manifest with an artifactType that marks it as an attestation. It has an empty config and one layer of media type application/vnd.in-toto+json. That layer's annotation names the predicate: SLSA provenance v1, a description of how the image was built. subject points back at the image manifest it describes. Tools that don't understand attestations skip the unknown/unknown entry, which is why docker pull and docker run ignore it. Two things follow for you in practice. Your own single-platform images are indexes once pushed, so the digest to record and deploy is the index digest. And when you copy an image between repositories, the attestations should come along. Promoting the index digest does that ("Tags, digests and promotion"). "Pinning, SBOMs, provenance and scanning" in Advanced container security covers what the provenance contains, the SBOM attestation, and how to verify both.
Some topics belong to other lessons. A running container adds a writable layer on top of these read-only ones ("Images, containers and the core commands" in Docker for beginners). Where the content store and snapshots live on disk is the subject of "Where Docker keeps data". Deleting a file in a later layer hides it, but its bytes stay in the earlier layer. "Slimming images and layer hygiene" in Advanced container security shows how that leaks data. Remove what you created:
Before you run the cleanup, add one more ENV line and one more RUN line that writes a file to the Dockerfile in ~/lab/imglayers, and rebuild. Predict how many rows docker history will show and how many entries .RootFS.Layers will list, then check both against the nine rows and four layers above. Only the RUN line should cost you a layer.
Takeaway
Record and deploy the index digest that docker push prints, since it pins every manifest, config and layer below it. Where this goes wrong is comparing digests across the wrong level: the layer digest in a manifest is a compressed blob, while .RootFS.Layers holds uncompressed diff IDs, so the two never match. A script that checks one against the other will report every image as tampered with.
Quick check
01docker image inspect myapp:1 --format '{{len .RootFS.Layers}}' prints 4 for an image whose Dockerfile is FROM alpine:3.24, WORKDIR /srv, ENV MODE=prod, RUN apk add --no-cache curl, COPY app /srv/app, USER 10001, CMD ["/srv/app"]. Which instructions produced the four layers?
Incorrect — ENV only changes the config. It adds a history entry with empty_layer, not a layer.
Correct — WORKDIR created /srv, RUN and COPY changed files; ENV, USER and CMD changed only the config.
Incorrect — USER and CMD are metadata, and the base image layer always counts.
Incorrect — That would be six. Metadata-only instructions add no layer at all.
02docker buildx imagetools inspect --raw of an image shows two manifests: linux/amd64 and one with platform unknown/unknown and the annotation vnd.docker.reference.type: attestation-manifest. What is the second one?
Incorrect — The platform is unknown because it is not runnable at all; the annotation says what it is.
Incorrect — Clients skip attestation manifests when choosing a platform; pulls work normally.
Incorrect — The config is a blob referenced from the amd64 manifest, not an index entry.
Correct — Its in-toto layer describes how the manifest named in vnd.docker.reference.digest was built.
03You compare the layer digests in an image manifest from the registry with .RootFS.Layers from docker image inspect on the same image. None match. Why?
Correct — The lab hashed one layer both ways: the gzip blob matched the manifest, the decompressed stream matched diff_ids.
Incorrect — Images are immutable; a change would give a different image, and the ID would differ too.
Incorrect — diff IDs come from the config blob, which is identical everywhere the image is pulled.
Incorrect — A registry stores blobs as pushed; the lab showed the blob hashing to its manifest digest.