Production best practices

A pre-ship checklist for images and the containers that run them.

Run ./preship-check.sh against a container started with a plain docker run, and nine of fifteen checks fail. You get a writable root filesystem, every default capability, no memory, CPU or process limit, no init process, no restart policy, and logs that grow until the disk is full. The image was fine. The run wasn't. What follows is a pre-ship checklist for both halves, applied to a small Node.js service, with a command behind every item. Each item has an owner lesson that explains the mechanism. Here you apply all of them together and check them the same way every time. You'll work on the main lab VM, secopslog-docker, in ~/lab/bestpractices with the lesson files.

The image

The service is the payments API: Node's built-in http module, one small dependency (ms), a /healthz route, and a SIGTERM handler that closes the server and exits 0. The lesson files include package.json, the package-lock.json that npm ci requires, server.js and healthcheck.js. There's also a .dockerignore that keeps .git, .env files and a local node_modules out of the build context. The Dockerfile:

# node:26-alpine, pinned to the index digest the tag pointed at when this lab was written.
# Renovate or Dependabot moves the pin forward through a reviewed pull request.
FROM node:26-alpine@sha256:40cd011b55415bc894e45b02fdd5a5dfde056921b7d23a64c74f80497af48797 AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
 
FROM node:26-alpine@sha256:40cd011b55415bc894e45b02fdd5a5dfde056921b7d23a64c74f80497af48797
# The app never runs a package manager. Remove npm, npx, corepack and yarn so they are
# not shipped, scanned and patched with every release (the base layer still holds the bytes).
RUN rm -rf /usr/local/lib/node_modules /opt/yarn-v* \
      /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack /usr/local/bin/yarn /usr/local/bin/yarnpkg
WORKDIR /app
ENV NODE_ENV=production PORT=3000
COPY --from=deps /app/node_modules ./node_modules
COPY package.json server.js healthcheck.js ./
# UID/GID 1000 is the image's "node" user. A numeric USER lets Kubernetes
# runAsNonRoot verify it without reading /etc/passwd.
USER 1000:1000
EXPOSE 3000
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
  CMD ["node", "healthcheck.js"]
CMD ["node", "server.js"]

Start with the pin. Both stages name node:26-alpine by tag and index digest. Get the digest from the registry; don't copy it from an old README:

The pin matched the tag when this lab ran. When the Node maintainers rebuild 26-alpine, the second line changes. Your Dockerfile keeps building from the pinned bytes until a reviewed change moves the pin. "Pinning, SBOMs, provenance and scanning" (Advanced container security) covers automating that with Renovate or Dependabot. Node 26 is an even-numbered line. In October 2026 it takes the active-LTS role from Node 24, which moves to maintenance, so check the Node release schedule when you pick or move a line. Pin the index digest, not one platform's manifest, so arm64 and amd64 builders each get their own image.

Your deps stage copies only package.json and package-lock.json, then runs npm ci --omit=dev. That installs exactly the lockfile's versions and leaves development dependencies out. Because the source is copied later, editing server.js doesn't invalidate the install. Build once (with a fake token in .env to test the ignore file), change the source, and rebuild. The change here is a comment with the current time appended to server.js, so it's new every time you run it:

Every step down to COPY --from=deps is CACHED, including RUN npm ci; only the last COPY ran again. On a real project with hundreds of packages, that ordering is the difference between a ten-second and a three-minute build. "Layers and the build cache" (Docker for beginners) explains the cache rules.

The runtime stage deletes npm, npx, corepack and yarn. The app never calls them, and the npm bundled in the base image carries its own dependency tree. A Trivy 0.76.1 scan of the node:26-alpine base found four HIGH findings with fixes, all in the bundled npm and none in the app. With npm deleted, the Trivy 0.76.1 scan in "Docker in CI/CD: build to promote" finds nothing outside the app's own packages. Deleting them in a later layer doesn't shrink the image, since the base layer still holds the bytes. But the files are gone from the container's filesystem, so scanners and anyone inside the container no longer find them. A distroless Node base goes further ("Minimal bases: distroless, scratch, static and Alpine" in Advanced container security).

Check three properties on every image you ship. User is 1000:1000, numeric, which is the image's node account. Kubernetes runAsNonRoot can verify a number but not a name. The health check runs node healthcheck.js, a four-line script using Node's built-in fetch, so it needs no curl or wget. That matters as soon as someone switches the base to a Debian slim variant to escape an Alpine musl issue:

A wget health check copied from an Alpine image, where BusyBox provides it, fails on every probe there. The container sits unhealthy while serving traffic perfectly. The listing of /app shows no .env, because the .dockerignore kept the fake token out. And the runtime image has no npm.

The run

An image can't set its own limits or drop its own capabilities. That's the run configuration, here a Compose file:

name: lab-ship
 
services:
  api:
    build: .
    image: lab-ship-api:1.4.2
    ports:
      - "127.0.0.1:3000:3000"
    read_only: true
    tmpfs:
      - /tmp:size=16m,mode=1777
    cap_drop: [ALL]
    security_opt:
      - no-new-privileges:true
    init: true
    mem_limit: 256m
    memswap_limit: 256m
    cpus: 1.0
    pids_limit: 100
    restart: unless-stopped
    logging:
      driver: local
      options:
        max-size: 10m
        max-file: "3"

Here's what each setting does. read_only comes with a small tmpfs for /tmp. cap_drop: [ALL] fits because a Node server on port 3000 needs no capability. no-new-privileges stops any setuid binary from raising privileges. init: true makes Docker's docker-init run as PID 1, forward signals and reap zombies. memswap_limit equals mem_limit, so the container can't use swap beyond its memory limit. There's a CPU quota and a process limit. restart: unless-stopped brings it back after a crash or a host reboot, while a deliberate stop stays stopped. And the local log driver rotates by size. Start it and run the checklist script from the lesson files:

preship-check.sh reads the container's effective settings with docker inspect and exits 1 if any check fails, so your deploy job can gate on it. Besides the items in the table, it fails a container that shares a host namespace (network_mode: host, pid: host and the like). It also fails one that bind-mounts the Docker socket or a sensitive host path, or gets host devices. It's a common subset, not a full review. Capabilities added back after cap_drop: [ALL] are listed for a human to judge, and seccomp and AppArmor profiles aren't checked. A log driver other than json-file, local or none passes, because its rotation happens in the system it sends to. "Host namespace sharing as attack surface" and "The big three: privileged, docker.sock and host mounts" in Advanced container security explain why those settings are on the list.

The script reads configuration. The process inside the container shows what the kernel actually applied:

CapEff is all zeros and NoNewPrivs is 1 for a process started in the container. PID 1 is docker-init, with Node as its child. A write to /tmp works, and a write you try under /app fails with EROFS (exit=1 is node's exit status). The cgroup files show 256 MiB of memory, no swap, 100 processes and one CPU (100000 microseconds of every 100000).

The same image, run plainly

You get the same image, the same healthy status, and nine failures. The image-level items pass because they live in the image you built. Everything else is what you get by default from docker run: unlimited memory, CPU and processes, the default capability set, a writable root, json-file logs without rotation and no restart. Run the check in the deploy job, not once by hand.

When a limit is hit

A memory limit turns a leak into a restart instead of a host-wide out-of-memory event that kills processes at random. Here's what that looks like from your side as the operator:

tail /dev/zero reads a stream with no newline and buffers it until the 32 MiB limit. OOMKilled=true and exit code 137 (128 + 9, SIGKILL) are the kernel's out-of-memory killer at work. RestartCount=3 with the container now exited is on-failure:3 giving up. The logs of such a container usually say nothing, because a SIGKILL can't be caught. Read the state, then you either raise the limit or fix the leak. "cgroups v2 and resource containment" (Advanced container security) covers how the kernel picks and what memory.high adds. Set --memory-swap equal to --memory, as here. With --memory alone, Docker allows the same amount again in swap on hosts that have swap, and the container slows down long before it's killed.

Finally, here's the reason for exec-form CMD, the SIGTERM handler and the init process:

It took a fraction of a second. A process that ignores SIGTERM would hold every stop, deploy and node drain for the full ten-second timeout before Docker sends SIGKILL.

This checklist doesn't cover where the image came from or whether it has known vulnerabilities. That belongs in the pipeline, before anything reaches a registry. "Docker in CI/CD: build to promote" puts the scan, SBOM, signature and promotion in order.

Try this

Repeat the plain docker run step, but add --read-only and --cap-drop ALL to it, then run ./preship-check.sh against that container. Predict which of the nine failures disappear and which stay, using the checklist table. Compare the result with the plain run's output above before you remove the container.

Takeaway

A healthy image says nothing about the run: the same image passed its health check while the plain docker run failed nine of fifteen checks. Gate deploys on the script's exit status 1, not on a person remembering flags. When a limit does bite, OOMKilled=true and exit code 137 in docker inspect are the evidence, because the container's own logs stay silent after a SIGKILL.

Related