Runtime secrets, done right

Deliver secrets as files, not environment variables, and close the run-time leaks.

If you pass a password with -e DB_PASSWORD=..., anyone who can reach the daemon can read it, whatever else you did to the container. "Environment variables and configuration" (Docker for beginners) already warned that environment values sit in the container config, where tooling reads them back. Here you'll see exactly where a secret passed that way turns up. Then you'll move it to a file and meet the ways a file-based secret can still leak at run time. Everything runs on the main lab VM (secopslog-docker), in ~/lab/runsecrets. All values are fake.

Where an environment secret shows up

Start one container with a secret in its environment, then read it back from outside the container and from the host:

docker inspect prints the value to anyone in the docker group. "The Docker socket and daemon hardening" shows that's the same as root on the host. /proc/<pid>/environ is the environment block the process was handed at start. It's read from the process's memory, and host root can read it. If your code unsets the variable later, that block isn't cleared. And every child process inherits it. Two of those three reads came from outside the container entirely. So the in-container user and a read-only root filesystem change nothing. There are two more exits that are easy to forget: a crash that dumps the environment, and the log it lands in.

The process printed its configuration to stderr on the way down. Docker wrote that to the container's JSON log file. Now the secret sits on disk in the clear, and grep counts it once. An error tracker that attaches the process environment to every exception report forwards it the same way. Before you change how you deliver secrets, find the ones already running. One sweep over every container's environment finds the secret-shaped variables:

Both containers carry the fake password. On a real host, this list routinely turns up live API keys. Rotate what it finds, then fix the delivery so the replacement doesn't land in the same places.

Deliver it as a file

A mounted secret file never enters the environment. So docker inspect, /proc/<pid>/environ and child processes have nothing to show. Compose does this directly. You declare a secret backed by a file on disk and list it on the service. Compose then mounts it read-only under /run/secrets/. Many official images already expect a file through the *_FILE convention. You point the image at the path instead of giving it the value:

name: lab-runsecrets
 
services:
  db:
    image: postgres:18-alpine
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password   # the image reads the file, once, at first start
    secrets: [db_password]
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 2s
      retries: 15
 
  report:
    image: alpine:3.24
    user: "10001:10001"
    depends_on:
      db: { condition: service_healthy }
    secrets:
      - source: db_password
        target: db_password
        uid: "10001"
        gid: "10001"
        mode: 0400
    command: ["sh", "-c", "wc -c /run/secrets/db_password && exec sleep 3600"]
 
secrets:
  db_password:
    file: ./secrets/db_password.txt   # plain text on the host: keep it out of Git and backups

The db service gets POSTGRES_PASSWORD_FILE rather than POSTGRES_PASSWORD. The report service reads the same secret as a non-root user. Create the backing file and bring it up:

The report container couldn't read the file. Its process runs as UID 10001, and the mounted file is owned by the host user that created it. That's the first thing to understand about Compose file secrets outside Swarm. The long-syntax uid, gid and mode keys are Swarm-only, and Compose ignores them with a warning (you can see it in the previous step). So the file arrives with its host ownership and mode, and the reader's UID must match. Look at what actually got mounted and how it reads from outside:

The secret is a read-only bind mount of the host file, not an encrypted object. What you bought is "not in the environment", not "encrypted at rest". So the backing file needs the same care as any credential (keep it out of Git). The db container's config carries only POSTGRES_PASSWORD_FILE, which is a path. The host's view of the postgres process environment holds no password. The entrypoint reads the file and exports the value into its own short-lived init process. Then it unsets it before the long-running server starts.

That last part is image-specific. The postgres image clears it. "Recipe: PostgreSQL, MySQL and MongoDB" (Docker in depth) notes that MySQL's server keeps MYSQL_ROOT_PASSWORD in /proc/1/environ even when you pass it through _FILE. The _FILE convention moves the secret off the command line and out of your Compose file. Whether it stays out of the running process's environment depends on the image.

Fix the ownership so the non-root reader can open it, and the report service reads the secret:

Read at startup, and what rotation needs

An image that reads a secret only at first initialisation won't notice a changed file. Query the database with the current secret. Then change the backing file and restart the database. The database still has the old password, so a client that uses the file's new contents is rejected:

The first query authenticated. After you rotate the file and restart, a client using the file's new contents is rejected. That's because postgres set its password from the file only when it first initialised its data directory. Changing a secret file isn't rotation on its own. Real rotation needs the credential changed at the source (the database, the cloud IAM system), and the consumer has to re-read it. That's why the stronger pattern keeps the source of truth outside the container. A secrets manager such as HashiCorp Vault issues short-lived credentials. A sidecar writes each fresh one into the file the app already reads, on a tmpfs so it never reaches the disk. The credential expires on its own, every fetch is logged, and the container never holds a long-lived secret.

The export anti-pattern

A file-based secret stays out of the environment right up until a startup script copies it back in. This one line, common in hand-rolled entrypoints, undoes the whole thing:

After export DB_PASSWORD="$(cat ...)", the config stays clean. But /proc/<pid>/environ carries the value again, and every child inherits it. Have the application open the file itself. If a tool genuinely needs an environment variable, set it inline on that one process's exec line rather than exporting it into the shell. You should know it's still readable in that single process's environ.

Keep it out of the image, too

A secret written into the container filesystem at run time lives in the writable layer, and docker commit captures that layer. A tmpfs mount doesn't, because it never touches the layer:

The first container wrote a token to its root filesystem. docker diff shows the change, and the committed image carries the file for anyone who pulls it. The second wrote to a tmpfs at /run/secrets. Here docker diff shows only the mountpoint /run/secrets being created (C /run, A /run/secrets), not the token. The committed image's /run/secrets is empty. Write secrets that a process fetches at run time to a tmpfs.

Compose file secrets outside Swarm aren't on one. They're read-only bind mounts of a host file, as the mount table above showed. So that file on the host disk is where the secret lives at rest, and it's what you protect (Swarm secrets do land on a tmpfs). Keep build-time secrets out of image layers with BuildKit's --mount=type=secret, which "Build-time secrets and how images leak them" covers. A distroless image with an env-delivered secret still leaks it. Stripping the image does nothing for a password in the container config that docker inspect prints.

Try this

On the main lab VM, start a fresh container from alpine:3.24 with a fake value in -e DB_PASSWORD=... and --read-only, plus a non-root --user 10001. Before you read anything back, predict which of the reads from the first section still show the value. Then run the same docker inspect -f '{{json .Config.Env}}' and the host-side /proc/$P/environ read, and compare with the output above. Remove the container with docker rm -f when you're done.

Takeaway

A secret in -e is readable from docker inspect, from host /proc/<pid>/environ and from the JSON log file once a crash prints it, and no in-container hardening changes that. Moving it to a file only helps until something reads it back into the environment: the one-line export brought the value straight back into /proc/<pid>/environ. In reviews, check the entrypoint and the image's own _FILE handling too, since MySQL keeps the password in its environment where postgres unsets it.

Related