Recipe: NGINX as a reverse proxy

TLS edge, static files, proxying, and re-resolving names after a backend moves.

A backend container restarts during a deploy and comes back healthy. Yet every request through the proxy now answers 502 Bad Gateway until somebody reloads NGINX. The NGINX error log has the reason: connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://172.18.0.2:8000/". That address no longer belongs to the backend. You'll build the usual edge for a Compose stack: NGINX terminating TLS, serving static files and proxying /api/ to a private backend. Then you'll reproduce that 502 and fix it with Docker's DNS server. It runs on the main lab VM, secopslog-docker, in ~/lab/r-nginx with the lesson files.

The stack

There are two services. web is nginxinc/nginx-unprivileged, the NGINX image built to run as UID 101 and to listen on 8080 and 8443. api is a twenty-line Python server that echoes what the proxy sent it, so you can see the headers arrive:

name: lab-edge
 
services:
  web:
    image: nginxinc/nginx-unprivileged:1.31-alpine@sha256:15c994d10d6d78658721c3bcafff14cb281fba2a4bdf9d5ba92c416a472516e3
    ports:
      - "127.0.0.1:80:8080"
      - "127.0.0.1:443:8443"
    volumes:
      - ./${EDGE_CONF:-default.conf}:/etc/nginx/conf.d/default.conf:ro
      - ./site:/usr/share/nginx/html:ro
    secrets: [tls_cert, tls_key]
    read_only: true
    tmpfs:
      - /tmp
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/healthz"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
    depends_on:
      api:
        condition: service_healthy
    restart: unless-stopped
 
  api:
    image: python:3.15-slim@sha256:533b0b2f9b5d46fd097091cf826f27902f1d6c3bca714b6dc74551356aa97a51
    command: ["python", "/app/api.py"]
    init: true
    user: "65534:65534"
    volumes:
      - ./api/api.py:/app/api.py:ro
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=2)"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
    restart: unless-stopped
 
secrets:
  tls_cert:
    file: ./certs/fullchain.pem
  tls_key:
    file: ./certs/privkey.pem

Only web publishes ports, and only on the VM's loopback address. api has no ports: at all, so the one way to it is through the proxy on the Compose network. Both images are pinned by tag and digest. Both containers run with a read-only root filesystem, every capability dropped and no-new-privileges. NGINX only needs /tmp writable: the unprivileged image keeps its PID file and temporary request bodies there, hence the tmpfs. init: true on api puts a small init process in front of Python. That way docker compose stop takes a fraction of a second instead of the ten-second kill timeout ("CMD, ENTRYPOINT and PID 1" (Docker for beginners) explains why). The certificate and key arrive as Compose secrets. Both services have health checks. web checks only that NGINX answers its own /healthz, so a dead backend doesn't mark the proxy unhealthy. That's a choice: if you'd rather have the orchestrator replace the proxy when the backend is gone, probe through it instead.

The NGINX configuration:

# Docker's embedded DNS server. Re-check answers every 10 s instead of
# trusting the record's 600 s TTL.
resolver 127.0.0.11 valid=10s ipv6=off;
resolver_timeout 3s;
 
server {
    listen 8080;
    location = /healthz { access_log off; return 200 "ok\n"; }
    location / { return 301 https://$host$request_uri; }
}
 
server {
    listen 8443 ssl;
    ssl_certificate     /run/secrets/tls_cert;
    ssl_certificate_key /run/secrets/tls_key;
 
    location / {
        root /usr/share/nginx/html;
        try_files $uri $uri/ /index.html;
    }
 
    location /api/ {
        # A variable in proxy_pass makes NGINX resolve "api" at request time.
        set $api_upstream http://api:8000;
        # With a variable, proxy_pass sends the URI unchanged, so strip /api here.
        rewrite ^/api/(.*)$ /$1 break;
        proxy_pass $api_upstream;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The plain-HTTP server answers /healthz and redirects everything else to HTTPS. The TLS server serves files from /usr/share/nginx/html and forwards /api/ to the backend with four headers. They're the original Host, the client address in X-Real-IP and X-Forwarded-For, and X-Forwarded-Proto: https. That last one tells the application the client used TLS, even though the hop inside Docker is plain HTTP. The resolver line, the variable and the rewrite are the fix for the 502; the section after next explains them. ssl_protocols isn't set, because the default since NGINX 1.23.4 is already TLS 1.2 and 1.3.

The key nobody can read

Create a self-signed certificate for localhost. A real deployment uses one from your CA or Let's Encrypt; the configuration is the same:

OpenSSL writes the private key as 0600, owned by ubuntu. Start the stack:

--wait waits for the health checks and exits 1 because web never became healthy. Outside Swarm mode, Compose hands a secret to the container as a read-only bind mount of the host file. So it keeps the host owner (UID 1000) and mode (0600). NGINX runs as UID 101, which can't open it, and exits at startup. The fix is to make the file readable inside the container and protect it on the host with the directory instead:

Mode 0644 on the key looks wrong until you look at the directory. 0700 means no other host user can reach the file, while inside the container the bind mount exposes the file alone. Inside the container, though, 0644 lets every process read the key, whatever its user. If you can change the file's group, chgrp 101 (the image's nginx group) with mode 0640 narrows that to the NGINX user. The lab keeps 0644 because it's the setting that works without root on the host. The long-syntax uid, gid and mode fields on a Compose secret look like the cleaner fix, but they're ignored outside Swarm. "Recipe: WordPress and MySQL with secrets" shows the warning Compose prints. The ps output confirms the topology: api has no published port.

Checking each path

Plain HTTP gets a 301 to HTTPS. The static page comes off disk over TLS. --cacert makes curl verify the self-signed certificate instead of skipping verification with -k. The API reply is the backend describing the request it received. The path arrived as /orders?page=2, without the /api prefix and with the query string intact. host is still localhost, and x_forwarded_proto is https. x_forwarded_for is 172.18.0.1, the network's gateway address. That's because curl connected to a port published on 127.0.0.1, which Docker serves through docker-proxy, the userland proxy. The proxy opens its own connection to NGINX from the gateway. A remote client reaching a port published on all addresses is DNATed and keeps its own address, unless the userland proxy handles it too ("Publishing ports and the packet path" covers both paths). Addresses on the Compose network vary between machines and runs.

One more claim is worth checking, because you'll see it repeated widely: only root can bind ports below 1024, so non-root NGINX must use 8080:

Rootful Docker sets net.ipv4.ip_unprivileged_port_start=0 in every container's network namespace, so UID 101 could bind port 80 here. The unprivileged image listens on 8080 by convention. That convention also works under rootless Docker and in Kubernetes with restricted policies. Run NGINX as non-root because it's non-root, not because of the port. "Capabilities, cap-drop and no-new-privileges" (Advanced container security) covers the sysctl.

Why the proxy keeps a dead address

Most NGINX examples you'll find write proxy_pass http://api:8000/;. static-upstream.conf is that version. The Compose file mounts whichever file EDGE_CONF names, so start web with it:

Now give the backend a new address. When you recreate a container, it usually gets the same address back, because the old one is released first. It changes when something else takes the address while the backend is down: a one-off job, another service scaling up, a rolling deploy. Simulate that with a job container:

api is healthy at 172.18.0.4, and the proxy answers 502. With a literal hostname, NGINX resolves api once, when it loads the configuration, and keeps that address until the next reload. It's still sending requests to 172.18.0.2, which now belongs to the job container. Here the job doesn't listen on 8000, so the connection is refused. If it did, NGINX would send your API traffic, with its headers and cookies, to whatever container holds the old address. Docker's DNS record for api was correct the whole time.

The fix in default.conf has three parts. resolver 127.0.0.11 points NGINX at Docker's embedded DNS server, which every container on a user-defined network can reach at that address ("Networks, drivers and DNS"). valid=10s caps how long NGINX trusts an answer. Docker's DNS answers with a TTL of 600 seconds, which would otherwise apply. And proxy_pass takes a variable. NGINX resolves names in a variable at request time through resolver, instead of once at startup. The variable changes one more behaviour. With a variable, proxy_pass passes the request URI through unchanged, and the trailing-slash prefix stripping of the literal form no longer happens. So the rewrite ... break line removes /api itself. There's a side benefit, too. NGINX starts even when api doesn't exist yet, where the literal form fails at startup with host not found in upstream.

Put your fixed configuration back and move the backend again:

Every request succeeded and reached the backend at its new address. The backend's restart took longer than ten seconds here (the health check runs every ten seconds), so NGINX's cached answer had expired by the first request. A move faster than valid can still get 502 or 504 responses for up to ten seconds. Lower valid if that matters, at the cost of more DNS lookups. NGINX 1.27.3 and later also offer server api:8000 resolve; inside an upstream block with a zone. It re-resolves in the background and keeps upstream keepalive connections. Use it when you need load balancing over several backend addresses.

Renewing the certificate

NGINX reads certificates when it loads its configuration. Renew the files, test the configuration, reload, and check which certificate you now get as a client:

The serial numbers differ: the reload picked up the new certificate without dropping connections. Run nginx -t first, always. A reload with a broken configuration is refused, but a restart with one takes the edge down. Single-file bind mounts, which Compose secrets are, have one trap. OpenSSL rewrote the files in place, so the container saw the new content. Some tools write a new file and rename it over the old one (many editors, certbot's symlink swap, sed -i). Those leave the container looking at the old inode until the container is recreated. Mount a directory instead when your renewal tool works that way, and make it readable by UID 101.

Baking the configuration into an image

For production you may prefer an image that carries its configuration and site, so the image is the whole artifact. Comments in a Dockerfile must start the line. A comment after an instruction is passed to it as arguments:

# Bake the config and the site into the image for production.
# The unprivileged variant runs as UID 101 and listens on 8080/8443.
FROM nginxinc/nginx-unprivileged:1.31-alpine@sha256:15c994d10d6d78658721c3bcafff14cb281fba2a4bdf9d5ba92c416a472516e3
COPY default.conf /etc/nginx/conf.d/default.conf
COPY site/ /usr/share/nginx/html/

The certificate and key still stay out of the image, because anyone who can pull an image can read every layer of it. Keep them as secrets or mounts. Clean up:

Try this

With the fixed configuration running, repeat the proxied curl from "Checking each path" against another path under /api/, with a different query string. Before you look, predict the path the backend reports and the values of host and x_forwarded_proto. Then compare them with the /orders?page=2 reply above.

Takeaway

Behind a Docker network, let NGINX resolve backends at request time through resolver 127.0.0.11 and a variable in proxy_pass. The literal proxy_pass http://api:8000/; kept sending traffic to 172.18.0.2 after api had moved to 172.18.0.4, and every request got a 502 while Docker's DNS was correct. That stale address is worse than an outage when another container answers on it, since the API's headers and cookies then go to the wrong place.

Related