Env vars, dotenv, and shells

Why convenient delivery becomes permanent exposure.

An environment variable is a sticky note pinned to a running program. The program reads it whenever it likes, anything the program starts gets its own photocopy of the note, and anyone who can look closely at the program can read what the note says. For a setting like LOG_LEVEL=debug that behaviour is exactly what you want. For a production database password it is a strange arrangement, and it is where most production credentials live today.

The mechanism is plain. The environment is a list of NAME=value strings that the operating system copies into a process's memory when it starts. Nothing in it is encrypted. There is no per-variable permission, no record of who read what, no expiry. It is deliberately easy to get at, because programs need to get at it, and every property that makes it convenient for your app makes it convenient for whatever else is running on the box.

Nobody is asking you to rip environment variables out of your stack this afternoon. The goal is to know the price of the convenience, so you can decide which credentials can afford it and which cannot.

What The Environment Actually Is

Linux keeps a public filing cabinet about every running program, and the drawer is called /proc. Every program has a PID (process identifier, the number the kernel uses to keep track of it), and each PID gets a directory under /proc that exposes the program's internals as files you can read with ordinary commands. Two of those files decide how far a secret travels.

# start a lab process carrying a fake credential in its environment
DB_PASSWORD='8f2Kd9Lq4Xn' sleep 600 &
PID=$(pgrep -n -f 'sleep 600')
 
# how the kernel protects each of the two files
ls -l /proc/$PID/environ /proc/$PID/cmdline

[1] 4812
-r--r--r-- 1 deploy deploy 0 Jul 27 10:14 /proc/4812/cmdline
-r-------- 1 deploy deploy 0 Jul 27 10:14 /proc/4812/environ

Read the permission bits on the left. cmdline, which holds the command line the process was started with, is -r--r--r--: readable by every account on the machine, including the low-privilege user your web server drops to and the service account a contractor was handed last year. environ is -r--------: readable by the user the process runs as, plus root, which can read anything. Both files report a size of zero because /proc builds their contents at the moment you ask rather than storing them anywhere.

So the environment is not public, but it is not confidential either. Same user means full read access. On a host where your application, a log shipper, a metrics agent, and a nightly cron job all run as the same service account, every one of them can read the others' secrets. Some hardened builds mount /proc with the hidepid option so you only see your own processes, which helps, and it is not the default on the distributions most teams run.

# entries are separated by invisible zero bytes; swap them for newlines
tr '\0' '\n' < /proc/$PID/environ | grep DB_PASSWORD

DB_PASSWORD=8f2Kd9Lq4Xn

That is the whole attack. No exploit, no privilege escalation, no vulnerability in your code. One tr and one grep.

Command Lines Are Worse

Because cmdline is world-readable, a credential passed as a command-line argument is available to every account on the host, not only to yours. That list of arguments goes by the name argv in most tool documentation. This is the one thing to fix before anything else in this lesson.

# the wrong way: the token is an argument
curl -sS -H 'Authorization: Bearer glpat-8xK2mQvT9zR4wLbN' \
  https://gitlab.example.com/api/v4/projects > /dev/null &
 
# any account on the host can run this, not only the one that started curl
ps -eo user,pid,args | grep '[c]url'

deploy    5127 curl -sS -H Authorization: Bearer glpat-8xK2mQvT9zR4wLbN https://gitlab.example.com/api/v4/projects

Notice the quotes are gone. Each argument is a separate string in the process, and ps prints them joined by spaces, so what you typed as one quoted header comes back as bare words. A few tools defend themselves here. The MySQL command-line client walks over its own argument list at startup and overwrites the password with x characters, so ps shows padding instead of the value. There is a short window before that overwrite lands where the real password is still visible, MySQL's own documentation says the trick is not reliable on every system, and most other tools do not attempt it at all. Feed credentials through a file or through standard input instead. curl reads headers from a file with -H @path, docker login takes --password-stdin, rsync and restic take --password-file, and psql reads the PGPASSFILE environment variable pointing at a file rather than the password itself.

# umask 077 makes anything created here mode 0600
umask 077
printf 'Authorization: Bearer %s\n' "$(cat /run/ci/gitlab_token)" > /run/ci/gitlab.hdr
 
curl -sS -H @/run/ci/gitlab.hdr \
  https://gitlab.example.com/api/v4/projects > /dev/null &
ps -eo user,pid,args | grep '[c]url'

deploy    5310 curl -sS -H @/run/ci/gitlab.hdr https://gitlab.example.com/api/v4/projects
# the argument list now names a file instead of carrying the token

Every Child Process Gets A Copy

When a process starts another process, the child receives a copy of the entire environment. It is never asked what it needs and never told what it got. That inheritance is why the environment is the richest single target in a build pipeline.

export STRIPE_SECRET_KEY='sk_live_51NqB7hK2mVx8RtYw'
 
# one line any script in your build is free to run
node -e "console.log(Object.keys(process.env).filter(k => /KEY|TOKEN|SECRET|PASSWORD/.test(k)))"

[
  'AWS_SECRET_ACCESS_KEY',
  'GITHUB_TOKEN',
  'NPM_TOKEN',
  'STRIPE_SECRET_KEY'
]

That one-liner is the kind of thing a package's postinstall script can run. postinstall is a command a dependency is allowed to execute automatically the moment it is installed, with no prompt and no review. Your CI (continuous integration, the service that builds and tests your code on every push) runner exports every project secret into the environment and then runs npm ci, which fires postinstall scripts for hundreds of packages written by people you have never met. The Codecov compromise in 2021 worked on this exact surface: an attacker altered a widely used upload script so that it read the environment of thousands of build jobs and posted the contents to a server they controlled. The script kept doing its advertised job the whole time, which is why it ran for roughly two months before anyone noticed.

Where Dotenv Files Go Wrong

A .env file is the spare key under the doormat. On your own laptop it is a sensible convenience. The trouble starts when the doormat gets photographed, copied, and shipped to a container registry.

A dotenv library reads NAME=value lines from that file at startup and loads them into the process environment, so everything above applies the second the file is read. The file adds three fresh problems of its own: it gets committed, it gets copied into container images, and it gets pasted into chat when a new developer needs to run the app locally.

Start with the commit. .gitignore only affects files git is not already tracking. If .env was committed once, six months ago, adding it to .gitignore today changes nothing at all.

echo '.env' >> .gitignore
 
# is git ignoring it now?
git check-ignore -v .env; echo "exit=$?"
 
# is git still tracking it?
git ls-files --error-unmatch .env

exit=1
.env
# check-ignore printed nothing and exited 1, while ls-files found the file:
# .gitignore has no say over a file git already knows about

That silence from check-ignore is the useful signal. It consults the index before it answers, so a tracked file is reported as not ignored no matter what your patterns say. Add --no-index and it will happily tell you .gitignore:1:.env, which is how people talk themselves into believing the problem is solved.

git rm --cached .env stops tracking the file from the next commit onward. The value stays in every commit before that, in every clone, and in every fork, which is the next lesson's subject. Rotate the credential and treat untracking as tidying up, never as the fix.

The second problem catches more teams than the first, because the two ignore files look alike and share nothing. Git never reads .dockerignore, and a Docker build never reads .gitignore. A COPY . . line copies your working directory into the image, .env included.

FROM node:20-slim
WORKDIR /app
 
# copies the whole working directory, including files git was told to ignore
COPY . .
 
RUN npm ci --omit=dev
CMD ["node", "server.js"]

docker build -t payments-api:1.4.2 .
docker run --rm payments-api:1.4.2 cat /app/.env

 => [1/4] FROM docker.io/library/node:20-slim              2.7s
 => [2/4] WORKDIR /app                                     0.0s
 => [3/4] COPY . .                                         0.1s
 => [4/4] RUN npm ci --omit=dev                            8.4s
 => exporting to image                                     0.6s
 
DATABASE_URL=postgres://app:8f2Kd9Lq4Xn@db.internal:5432/payments
STRIPE_SECRET_KEY=sk_live_51NqB7hK2mVx8RtYw
# the file git never saw is now a layer in an image sitting in your registry

Anyone who can pull that image holds both credentials, which includes every CI job, every developer laptop, and every cache in between. Deleting the file in a later RUN line does not help either, because the layer that contains it is still in the image and anyone can unpack it. The control is a .dockerignore file in the build context, so the file never enters the build at all.

# read by the Docker build only; git never looks at this file
.env
.env.*
!.env.example
.git
.git-credentials
*.pem
*.p12
credentials.json
node_modules

Both of the first two lines are needed, because .env.* matches .env.production but not .env itself. The ! line then puts .env.example back, since the last matching rule wins. And .git on that list matters as much as .env. Copying the repository's .git directory into an image ships every secret that was ever committed to that repo, including the ones somebody deleted two years ago and assumed were gone.

For the third problem, keep a .env.example in the repo with the keys present and the values empty, so a new developer learns what to fill in without anyone pasting live values into a chat window. Run a secret scanner in pre-commit and in CI whatever else you do. An ignore file keeps a value out of one commit. It does nothing about your image, your backups, your laptop, or the copy sitting in a teammate's Downloads folder.

Your Shell Remembers Everything

Typing export STRIPE_SECRET_KEY=sk_live_... into a terminal writes that line into a plain text file in your home directory, where it sits until the history file rolls over. Plenty of people then sync their dotfiles to a repository.

export GITLAB_TOKEN='glpat-8xK2mQvT9zR4wLbN'
 
# bash writes the file when the shell exits; -a flushes it now so you can look
history -a
tail -2 ~/.bash_history

export GITLAB_TOKEN='glpat-8xK2mQvT9zR4wLbN'
history -a

Two habits close this. A leading space keeps a command out of the history file, and an interactive read keeps the value off the command line entirely.

# bash: drop any command that starts with a space (zsh: setopt HIST_IGNORE_SPACE)
HISTCONTROL=ignorespace
 export GITLAB_TOKEN="$(cat /run/ci/gitlab_token)"
 
# better: the value never appears on a command line at all
read -rs -p 'Token: ' GITLAB_TOKEN && export GITLAB_TOKEN
history -a && tail -2 ~/.bash_history

Token: read -rs -p 'Token: ' GITLAB_TOKEN && export GITLAB_TOKEN
history -a && tail -2 ~/.bash_history
# the space-prefixed export never reached the file, and the read command is
# there without the value you typed into it

-s stops the terminal echoing your keystrokes, which is why the prompt and the next line run together in that output, and -r stops backslashes in the pasted value being swallowed as escape characters. The history file deserves respect as a target in its own right. Bash creates it mode 0600, so the permissions are fine, and permissions are not the issue. It gets backed up, synced into dotfiles repos, and swept up wholesale by support bundles and forensic collection scripts.

Orchestrators Widen The Blast Radius

Kubernetes gives the environment a far bigger audience, and one level of nesting in a YAML file (the indented text format Kubernetes configuration is written in) decides how big.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api:1.4.2
          env:
            - name: LOG_LEVEL
              value: "info"              # configuration: plain text is correct here
            - name: DB_PASSWORD
              value: "8f2Kd9Lq4Xn"       # the credential itself, in the manifest
            - name: STRIPE_SECRET_KEY
              valueFrom:
                secretKeyRef:            # a pointer instead of the value
                  name: payments-secrets
                  key: stripe

kubectl get deploy payments-api \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq

[
  {
    "name": "LOG_LEVEL",
    "value": "info"
  },
  {
    "name": "DB_PASSWORD",
    "value": "8f2Kd9Lq4Xn"
  },
  {
    "name": "STRIPE_SECRET_KEY",
    "valueFrom": {
      "secretKeyRef": {
        "name": "payments-secrets",
        "key": "stripe"
      }
    }
  }
]

The DB_PASSWORD line hands your database password to anyone allowed to read Deployments in that namespace, and read access to Deployments is given away freely: every developer, every dashboard, every GitOps bot, every new hire with a view role under RBAC (role-based access control, the rules deciding who may read which Kubernetes objects). The manifest also lives in a git repository, so one careless line leaked the same value twice.

The secretKeyRef version stores a pointer, so the Deployment gives away the name of a Secret rather than its contents. That is a genuine improvement and a modest one. Anyone with read access to Secrets in that namespace fetches the value in one command, anyone who can run kubectl exec prints the container's environment directly, and the value inside the Secret object is base64. Base64 is an encoding, a reversible way of writing arbitrary bytes using safe characters, and it is not encryption of any kind. There is no key and no secret involved in reading it back. The stored bytes in etcd, the database behind your cluster, are also unencrypted unless somebody explicitly turned on encryption at rest with an EncryptionConfiguration on the API server, which is off by default. A later lesson pulls that apart properly.

Every platform repeats the shape. An Amazon ECS (Elastic Container Service, AWS's own container scheduler) task definition has an environment block that stores values in plain text for anyone who can call describe-task-definition, and a separate secrets block whose valueFrom field takes an ARN (Amazon Resource Name, the unique identifier of an AWS resource) pointing at Secrets Manager or Parameter Store. AWS Lambda environment variables are encrypted at rest with a KMS (Key Management Service, Amazon's managed encryption key service) key and handed back in plain text to anyone permitted to call GetFunctionConfiguration. Encrypted at rest describes the disk. It says nothing about who may ask the API for the value.

What Environment Variables Are Still Right For

LOG_LEVEL, NODE_ENV, AWS_REGION, FEATURE_NEW_CHECKOUT, a telemetry endpoint: put all of it in the environment and stop thinking about it. None of those values grant access to anything, so the porousness costs you nothing.

The twelve-factor app methodology, a set of guidelines Heroku published in 2011 that half the industry copied, told everyone to keep configuration in the environment. That advice was about portability, and it was written when the common alternative was a password hardcoded into config/production.rb and committed to the repo. The environment was a large improvement on that. It was never a claim that the environment is confidential. Split the two ideas in your code review checklist: configuration goes in the environment without discussion, credentials get a decision.

Here is the honest ledger. Environment delivery buys you no file permissions to get wrong, no volume to mount, no extra file written at runtime, and identical behaviour on a laptop and in a cluster. It costs you every child process, every crash reporter, every docker inspect, everyone with get deploy, and a copy in whatever manifest or CI settings page defined the value in the first place. The part people forget is rotation. Nothing outside a running process can change its environment, so a new value means a restart. A rotation that needs a rolling restart of forty services at 3am is a rotation that quietly never happens.

Moving A Credential Out Of The Environment

The alternative is to make the program walk over and pick up its key rather than wearing it pinned to its shirt. On a plain Linux host that means a file with tight permissions on a temporary filesystem.

sudo mkdir -p /run/payments
sudo install -o app -g app -m 600 /dev/null /run/payments/db_password
printf '%s' '8f2Kd9Lq4Xn' | sudo tee /run/payments/db_password > /dev/null
 
ls -l /run/payments/db_password

-rw------- 1 app app 11 Jul 27 10:22 /run/payments/db_password

/run is a tmpfs on most distributions, meaning a filesystem that lives in RAM (random access memory) and vanishes at reboot, so the credential never touches a disk that could be imaged, snapshotted, or backed up. Mode 0600 means the owning user reads and writes it and nobody else reads it at all. The two-step dance with install matters: it creates the file with the right owner and mode while it is still empty, and tee then truncates and rewrites that same file rather than deleting and recreating it, so the permissions you set survive the write. Eleven bytes, no trailing newline, because printf '%s' does not add one.

const fs = require('node:fs');
 
// read at connect time, not once at boot into a variable that never changes
function dbPassword() {
  return fs.readFileSync('/run/payments/db_password', 'utf8').trim();
}
 
module.exports = { dbPassword };

Reading at connect time is what makes rotation cheap. Write a new value into the file and the next connection uses it, with no restart and no deploy. Connections already open keep the old credential until the pool recycles them, so size your rotation window around the pool's maximum connection lifetime rather than assuming the switch is instant.

Then verify it, because a control you have not checked is a belief.

PID=$(pgrep -n -f 'node server.js')
 
# sudo has to wrap the read, not the tr: a plain '< /proc/...' redirect is opened
# by your shell before sudo ever runs, so it fails and the check reports success
sudo cat /proc/$PID/environ | tr '\0' '\n' \
  | grep -Ei 'pass|secret|token|key' \
  || echo 'no credential-shaped variables in environ'
 
# and nothing in the command line, which every account can read
tr '\0' ' ' < /proc/$PID/cmdline; echo

no credential-shaped variables in environ
node server.js

That comment about the redirect is the trap worth remembering. grep exits non-zero both when the environment is clean and when the read was denied, so a check running as the wrong user prints a cheerful all-clear about a process it could not open. Run it as root, and if you want to sleep at night, make the script fail loudly when cat returns an error instead of falling through to the echo.

Point that check at every host from a nightly job and you own one of the cheapest detections in this course. Any host that answers with a value to the right of an = sign is a finding, and the finding names the service and the variable for you.

Platforms have their own version of the same move. Kubernetes projects a Secret as files under a path you choose, systemd has LoadCredential=, which drops a value into a 0400 file readable by one service and keeps it out of both ps and the environment, and Docker has secret mounts. The Kubernetes file mount comes with a bonus the environment cannot match: update the Secret and the kubelet refreshes the file inside the running pod, usually within a minute or two. One exception trips people up repeatedly. If you mount with subPath to place a single file at an exact location, the mount is resolved once when the container starts and never updates again. Mount the directory instead.

Try This

Do this on a laptop or a throwaway VM (virtual machine, a disposable computer running inside your real one), never on a production host, and use the fake token below rather than anything real. You are proving the read surfaces exist, not testing your luck.

# 1. a credential in the environment: only your own account can read it
DEMO_TOKEN='glpat-notARealToken0000000000' sleep 300 &
tr '\0' '\n' < /proc/$!/environ | grep DEMO_TOKEN
 
# 2. the same value as an argument. 192.0.2.1 is a reserved test address that
#    routes nowhere, so curl sits and waits, holding the token in its argv
curl -s --max-time 300 \
  -H 'Authorization: Bearer glpat-notARealToken0000000000' \
  http://192.0.2.1/ > /dev/null &
sudo -u nobody ps -eo user,pid,args | grep '[g]lpat'
 
# 3. the file route: node reads the token out of a 0600 file and stays up, and
#    neither of its two /proc files gives the value away
umask 077
printf '%s' 'glpat-notARealToken0000000000' > /tmp/demo.token
ls -l /tmp/demo.token
node -e "require('fs').readFileSync('/tmp/demo.token','utf8'); setTimeout(()=>{}, 300000)" &
tr '\0' '\n' < /proc/$!/environ | grep -Ei 'token|glpat' \
  || echo 'no credential-shaped variable in environ'
tr '\0' ' ' < /proc/$!/cmdline; echo

DEMO_TOKEN=glpat-notARealToken0000000000
deploy    6032 curl -s --max-time 300 -H Authorization: Bearer glpat-notARealToken0000000000 http://192.0.2.1/
-rw------- 1 deploy deploy 29 Jul 27 10:31 /tmp/demo.token
no credential-shaped variable in environ
node -e require('fs').readFileSync('/tmp/demo.token','utf8'); setTimeout(()=>{}, 300000)
# step 2 is the one that matters: the account 'nobody' read a token owned by 'deploy'

Use a real command like curl for step 2 rather than wrapping the token in bash -c '...'. Bash replaces itself with the last command in the script when it can, so the wrapper's arguments disappear from /proc and you end up proving the opposite of what you meant to. Step 3 only means something read against step 1. Same token, same host, and the node process really did load the value, but this time it arrived through a file rather than the environment, so neither /proc file has anything to hand over.

Takeaway

The environment is not a hiding place, and it was never built to be one. It is a note the operating system copies to every child process, hands to any tool that prints configuration, and cannot change while the process is running. That is a fine deal for a log level and a bad one for a live key, so make it a decision you take per value instead of a default you inherit.

Next comes the place where a leaked value is hardest to take back: git history, where a deleted secret keeps working in every clone that already exists.

Related