Recipe: Redis

Cache or datastore: persistence, memory limits and ACL users.

Start the official Redis image with no config and ask it a question from another container:

protected-mode is no, and a client that knows nothing but the container's name wrote a key. Redis itself has protected mode on by default. With no password and no explicit bind, it refuses clients that don't come from loopback. The official image switches that off, because in a container every client arrives over a network interface, and the default would make the image unusable. Redis notices and logs the warning above, which few people read. So docker run -p 6379:6379 redis on a host with a public address is an open database on the internet. Anything that scans for port 6379 can reach it. Whatever you add in this recipe has to replace protected mode, not rely on it.

Everything runs on the main lab VM, secopslog-docker, as ubuntu in ~/lab/r-redis. The lesson files download has redis.conf, compose.yaml, make-acl.sh and rcli.

Cache or datastore

Decide this first, because it sets most of your config:

Eviction is silent. The client that triggers it gets OK, and the keys that made room simply stop existing. That's right for a cache and wrong for anything that exists only in Redis. You'll build the datastore variant here. The memory section at the end shows what changes when you switch the same instance to cache behaviour.

Access control with an ACL file

requirepass sets one password for the default user, and every client you have then gets full power, including FLUSHALL and CONFIG SET. ACL users (Redis 6 and later) are better. Each client gets a name, a password and only the commands and keys it needs. You'll define three users and write them to an ACL file with a small script:

#!/bin/sh
# Writes secrets/users.acl from the two password files. The ACL file stores SHA-256 hashes only.
set -eu
hash() { printf %s "$(cat "$1")" | sha256sum | cut -d' ' -f1; }
{
  echo "user default off"
  echo "user app on #$(hash secrets/app_password.txt) ~app:* -@all +@read +@write +@keyspace +ping -@dangerous"
  echo "user ops on #$(hash secrets/ops_password.txt) ~* &* +@all"
} > secrets/users.acl

default is switched off, so a client that doesn't log in can do nothing. app may read, write and expire keys whose names start with app:, plus PING for the health check. It gets nothing in the @dangerous category (FLUSHALL, CONFIG, KEYS, DEBUG and the like). ops is the administrator you use for maintenance. A # followed by a SHA-256 hash stores the password's hash instead of the password. So anyone can read the ACL file without learning either password. Create the passwords and the ACL file:

Your hashes will differ on every run. The files are 0644 in a 0700 directory. Redis runs as the redis user (UID 999) and reads the ACL file itself. A Compose secret without Swarm is a bind mount that keeps the host file's owner and mode, so 0600 would lock Redis out ("Recipe: PostgreSQL, MySQL and MongoDB" shows that failure for Postgres and Mongo).

The configuration

# Redis as a small datastore: writes are journaled and nothing is evicted silently.
# appendonly is set on the command line in compose.yaml (an RDB restore needs it off for one start).
aclfile /run/secrets/redis_users_acl
appendfsync everysec
maxmemory 64mb
maxmemory-policy noeviction

name: lab-cache
 
services:
  redis:
    image: redis:8-alpine
    # set REDIS_APPENDONLY=no only for the first start of an RDB restore
    command: ["redis-server", "/usr/local/etc/redis/redis.conf", "--appendonly", "${REDIS_APPENDONLY:-yes}"]
    volumes:
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
      - redisdata:/data
    secrets:
      - redis_users_acl
      - redis_app_password
      - redis_ops_password
    mem_limit: 256m
    healthcheck:
      test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(cat /run/secrets/redis_app_password)\" redis-cli --user app ping | grep -q PONG"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 30s
      start_interval: 1s
    # time to finish the AOF/RDB write on shutdown before Docker sends SIGKILL (default 10s)
    stop_grace_period: 1m
    restart: unless-stopped
    # no ports: services on this project's network reach it as redis:6379
 
volumes:
  redisdata:
 
secrets:
  redis_users_acl:
    file: ./secrets/users.acl
  redis_app_password:
    file: ./secrets/app_password.txt
  redis_ops_password:
    file: ./secrets/ops_password.txt

aclfile points at the secret. appendfsync everysec is the default, written down because it's the durability promise. The append-only file (AOF) is flushed to disk once a second, so a power loss costs at most about a second of acknowledged writes. always flushes on every write and is much slower; no leaves it to the kernel. maxmemory 64mb with noeviction is the datastore choice you picked from the table. appendonly is passed on the command line instead, so one restore step below can switch it off without editing the file.

The command starts with redis-server. The image's entrypoint drops from root to the redis user (with setpriv, keeping no capabilities) when it starts the server. It also recognises a first argument ending in .conf, so a bare config path works too. A command that starts with sh -c would skip the drop and run the server as root. The entrypoint also fixes ownership of files in /data and of the config file when the redis user can't read them. That fails harmlessly on this read-only mount, and it's one more reason to make the file readable. The health check logs in as app and greps for PONG, because redis-cli exits 0 even when the server answers with an error such as NOAUTH. mem_limit: 256m leaves the server room above maxmemory, for reasons the memory section shows. And there's no ports: entry. Services in this project reach Redis as redis:6379 on the project network, and nothing else can.

Typing a password for every redis-cli call is clumsy, and passing one on a command line leaks it into process listings. So a small wrapper runs redis-cli inside the container as an ACL user, with the password read from the secret file there:

#!/bin/sh
# usage: ./rcli USER ARGS...  runs redis-cli in the redis container as the ACL user USER.
# The password is read from the secret file inside the container, so it never appears in a host
# command line or in redis-cli's arguments.
user=$1; shift
exec docker compose exec -T redis sh -c \
  'REDISCLI_AUTH="$(cat "/run/secrets/redis_${0}_password")" exec redis-cli --user "$0" "$@"' "$user" "$@"

Up, and who can do what

The PORTS column shows 6379/tcp with no host address or arrow: exposed inside the network, published nowhere. A client container on the project network reaches the server by name and gets NOAUTH. exit=0 shows redis-cli reporting success for it, as promised. The app user writes and reads its own keys. It's refused a key outside app:, and it's refused CONFIG and FLUSHALL. Through ops, you can confirm the settings: 64 MiB, noeviction, AOF on. protected-mode is still no, which no longer matters because nothing's reachable without a password. The server process runs as redis.

If a tool on the host itself needs Redis, publish the port on loopback only, 127.0.0.1:6379:6379, and keep the ACL users. A bare 6379:6379 publishes on every address of the host. The Docker-managed firewall rules for published ports take effect before most host firewall setups ("Publishing ports and the packet path" explains why ufw doesn't help there).

Persistence: AOF and RDB

Your key survived a restart because Redis replayed the AOF on start. Since Redis 7 the AOF is a directory. It holds a base file in RDB format, an incremental file with the writes since, and a manifest that ties them together. dump.rdb is a point-in-time snapshot, written on shutdown and on the default schedule. That's after 1 hour if 1 key changed, 5 minutes if 100 changed, or 1 minute if 10,000 changed. The final write on shutdown is why the Compose file sets stop_grace_period: 1m. With a large dataset it can take longer than Docker's default ten seconds, after which Docker sends SIGKILL and the write is lost. If you've enabled the AOF, restarts load it and ignore dump.rdb. You'll see why that rule matters in a moment.

Backup and restore

redis-cli --rdb asks the server for a fresh snapshot over the replication protocol and writes it out, here to stdout and into a host file. It needs a user allowed to run SYNC, which is why it runs as ops:

The progress messages go to stderr, apart from the snapshot itself. The file starts with the REDIS magic and the RDB format version. Your sizes will vary. Now the trap. Lose the volume, copy the backup into a fresh one as dump.rdb, and start the normal config:

The key's gone (the empty line is the reply for a missing key). With appendonly yes and no AOF in the volume, Redis created a new, empty AOF base on start and loaded that instead of dump.rdb. Nothing failed, and the health check was green. The working procedure starts the server once with AOF off, so it loads the snapshot. Then it turns AOF on at run time, which writes a new AOF from memory:

The first GET shows the snapshot loaded. CONFIG SET appendonly yes started an AOF rewrite, and the loop waited until it finished. Then docker compose up -d recreated the container with the normal command line, which loaded the new AOF. Keep your backup until that last check passes. You can also copy a stopped instance's whole /data (AOF directory included), with the stop-copy-start method from "Volumes, bind mounts and tmpfs in practice". That's the other valid backup.

Memory: maxmemory, the policy and the container limit

Fill the instance past its 64 MiB with 400,000 keys of 200 bytes each. You'll use redis-cli --pipe, which streams commands without waiting for each reply:

With noeviction, Redis accepted writes until it reached maxmemory. Then it answered every further write with an OOM error, which --pipe counts in its summary (the exact split varies between runs). The app's next SET failed loudly too. That's the datastore behaviour you want: a failed write reaches someone, but a silently evicted job doesn't. Switch the same instance to cache behaviour:

Now the write succeeds, and evicted_keys counts the keys Redis dropped to make room. Redis samples a few keys per eviction (maxmemory-samples, 5 by default) rather than keeping an exact LRU order, so which keys go is approximate. Watch evicted_keys on a cache. A steady climb means your working set is bigger than maxmemory.

The container limit is separate from maxmemory and must be larger. Redis uses more memory than its data. There's allocator fragmentation, client buffers, and above all the copy-on-write cost of a fork. RDB snapshots and AOF rewrites run in a child process that shares memory with the parent until the parent changes a page. Under heavy writes during a rewrite, the total can approach twice the dataset. Set maxmemory to roughly half to two thirds of the container limit. If the kernel's memory limit is hit first, Redis is killed instead of answering with errors, and you lose everything since the last fsync. Redis also logs a warning when the host has vm.overcommit_memory at 0, because a fork can then fail under memory pressure. That's a host sysctl, set on the Docker host (not in the container), and the lab VM leaves it unchanged.

Clean up:

Try this

Before the cleanup step, run ./rcli app KEYS 'app:*' and ./rcli app GET billing:1 the way the app step ran its commands. Predict each reply from the ACL rules above (KEYS sits in @dangerous, and billing:1 is outside app:), then compare with the NOPERM lines the app step printed. Finish with ./rcli ops INFO stats | grep '^evicted_keys' and check whether the count moved since the lru step.

Bottom line

Give Redis named ACL users and no published port. The stock image ships with protected-mode set to no, and it let a client that knew only the container's name write a key. Decide cache or datastore before anything else. With noeviction a full instance returns OOM errors, while allkeys-lru reported evicted_keys:243 and returned OK, so the wrong policy turns lost data into a silent success.

Related