Container forensics and incident response

Contain, capture and diff a suspect container before you destroy it.

The reflex that ends an outage, kill it and let it come back clean, destroys the evidence of a compromise. A container keeps that evidence in places the quick fixes erase in seconds. Live process memory and open sockets are gone when the process dies. The container's log and configuration are gone on docker rm with the default json-file driver. So is the writable layer stacked on the read-only image. Your job in the first few minutes is preservation, in one fixed order: contain, capture, investigate, remediate. You'll run that order on a benign suspect on the main lab VM (secopslog-docker), in ~/lab/forensics. The suspect plants its own fake evidence and exploits nothing.

The suspect logs one line, drops a copy of a binary and adds a crontab line. It opens a TCP connection to lab-c2, a stand-in for an attacker's server on the same network. Then it sleeps, with the fake command-and-control address in its environment:

Contain: freeze, record, then cut the network

Freeze first. docker pause stops every thread at once through the cgroup freezer. It acts underneath the process, so there's no signal for an attacker to trap, and nothing gets a chance to run a clean-up. A paused container also refuses docker exec. That's why every capture step runs from the host:

Then write down what the daemon knows before anything changes it. The host-side PID is the number the kernel gives the container's main process (not the 1 it sees inside). The next steps read the kernel's view of exactly that process. docker inspect is the full record of how the container was started: image digest, mounts, environment, restart policy, labels and network settings. docker logs is everything the process wrote to stdout and stderr. Both disappear with docker rm, so they go into the evidence directory now. The directory is mode 700. On a real host, write to a root-only location on separate storage, and copy it off the host as soon as you can:

Record the connections next, from the host. nsenter joins only the container's network namespace ("Namespaces from first principles" introduced the method). So the ss binary you run is the host's, and the attacker can't have replaced it:

The ESTAB line is the open channel: the suspect's nc connected to 172.18.0.2:4444, the address of lab-c2. On a real incident that peer address is often the most valuable volatile fact you'll collect. (The LISTEN line on 127.0.0.11 is Docker's embedded DNS resolver.) Now cut the network:

Disconnecting removes the container's interface, so the connection can no longer carry traffic. Anything sent on it fails, the socket times out, and the address that tied it to the network is gone. That's why you saved the socket table first. If the container is actively sending data out, cut first and accept the loss. Or drop its traffic with a host firewall rule on its address, which stops the flow and leaves the socket state readable.

Capture from the host, not from inside

A compromised container controls its own userspace, so its ps, ls and cat can lie, and its own files can be edited. Read the kernel's view from the host instead, and know which parts of it you can trust. Kernel-maintained entries such as /proc/<pid>/exe, fd, maps and status can't be forged from inside. exe is the running binary, and it still resolves even if the attacker unlinked the file (it then reads (deleted)). /proc/<pid>/cmdline and environ are different. The kernel reads them out of the process's own memory, so a process can overwrite them (argument rewriting is routine malware behaviour). And environ shows the environment block the process started with, not variables it set later. Treat those two as claims to corroborate. Pull the dropped artifact with docker cp, which reads the filesystem through the daemon and works on a paused container:

The live binary resolves, the dropped file comes out as an ELF executable, and the fake C2 address is read from the process's environment block.

Diff, export, commit, hash

The image ships read-only and never changes, so everything the container wrote sits in the thin writable layer on top. docker diff reads that layer directly. A well-behaved app writes almost nothing, so a dropped binary and a new crontab stand out:

A is added, C is changed. Take the whole filesystem as a tarball for offline analysis, and commit a runnable snapshot you can detonate later in an isolated lab. docker save writes that snapshot to a file. The evidence directory holds it too, so it doesn't depend on the image staying on this host:

The export holds the dropped binary and the new crontab. Both the export and the saved image are for a lab, never for re-running attacker code on a production host. Last, hash everything you took. A copied file is just a copy. A copied file with a checksum recorded at collection time is something an auditor will still trust months later:

The subshell keeps your shell in ~/lab/forensics. The hashes vary from run to run (timestamps, PIDs and addresses differ). What matters is that SHA256SUMS is written now and travels with the files.

Memory needs CRIU, and the lab does not install it

Disk shows what was written down. A payload that lives only in RAM leaves nothing there. To capture it, you checkpoint the running process tree with CRIU, which docker checkpoint drives:

# experimental daemon + CRIU installed on the host
$ docker checkpoint create --leave-running suspicious cp-incident-42
$ docker checkpoint ls suspicious

docker checkpoint is experimental on Docker 29. The daemon must run in experimental mode, and CRIU must be installed on the host. CRIU supports x86_64, arm64 and other architectures. The lab VM simply doesn't install it, so this course doesn't run it. --checkpoint-dir chooses where the images land. Without it they go under the container's directory in the daemon's data root, which docker rm deletes, so set it to your evidence storage. Without CRIU you stop at the filesystem, configuration, log and socket evidence above. That's enough to answer most of the case.

Investigate, then remediate

Lay the pieces side by side. docker diff and the recovered binary show what ran and what it dropped. The socket table and the environment show who it called. The logs and the inspect record show how it was started and what it printed. Your host auditd or Falco trail shows when it arrived. Only then remediate, and remediate never means restart. Before you remove anything, neutralise the restart policy so the daemon can't bring a copy back over your evidence:

Reconstruct the timeline from the daemon's own event stream, then remove the container. The evidence directory is what survives:

docker events with --since and --until gives each state change with a Unix timestamp. In this run it listed only the create, the start and the pause. The docker cp, the export, the update and the commit are missing, though an earlier run did show archive-path and export. So your own case notes with times are part of the record. Stream events to your log pipeline in production, because the daemon keeps only a short history and loses it on restart. After docker rm the container, its log file and its writable layer are gone. The ten files in ir-42 are what's left. With the evidence off the box, patch the image or close the way in. Rotate every credential the container could reach, and treat them as burned. Redeploy from a freshly built clean image. Then turn the drop you found (a new binary under /tmp writing to /etc/crontab) into a Falco rule, so the next attempt trips an alarm. Clean up:

Try this

On the main lab VM, stop before the clean-up step deletes ir-42. Run sha256sum -c SHA256SUMS inside ir-42 and predict the result: one OK line for each file in the hash listing above. Then append a line to ir-42/logs.txt and run the check again. Predict which file fails before you read the output, then clean up as before.

Takeaway

Freeze and record before you touch the network or the restart policy. docker inspect, docker logs and the writable layer that held /tmp/.miner all vanished with docker rm, and only the ten files in ir-42 survived. Cases usually go wrong in two places: an unless-stopped policy like the suspect's re-runs the entrypoint, or a responder trusts cmdline and environ, which the process can rewrite.

Related