CoursesAdvanced Linux securityForensics & incident response

Live triage: contain, do not destroy

The first hour, done right.

Advanced14 min · lesson 15 of 17

When a detection fires or a hunt hits, you shift from finding to responding, and the first hour sets the tone for the whole incident. The cardinal rule of live triage mirrors the container-forensics lesson: contain without destroying the evidence. The panic instinct — reboot it, kill the process, wipe and reinstall — is exactly wrong, because it erases the volatile evidence (running processes, network connections, memory, open files) that tells you what happened and how far it spread. Preserve first, then contain, then investigate.

The first hour: order of operations
1preserve
capture volatile state
2isolate
cut network, do not power off
3investigate
scope from the evidence
4remediate
only after scoping
Isolate the host from the network (contain the spread) but keep it running — powering off destroys memory and running-process evidence.

Capture volatile evidence first

Volatile evidence disappears on reboot and degrades every minute, so capture it early: the full process list and their open files and network connections, the current network connections, logged-in users, loaded kernel modules, and — most valuable and most perishable — a memory image, which can reveal injected code, hidden processes, and secrets that never touch disk. Order of volatility guides you: grab memory and live state before disk, because disk survives and memory does not. On a possibly-rootkitted host, gather with trusted static tools and, ideally, corroborate with off-host telemetry.

terminal
# capture volatile state BEFORE containment/reboot (use trusted static tools)
$ ps -efww > /evidence/ps.txt # processes + args
$ ss -tanp > /evidence/connections.txt # network connections + owning process
$ lsof -n > /evidence/openfiles.txt # open files/sockets
$ cat /proc/modules > /evidence/modules.txt
# a memory image (most perishable, most valuable) with AVML/LiME:
$ sudo ./avml /evidence/memory.lime

Isolate, do not power off

Containment means stopping the spread while preserving the scene. Cut the host off the network (or move it to a quarantine segment) so the attacker loses access and cannot pivot further or destroy evidence remotely — but keep it running, because powering off discards memory and the live process state you have not finished capturing. Do not log in as the account that may be compromised, avoid running the host’s own possibly-subverted tools where you can, and document every action with timestamps: your response is also part of the record.

Reboot/reimage on reflex and you destroy the case
The most common and most costly incident-response mistake is the reflexive "just rebuild it" — it evicts the attacker from one host but destroys the evidence you needed to answer the questions that actually matter: how did they get in, what else did they touch, what did they take, are they still in other hosts. Isolate and preserve first; you can always rebuild after you have captured what you need. A remediated host with no captured evidence is an incident you will never fully understand — or close.