Tracing with perf & eBPF tools
See what a slow process is actually doing.
When you know a process is slow but not why, tracing shows what it is actually doing — which functions burn CPU, which syscalls it waits on, where the time goes inside it. The classic starting tool is strace, which prints every system call a process makes and how long each took; it is invaluable for "the program is stuck" (you see exactly which syscall it is blocked in) though it slows the target, so it is a diagnostic, not a production monitor.
$ strace -T -p 812 2>&1 | head # every syscall the process makes, with timingsread(7, ...) <0.000012>futex(...) <2.104882> # blocked 2.1s on a lock — there is your stall$ strace -c -p 812 # -c: a summary of which syscalls cost the most time
perf: sampling the CPU
For CPU-bound work, perf samples the running process thousands of times a second and tells you which functions were on-CPU most — a statistical profile with almost no overhead. perf top is a live view of the hottest functions system-wide; perf record / perf report profiles a specific command. This is how you find the hot loop or the expensive function without instrumenting the code, and it works on compiled binaries and the kernel alike.
$ sudo perf top # live: the hottest functions across the system$ sudo perf record -g -- ./slow-job # profile a command (with call graphs)$ sudo perf report # browse where the time went, by function
eBPF tools: the modern toolkit
The current generation of performance tools is built on eBPF (the same technology from the security course), giving low-overhead, production-safe answers to specific questions. The bcc and bpftrace collections include ready-made tools: biolatency (disk IO latency as a histogram), execsnoop (every new process), opensnoop (every file opened), tcpconnect (every outbound connection). They answer targeted questions cheaply enough to run on live systems, which strace and even perf cannot always do.
$ sudo execsnoop-bpfcc # every process that executes, live (find surprise spawns)$ sudo biolatency-bpfcc # disk IO latency as a histogram — is it bimodal?$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'# ^ count file opens per program — a one-liner answer to "who is hammering the disk?"