Flame graphs & CPU profiling

perf + FlameGraph: see the hot path at a glance.

A flame graph is the single most famous performance-visualization in the Linux world — it turns thousands of CPU samples into one picture that shows, at a glance, where a program spends its time. perf tells you the hot functions as a list; a flame graph shows the whole call stack as stacked bars, so you see not just "function X is hot" but the entire path of calls that led there and how the time is distributed across it. Brendan Gregg’s FlameGraph tools made this the standard way to read a profile.

Generating one

The recipe is three steps: record with perf, collapse the stacks into one line each, and render to an interactive SVG. You profile the workload (system-wide or a specific process), fold the samples, and pipe through flamegraph.pl. The result is an SVG you open in a browser and click to zoom into any subtree — the practical way to hand a profile to someone else, or to compare before-and-after a fix.

# 1) record CPU stacks for 30s (system-wide, with call graphs)
$ sudo perf record -F 99 -a -g -- sleep 30
# 2) fold the stacks, 3) render to an interactive SVG
$ sudo perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flame.svg
$ xdg-open flame.svg      # click any box to zoom; search to highlight a function

What flame graphs reveal

The pattern-reading is the skill. A wide plateau at the top means a leaf function burning CPU directly — optimize that function. A wide tower means a deep call chain where the cost is spread across layers — often a sign of an expensive abstraction or too many calls. Two flame graphs side by side (a "differential") show exactly what a change made faster or slower. There are also off-CPU flame graphs (built from scheduler traces instead of CPU samples) that show where a program spends time blocked and waiting, not running — the other half of the latency story.