Courses Advanced Linux internals & toolingAdvanced tooling & observability

The BPF toolkit (bcc & bpftrace)

The famous production-safe tracing tools.

Advanced14 min · lesson 14 of 17

The most celebrated modern Linux performance tools are built on eBPF, and they answer specific questions on live production systems with almost no overhead — the thing strace and even perf cannot always do safely. Two projects dominate: bcc (a collection of ready-made Python/C tools) and bpftrace (a high-level tracing language for quick one-liners). Together they are the "BPF performance tools" that redefined Linux observability, giving you a targeted instrument for nearly every subsystem.

The famous BCC tools, by subsystem

processes & syscalls

execsnoop

every new process

opensnoop

every file opened

syscount

syscalls by count

disk & network

biolatency

disk I/O latency histogram

biosnoop

per-I/O detail + latency

tcpconnect / tcplife

connections + their lifespan

Each tool answers one question cheaply enough to run on a busy production host — the whole point.

bcc: ready-made tools

The bcc collection ships dozens of purpose-built tools, each solving a classic diagnostic. execsnoop catches surprise short-lived processes that top never sees; opensnoop shows exactly which files a program touches; biolatency renders disk latency as a histogram so you can spot a bimodal "usually fast, sometimes terrible" distribution; tcplife logs every TCP connection with its duration and bytes. You run them by name (often with a -bpfcc suffix) and read the output — no code to write.

terminal

$ sudo execsnoop-bpfcc                 # every process that runs, live
PCOMM        PID    PPID   ARGS
sh           4821   812    /bin/sh -c /opt/app/hook.sh
$ sudo biolatency-bpfcc                 # disk latency as a histogram (Ctrl-C to print)
     usecs        : count     distribution
     128 -> 255   : 1832      |********************|
    2048 -> 4095  : 47        |*                   |   <- a slow tail worth chasing
$ sudo tcplife-bpfcc                    # every TCP connection: pid, addrs, bytes, duration

bpftrace: one-liners

When no ready-made tool fits, bpftrace lets you write a custom probe in one line — an awk-like language for the kernel. You attach to a tracepoint, kprobe, or uprobe and aggregate on the fly: count syscalls per process, histogram a function’s latency, tally which files are opened most. It is the fast path from "I wonder..." to a real answer, and it is safe to run live because it rides the same verified eBPF machinery.

terminal

# count syscalls per process, live
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
@[nginx]: 48211
@[postgres]: 15903
# histogram of read() sizes
$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read { @ = hist(args->ret); }'

eBPF tools need a recent kernel and privilege

The BPF toolkit depends on a reasonably modern kernel (roughly 4.9+ for bcc, newer for the best features) and requires root/CAP_BPF because it loads kernel programs — so it may not be available on old or locked-down hosts. Where it is, it is the right tool for live investigation; where it is not, fall back to perf sampling and the /proc-based classics. And remember it is observation, not a fix: these tools find the problem fast, but the remediation is still yours to design.

Flame graphs & CPU profilingNetwork debugging tools