Log pipelines
Collect, normalize, enrich; guard the blind spots.
Every detection depends on the telemetry beneath it, so the log pipeline is the foundation of the whole program. Collection, normalization, and enrichment decide what you can detect — and a gap here is a blind spot no clever rule can overcome.
Collect, normalize, enrich
A pipeline moves events through stages. Collection gathers them from sources — endpoint agents, cloud log exports, Kubernetes audit, network sensors — using shippers like Fluent Bit, Vector, or Beats. Normalization maps disparate formats to a common schema (OCSF or Elastic Common Schema) so a rule can reference the same field name across AWS, endpoint, and app logs. Enrichment adds context — geo-IP, asset owner, threat-intel tags, identity — so an alert arrives actionable rather than as a bare event. Get these right and detections written once work across heterogeneous sources; get them wrong and every rule fights inconsistent, contextless data.
# sources ──▶ collection ──▶ normalize ──▶ enrich ──▶ route# CloudTrail Vector/Fluent → OCSF/ECS +geo-IP → SIEM (hot)# K8s audit Bit / Beats common schema +asset → object store (cold)# endpoint +intel tag → detection engine## Normalization is why ONE rule can match "user X did Y" whether the event# came from AWS, Azure, or an endpoint — the field names line up.
Guard the pipeline itself
The pipeline is also an attack surface and a source of silent failure. Parsing errors can drop the very fields a detection needs, turning a green rule into a blind one, so monitor parse-failure rates. Log-source health monitoring is a detection in its own right: attackers disable logging, and a source that suddenly goes quiet should raise an alert, not go unnoticed. And because ingesting everything is expensive, tier the data — keep high-value security logs hot and searchable, archive the rest cheaply — so cost does not force you to drop the telemetry you need. The pipeline is not plumbing to ignore; it is where detection coverage is won or lost.