Parse auditd logs with Python before they hit your SIEM

Raw audit records are unreadable and expensive to index. Normalize them with 80 lines of Python and cut ingest cost.

auditd is the kernel's flight recorder — it can tell you exactly who ran sudo, what process wrote to /etc/shadow, and when. The catch is the format: a single logical event is spread across several type= records, each a wall of key=value pairs. Python turns 'who touched this file last week' from an afternoon of grep gymnastics into a function you run in a second.

This is what one event looks like on disk — three records that belong together, tied by the id inside msg=audit(...):

Group records into events

The whole trick is the event id — the number after the colon in audit(timestamp:ID). Split each line into a dict, pull that id, and collect every record that shares it. Now one event is one Python object instead of three scattered lines.

import re, collections
 
FIELD = re.compile(r'(\w+)=(?:"([^"]*)"|(\S+))')
EVENT = re.compile(r'audit\((\d+\.\d+):(\d+)\)')
 
def parse(path):
    events = collections.defaultdict(dict)
    for line in open(path):
        m = EVENT.search(line)
        if not m:
            continue
        eid = m.group(2)                         # the event id
        for k, q, u in FIELD.findall(line):
            events[eid][k] = q or u              # merge all records
    return events

Answer a real question

With events assembled, questions become one-liners. 'Which non-root user edited a file we watch under the identity key, and with what binary?' — filter on the key and read the fields you kept.

for eid, e in parse('/var/log/audit/audit.log').items():
    if e.get('key') == 'identity' and e.get('auid') not in (None, '4294967295'):
        print(f"uid={e['auid']} ran {e.get('exe')} on {e.get('name')}")

Make sure the events exist

None of this works if the events were never recorded. Add watch rules so auditd emits on the paths and syscalls you care about — a write to /etc/passwd should always leave a trace tagged with a key you can filter on.

# watch the identity files for write + attribute change
-w /etc/passwd  -p wa -k identity
-w /etc/shadow  -p wa -k identity
-w /etc/sudoers -p wa -k identity

Where this goes next

Emit each parsed event as one JSON line and you've built the bridge to everything downstream — ship it to Loki or Elasticsearch and the same 'who touched /etc/shadow' question becomes a saved dashboard query across your whole fleet, not one host at a time.

Related posts