Observable scripts: exit codes, logs, metrics & idempotency

Meaningful exit codes, structured logs, --dry-run, idempotency and audit trails.

Automation runs unattended, so it has to explain itself when something goes wrong — and prove it did the right thing when it went right. That means meaningful exit codes, structured logs a pipeline can index, a --dry-run mode people trust, idempotency so a re-run is safe, and an audit trail for anything that changes state. Observability is what turns a black-box script into an operable one.

Exit codes and structured logs

The exit code is the one signal every caller reads: 0 for success and distinct non-zero codes for distinct failures, documented so CI and wrappers can branch. Logs are the human- and machine-facing narrative: emit them as structured JSON to stderr with the context that matters (what target, what action, what outcome), keep stdout for the tool’s real output, and set levels so routine runs are quiet and problems are loud.

import sys, logging
 
EXIT_OK, EXIT_ERROR, EXIT_USAGE, EXIT_PARTIAL = 0, 1, 2, 3
 
def main(argv) -> int:
    try:
        args = parse(argv)
    except UsageError as e:
        log.error("bad usage", extra={"context": {"detail": str(e)}})
        return EXIT_USAGE
    failures = run(args)
    if failures and len(failures) < args.total:
        return EXIT_PARTIAL          # some succeeded — callers may retry the rest
    return EXIT_ERROR if failures else EXIT_OK
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))     # the return code IS the API

Dry-run and idempotency

Anything that changes state should support --dry-run: compute and print exactly what would happen, change nothing, exit. It is how people gain the confidence to run your tool in production. Idempotency is the companion property — running the tool twice leaves the system in the same state as running it once, because each action checks current state before acting. Together they make automation safe to re-run after a partial failure.

def ensure_rule(client, rule, dry_run: bool) -> str:
    current = client.get_rule(rule.id)
    if current == rule:
        return "unchanged"           # already correct -> idempotent no-op
    action = "create" if current is None else "update"
    if dry_run:
        log.info("would %s", action, extra={"context": {"rule": rule.id}})
        return f"dry-run:{action}"
    client.apply_rule(rule)          # safe to run again: it re-checks first
    log.info(action + "d", extra={"context": {"rule": rule.id}})
    return action

Metrics and audit trails

For recurring jobs, emit metrics so trends are visible: a Prometheus textfile the node exporter scrapes, or a pushgateway for short-lived runs — success/failure, duration, counts. For anything that changes state or touches secrets, write an audit record: who ran it, when, against what, with what outcome. That trail is what an incident review and a compliance auditor both ask for first.

# Prometheus textfile the node exporter picks up
cat > /var/lib/node_exporter/sec_rotate.prom <<EOF
# HELP sec_rotate_last_success_timestamp_seconds Last successful run.
# TYPE sec_rotate_last_success_timestamp_seconds gauge
sec_rotate_last_success_timestamp_seconds $(date +%s)
sec_rotate_rotated_total ${rotated_count}
EOF
 
# append-only audit record (structured, one JSON object per line)
printf '%s\n' "$(jq -nc --arg who "$USER" --arg tgt "$target" \
  --arg ts "$(date -Is)" '{ts:$ts, actor:$who, action:"rotate", target:$tgt, result:"ok"}')" \
  >> /var/log/sec-tools/audit.jsonl