Parsing

Firewall logs (KEY=VALUE pairs)

ufw.log and iptables logs are syslog; the pairs are in msg:

kelora examples/ufw_firewall.log --filter 'e.msg.contains("[UFW BLOCK]")' \
  -e 'e.absorb_kv("msg")' --freq SRC --freq DPT

DPT          (3 values):
  value  count
  22         2
  445        2
  5353       1
SRC          (3 values):
  value         count
  192.0.2.77        3
  203.0.113.5       1
  203.0.113.99      1

If your firewall log isn't recognized (-d shows only line), use line instead of msg.

key=value pairs inside messages

kelora examples/quickstart.log -f 'cols:ts(3) level *msg' --input-year 2024 -l error \
  -e 'e.absorb_kv("msg")' -J

{"ts":"Jan 15 10:00:15","level":"ERROR","msg":"Payment timeout","duration":"5s","gateway":"stripe","order":"1234"}
{"ts":"Jan 15 10:00:22","level":"ERROR","msg":"Gateway unreachable","host":"stripe.com"}
{"ts":"Jan 15 10:00:25","level":"ERROR","msg":"Rate limit exceeded","endpoint":"/charge","limit":"100","service":"payment-api","window":"60s"}
{"ts":"Jan 15 10:00:28","level":"ERROR","msg":"Authentication failed","ip":"192.168.1.50","reason":"invalid_token","user":"admin"}

Get Logs into Shape

Kubernetes container logs with JSON payloads

kelora pod.log -e 'e.absorb_json("msg")' — Kelora detects the CRI format, and absorb_json turns the JSON inside msg into fields. Get Logs into Shape

Keep only the JSON lines (or only the rest)

kelora examples/mixed_format.log -f json,line --filter 'e._format == "line"' -k line

line='Server starting up, plaintext logger active before JSON init'
line='Loading config from /etc/checkout/config.yml'
line='[legacy] 2024-01-15 10:00:04 ERROR gateway connection reset by peer'
line='Traceback (most recent call last):'
line='  File "worker.py", line 42, in process_order'

A custom format

Work through Get Logs into Shape: named format, cascade, cols:, regex:, then extraction in a script.