Errors and Incidents¶

Which services fail most?¶

kelora examples/simple_json.jsonl -l error,critical --freq service

service      (4 values):
  value     count
  api           1
  auth          1
  database      1
  disk          1

Summarize

What kinds of errors are there?¶

Group messages that differ only in IDs, IPs, or numbers:

kelora examples/production-errors.jsonl --drain -k message

templates (3 items):
  4: Failed to connect to <ipv4>
  3: Timeout on request <uuid>
  3: User <email> sent invalid request

To keep the pattern as a field for further filtering, use normalized(), which replaces IPs, emails, UUIDs, and the like with placeholders:

kelora examples/production-errors.jsonl -m -e 'track_freq("pattern", e.message.normalized())'

pattern      (3 values):
  value                              count
  Failed to connect to <ipv4>            4
  Timeout on request <uuid>              3
  User <email> sent invalid request      3

Summarize

When did it start?¶

Count errors per time window; the first busy window is your start. For the exact first and last error, -s prints the output time span.

kelora examples/api_latency_incident.jsonl -l error --span 5m --span-summary

2025-01-20T14:00:00Z  events=0
2025-01-20T14:05:00Z  events=0
2025-01-20T14:10:00Z  events=0
2025-01-20T14:15:00Z  events=0
2025-01-20T14:20:00Z  events=17
2025-01-20T14:25:00Z  events=0
2025-01-20T14:30:00Z  events=0
2025-01-20T14:35:00Z  events=0

Group into Spans

What changed after the deploy?¶

Split the log at a time and compare its message patterns before and after:

kelora examples/shop.log --drain-diff --cut-at '2026-10-06 14:15' -k msg

--- examples/shop.log before 2026-10-06T14:15:00Z  61 events  2026-10-06T14:00:23Z .. 2026-10-06T14:14:40Z
+++ examples/shop.log from 2026-10-06T14:15:00Z  106 events  2026-10-06T14:15:03Z .. 2026-10-06T14:40:03Z

+        17  payment for order <num> failed: timeout after <duration> (provider…
+         3  retrying payment for order <num> (attempt <num>)
* 2.1× less  order <num> placed in <duration>

3 templates unchanged in frequency | field: msg
3 of them changed a little, but 61 and 106 events are too few to tell that from random variation

+ marks a pattern that appeared, - one that disappeared, * one whose rate changed. To split at the deploy message instead, use --cut-before 'e.msg.contains("deploy")'; for two files, kelora --drain-diff before.log after.log -k msg. Summarize

Which exceptions do the stack traces contain?¶

Join each trace into one event, then extract the exception class:

kelora examples/stacktrace_java.log -f line -M java \
  -e 'e.exception = e.line.extract_regex("([\\w.]+(?:Exception|Error)):", 1).or_empty()' \
  --freq exception

exception    (2 values):
  value                                     count
  java.lang.OutOfMemoryError                    1
  org.springframework.dao.DataAccessResourceFailureException      1

-M java suits output without timestamps; for timestamped logs use -M timestamp. Get Logs into Shape

When did each exception first appear?¶

Remember the first timestamp per value in state, count with track_freq, and print both at the end:

kelora examples/multiline_stacktrace.log -M timestamp -q \
  -e 'let x = e.msg.extract_regex("([\\w.]+(?:Exception|Error)):", 1);
      if x != "" { track_freq("n", x); if !state.contains(x) { state[x] = e.ts } }' \
  --end 'for k in state.keys() { print(`${k}  first: ${state[k]}  count: ${metrics.n[k]}`) }'

RuntimeError  first: 2024-01-15 10:03:00  count: 1
ValueError  first: 2024-01-15 10:01:00  count: 1
java.sql.SQLException  first: 2024-01-15 10:02:00  count: 1

Cross-Event Logic

Show the events around each error¶

kelora app.log -l error -B 2 -A 2 — like grep -B/-A, with each event's role marked. Filter

One timeline from several services¶

kelora --merge-sorted examples/merge_api.jsonl examples/merge_worker.jsonl -l error,warn -k ts,service,msg

ts='2024-04-09T09:41:02Z' service='api' msg='upstream latency rising'
ts='2024-04-09T09:41:03Z' service='worker' msg='retry queue depth=120'
ts='2024-04-09T09:41:04Z' service='api' msg='database timeout'
ts='2024-04-09T09:41:06Z' service='worker' msg='lost connection to database'
ts='2024-04-09T09:41:07Z' service='api' msg='request failed: 503'

Each file must be in time order. Big Files, Many Files