Roll Up Logs with Span Windows¶

Batch events into fixed-size or time-based windows using --span so you can compute summaries without losing streaming behavior.

When to Use Spans¶

Produce per-minute or per-N-event digests during incident reviews.
Generate lightweight dashboards straight from log streams.
Detect bursts or trends without shipping data into another analytics system.

Before You Start¶

Examples use examples/simple_json.jsonl and assume a parsed timestamp (timestamp field). Adapt field names as needed.
--span disables parallel mode; span processing requires ordered input.
Late or timestamp-free events require special handling (see Step 4).

Step 1: Select a Window Strategy¶

Count-based: --span 500 closes the window every 500 events that pass all filters.
Time-based: --span 5m groups events by wall-clock duration (supports s, m, h).
Optional --span-close '...Rhai script...' runs when the window closes. Use it for custom summaries or output.

Step 2: Count-Based Example¶

Summarise every 200 events for quick batch metrics.

kelora -j examples/simple_json.jsonl \
  --span 200 \
  --span-close '
    let metrics = span.metrics;
    print(`${span.id},events=${span.size},errors=${metrics.get_path("level|ERROR", 0)}`);
  ' \
  --exec '
    track_count("total");
    track_count("level|" + e.level);
  '

span.id is an incrementing identifier (span-000001, ...).
span.metrics exposes only the deltas collected within that window.
Because the original events are still streamed, you can attach more --exec or --filter stages before or after the span logic.

Step 3: Time-Based Example¶

Roll up five-minute error summaries aligned to timestamps.

kelora -j examples/simple_json.jsonl \
  --span 5m \
  --exec '
    track_count("total");
    track_count("level|" + e.level);
  ' \
  --span-close '
    let total = span.metrics.get_path("total", 0);
    let errors = span.metrics.get_path("level|ERROR", 0);
    if total > 0 {
      let rate = (errors.to_float() / total.to_float()) * 100.0;
      print(`${span.start},total=${total},errors=${errors},error_rate=${rate}`);
    }
  '

span.start and span.end use DateTime values from event timestamps.
Windows are tumbling; events fall into the window that matches their timestamp.
Ensure input is chronologically sorted to avoid unnecessary late events.

Step 4: Handle Late or Missing Timestamps¶

Inspect span metadata to see how events were classified.

kelora -j examples/simple_json.jsonl \
  --span 1m \
  --exec '
    if meta.span_status == "late" {
      eprint(`Late event: ${e.timestamp}`);
    } else if meta.span_status == "unassigned" {
      eprint(`Missing timestamp on line ${meta.line_num}`);
    }
  ' \
  --span-close 'print(span.id + ": " + span.size.to_string())'

meta.span_status is one of active, late, or unassigned.
Use --strict if missing timestamps should abort the run instead of being skipped.

Step 5: Export or Chain Spans¶

Write per-span summaries to files or feed them into downstream commands.

kelora -j examples/simple_json.jsonl \
  --span 10m \
  --exec 'track_count(e.service)' \
  --span-close '
    let path = "/tmp/span-" + span.id + ".csv";
    for (service, count) in span.metrics {
      append_file(path, `${span.start},${service},${count}\n`);
    }
  ' \
  --allow-fs-writes

Remember to enable --allow-fs-writes when using append_file().
Alternatively, pipe Kelora output into jq, DuckDB, or spreadsheets for visualisation.

Variations¶

Per-service spans

kelora -j app.log \
  --filter 'e.service == "payments"' \
  --span 100 \
  --span-close 'print(span.id + ",payments=" + span.size.to_string())'

Sliding error thresholds
Combine --span with Design Streaming Alerts to trigger notifications when span metrics cross a threshold.
Hybrid reporting
Run two spans simultaneously by invoking Kelora twice: one count-based for rapid feedback, another time-based for dashboards.

Best Practices¶

Sort archives by timestamp before running time-based spans to minimise late events.
Keep span window sizes reasonable; extremely large spans consume more memory.
Capture --stats output alongside span summaries for auditing.