Scripting Stages¶

Deep dive into Kelora's Rhai scripting stages: --begin, --filter, --exec, and --end.

Overview¶

Kelora provides four scripting stages for transforming log data with Rhai scripts:

--begin  →  [Process Events]  →  --end
              ↓
           --filter + --exec (per event, in CLI order)

Stage	Runs	Purpose	Access
`--begin`	Once before processing	Initialize state, load data	`conf` map, file helpers
`--filter`	Once per event	Select events to keep/skip	`e` (event), `conf`, `meta`
`--exec`	Once per event	Transform events	`e` (event), `conf`, `meta`, tracking
`--end`	Once after processing	Summarize, report	`metrics`, `conf`

Note: --filter and --exec can be specified multiple times and execute in exact CLI order.

Begin Stage¶

Purpose¶

The --begin stage runs once before any events are processed. Use it to:

Initialize lookup tables
Load reference data from files
Set up shared configuration
Prepare the conf map for use in other stages

The `conf` Map¶

The global conf map is read-write in --begin and read-only in later stages.

kelora -j \
    --begin 'conf.valid_users = ["alice", "bob", "charlie"]' \
    --exec 'e.is_valid = e.user in conf.valid_users' \
    app.log

Available Helpers¶

Special functions available only in --begin:

`read_lines(path)`¶

Read file as array of strings (one per line, UTF-8).

kelora -j \
    --begin 'conf.blocked_ips = read_lines("blocked.txt")' \
    --exec 'if e.ip in conf.blocked_ips { e = () }' \
    app.log

`read_file(path)`¶

Read entire file as single string (UTF-8).

kelora -j \
    --begin 'conf.template = read_file("template.txt")' \
    --end 'print(conf.template.replace("{count}", metrics["total"].to_string()))' \
    app.log

`read_json(path)`¶

Parse JSON file (convenience helper).

kelora -j \
    --begin 'conf.users = read_json("users.json")' \
    --exec 'e.user_name = conf.users.get(e.user_id, "unknown")' \
    app.log

Examples¶

Load Lookup Table¶

kelora -j \
    --begin 'conf.services = #{api: "API Gateway", db: "Database", cache: "Redis"}' \
    --exec 'e.service_name = conf.services.get(e.service, e.service)' \
    app.log

Load IP Geolocation Data¶

kelora -j \
    --begin 'conf.ip_to_country = read_json("geoip.json")' \
    --exec 'e.country = conf.ip_to_country.get(e.ip, "unknown")' \
    app.log

Initialize Counters¶

kelora -j \
    --begin 'conf.start_time = now_utc()' \
    --end 'let duration = now_utc() - conf.start_time; print("Processed in " + duration + "s")' \
    app.log

Load Configuration¶

kelora -j \
    --begin 'conf.threshold = 1000; conf.alert_email = "ops@company.com"' \
    --exec 'if e.duration_ms > conf.threshold { eprint("⚠️  Slow request: " + e.path) }' \
    app.log

Filter Stage¶

Purpose¶

The --filter stage runs once per event to decide whether to keep or skip it. Use it to:

Select events matching specific criteria
Remove unwanted events (debug logs, health checks, etc.)
Combine multiple filter conditions in sequence
Control which events reach later --exec stages

Boolean Expressions¶

Filters must return true (keep event) or false (skip event):

kelora -j \
    --filter 'e.level == "ERROR"' \
    app.log

Behavior: - Returns true → Event passes to next stage - Returns false → Event is skipped (removed from pipeline) - Error in resilient mode → Treated as false, event skipped - Error in strict mode → Processing aborts

Access to Event Data¶

Filters have access to:

e - The current event (read-only)
conf - Configuration from --begin (read-only)
meta - Event metadata (line, line_num, filename)

kelora -j \
    --begin 'conf.min_duration = 1000' \
    --filter 'e.duration_ms > conf.min_duration' \
    app.log

Multiple Filters¶

Multiple --filter flags create an AND condition - events must pass all filters:

kelora -j \
    --filter 'e.level == "ERROR"' \
    --filter 'e.service == "api"' \
    --filter 'e.duration_ms > 1000' \
    app.log

Only events that are ERROR AND from api service AND slow will pass.

Common Patterns¶

Basic Field Matching¶

kelora -j --filter 'e.status >= 400' app.log
kelora -j --filter 'e.user == "admin"' app.log
kelora -j --filter 'e.service in ["api", "db"]' app.log

String Operations¶

kelora -j --filter 'e.message.contains("timeout")' app.log
kelora -j --filter 'e.path.starts_with("/api/")' app.log
kelora -j --filter 'e.level.to_upper() == "ERROR"' app.log

Regex Matching¶

kelora -j --filter 'e.message.has_matches(r"\d{3}-\d{3}-\d{4}")' app.log
kelora -j --filter 'e.ip.has_matches(r"^192\.168\.")' app.log

Existence Checks¶

kelora -j --filter 'e.contains("error")' app.log       # Has 'error' field
kelora -j --filter '"error" in e' app.log              # Same as above
kelora -j --filter 'e.contains("user_id")' app.log     # Has 'user_id' field

Complex Conditions¶

kelora -j \
    --filter '(e.status >= 500) || (e.status >= 400 && e.duration_ms > 5000)' \
    app.log

Using conf for Dynamic Filtering¶

kelora -j \
    --begin 'conf.blocked_ips = ["192.168.1.100", "10.0.0.50"]' \
    --filter '!(e.ip in conf.blocked_ips)' \
    app.log

Filtering with Metadata¶

# Only process events from specific file
kelora -j *.log --filter 'meta.filename == "production.log"'

# Skip first 100 lines
kelora -j app.log --filter 'meta.line_num > 100'

Filter vs --levels¶

For simple level filtering, --levels is more efficient than --filter:

Prefer:

kelora -j app.log --levels error,warn

Over:

kelora -j app.log --filter 'e.level in ["ERROR", "WARN"]'

However, --filter provides more flexibility for complex conditions.

Filter Output¶

Filters don't modify events - they only decide pass/skip:

# This works - filter just checks level
kelora -j --filter 'e.level == "ERROR"' app.log

# This does nothing - assignment in filter has no effect
kelora -j --filter 'e.level = "ERROR"' app.log  # Wrong!

To modify events, use --exec instead.

Exec Stage¶

Purpose¶

The --exec stage runs once per event. Use it to:

Transform event fields
Add computed fields
Filter events (via e = ())
Track metrics
Emit multiple events from arrays

The Event Variable¶

The current event is available as e. Modifications to e persist through subsequent --exec scripts.

kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'e.is_slow = e.duration_s > 1.0' \
    app.log

Multiple Exec Scripts¶

Multiple --exec scripts run in order. Each sees changes from previous scripts.

kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'track_avg("duration", e.duration_s)' \
    --exec 'if e.duration_s > 5.0 { e.alert = true }' \
    app.log

Execution order: 1. Convert duration_ms to duration_s 2. Track average duration 3. Add alert field for slow requests

Intermixing --filter and --exec¶

--filter and --exec are both script stages and execute in exact CLI order:

kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \    # Stage 1: Transform all events
    --filter 'e.duration_s > 1.0' \                    # Stage 2: Keep only slow events
    --exec 'track_count("slow")' \                     # Stage 3: Track (slow only)
    --exec 'e.alert = true' \                          # Stage 4: Add field (slow only)
    app.log

What happens: 1. First --exec adds duration_s field to all events 2. --filter removes events under 1.0s 3. Second --exec only processes slow events (tracks count) 4. Third --exec only processes slow events (adds alert field)

Later stages only see events that passed earlier filters. This allows precise control over which events are transformed or tracked.

Atomic Execution¶

In resilient mode (default), exec scripts execute atomically:

If an error occurs, changes are rolled back
Original event is returned unchanged
Processing continues with next event

kelora -j \
    --exec 'e.result = e.value.to_int() * 2' \
    app.log

If e.value is not a valid integer:

Error is recorded
Event passes through unchanged
No partial modifications

In strict mode (--strict), errors abort immediately.

Common Patterns¶

Transform Fields¶

kelora -j \
    --exec 'e.level = e.level.to_upper()' \
    --exec 'e.message = e.message.trim()' \
    app.log

Add Computed Fields¶

kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'e.timestamp_unix = e.timestamp.to_unix()' \
    app.log

Conditional Field Creation¶

kelora -j \
    --exec 'if e.status >= 500 { e.severity = "critical" } else if e.status >= 400 { e.severity = "error" }' \
    app.log

Remove Events¶

kelora -j \
    --exec 'if e.level == "DEBUG" { e = () }' \
    app.log

Track Metrics¶

kelora -j \
    --exec 'track_count(e.service)' \
    --exec 'track_avg("response_time", e.duration_ms)' \
    --metrics \
    app.log

Fan-Out Arrays¶

kelora -j \
    --exec 'emit_each(e.items)' \
    app.log

Each array element becomes a separate event.

Access to conf¶

The conf map from --begin is read-only in --exec:

kelora -j \
    --begin 'conf.multiplier = 2.5' \
    --exec 'e.adjusted = e.value * conf.multiplier' \
    app.log

Access to meta¶

The meta variable provides event metadata in --exec and --filter:

kelora -j \
    --exec 'e.source = meta.filename' \
    server1.log server2.log

Available metadata attributes:

meta.line - Original raw line from input (always available)
meta.line_num - Line number, 1-based (available when processing files)
meta.filename - Source filename (available with multiple files or explicit file arguments)

Multi-file tracking example:

kelora -j logs/*.log --metrics \
    --exec 'if e.level == "ERROR" { track_count(meta.filename) }' \
    --end 'for file in metrics.keys() { print(file + ": " + metrics[file] + " errors") }'

Debugging with line numbers:

kelora -j --filter 'e.status >= 500' \
    --exec 'eprint("⚠️  Server error at " + meta.filename + ":" + meta.line_num)' \
    app.log

Re-parsing with raw line:

kelora -j \
    --exec 'if e.message.contains("CUSTOM:") { e.custom = meta.line.after("CUSTOM:").parse_json() }' \
    app.log

End Stage¶

Purpose¶

The --end stage runs once after all events are processed. Use it to:

Summarize metrics
Generate reports
Print final statistics
Export aggregated data

The metrics Map¶

The global metrics map contains all tracked data from track_*() functions:

kelora -j \
    --exec 'track_count(e.service)' \
    --end 'for key in metrics.keys() { print(key + ": " + metrics[key]) }' \
    app.log

Available Data¶

In --end, you have access to:

metrics - All tracked metrics (counts, sums, averages, etc.)
conf - Read-only configuration from --begin
Standard Rhai functions (print, file helpers if --allow-fs-writes)

Examples¶

Print Summary Statistics¶

kelora -j \
    --exec 'track_count("total"); if e.level == "ERROR" { track_count("errors") }' \
    --end 'let error_rate = metrics.errors / metrics.total * 100; print("Error rate: " + error_rate + "%")' \
    app.log

Generate Report¶

kelora -j \
    --exec 'track_count(e.service)' \
    --end 'print("=== Service Report ==="); for svc in metrics.keys() { print(svc + ": " + metrics[svc] + " requests") }' \
    app.log

Export Metrics to File¶

kelora -j --allow-fs-writes \
    --exec 'track_count(e.service)' \
    --end 'append_file("report.txt", "Total services: " + metrics.len().to_string())' \
    app.log

Calculate Percentages¶

kelora -j \
    --exec 'track_count("total"); track_count(e.level)' \
    --end 'for level in ["INFO", "WARN", "ERROR"] { let pct = metrics.get(level, 0) / metrics.total * 100; print(level + ": " + pct + "%") }' \
    app.log

Stage Interaction¶

Data Flow Between Stages¶

--begin:  Initialize conf map
    ↓
    conf (read-only)
    ↓
--exec:   Process events, track metrics
    ↓
    metrics + conf (both read-only)
    ↓
--end:    Summarize and report

Complete Example¶

kelora -j \
    --begin 'conf.threshold = 1000; conf.start = now_utc()' \
    --exec 'if e.duration_ms > conf.threshold { track_count("slow") }' \
    --exec 'track_count("total")' \
    --end 'let elapsed = now_utc() - conf.start; print("Processed " + metrics.total + " events in " + elapsed + "s"); print("Slow requests: " + metrics.get("slow", 0))' \
    app.log

Flow: 1. --begin: Set threshold to 1000ms, record start time 2. --exec (per event): Track slow requests, track total 3. --end: Calculate elapsed time, print summary

Using Exec Files¶

`-E, --exec-file`¶

Load Rhai script from file for the exec stage:

transform.rhai:

// Convert duration to seconds
e.duration_s = e.duration_ms / 1000;

// Add severity based on status
if e.status >= 500 {
    e.severity = "critical";
} else if e.status >= 400 {
    e.severity = "error";
} else {
    e.severity = "ok";
}

// Track metrics
track_count(e.severity);
track_avg("response_time", e.duration_s);

Usage:

kelora -j -E transform.rhai --metrics app.log

`-I, --include`¶

Include Rhai library files before script stages:

helpers.rhai:

fn classify_status(status) {
    if status >= 500 {
        "server_error"
    } else if status >= 400 {
        "client_error"
    } else if status >= 300 {
        "redirect"
    } else if status >= 200 {
        "success"
    } else {
        "other"
    }
}

Usage:

kelora -j \
    -I helpers.rhai \
    --exec 'e.status_class = classify_status(e.status)' \
    app.log

Best Practices¶

Use --begin for Initialization¶

Good:

kelora -j \
    --begin 'conf.lookup = read_json("data.json")' \
    --exec 'e.name = conf.lookup.get(e.id, "unknown")' \
    app.log

Bad:

kelora -j \
    --exec 'let lookup = read_json("data.json"); e.name = lookup.get(e.id, "unknown")' \
    app.log

The bad example reads the file once per event (slow and wasteful).

Keep --exec Scripts Simple¶

Break complex logic into multiple --exec scripts:

Good:

kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'e.is_slow = e.duration_s > 1.0' \
    --exec 'if e.is_slow { track_count("slow_requests") }' \
    app.log

Bad:

kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000; e.is_slow = e.duration_s > 1.0; if e.is_slow { track_count("slow_requests") }' \
    app.log

The good example is easier to read and debug.

Use --end for Summaries¶

Good:

kelora -j \
    --exec 'track_count(e.service)' \
    --end 'print("Total services: " + metrics.len())' \
    app.log

Bad:

kelora -j \
    --exec 'track_count(e.service); print("Processing...")' \
    app.log

The bad example prints on every event (noisy).

Leverage File Helpers¶

For complex logic, use -E and -I:

kelora -j -I helpers.rhai -E transform.rhai --metrics app.log

This keeps command lines clean and logic maintainable.

Performance Considerations¶

Begin Stage Overhead¶

The --begin stage runs once, so file I/O here is acceptable:

kelora -j \
    --begin 'conf.large_dataset = read_json("10mb.json")' \
    --exec 'e.enriched = conf.large_dataset.get(e.id, #{})' \
    app.log

Exec Stage Optimization¶

The --exec stage runs per event. Avoid expensive operations:

Slow:

kelora -j \
    --exec 'let lookup = read_json("data.json"); e.name = lookup.get(e.id, "unknown")' \
    app.log

Fast:

kelora -j \
    --begin 'conf.lookup = read_json("data.json")' \
    --exec 'e.name = conf.lookup.get(e.id, "unknown")' \
    app.log

End Stage Overhead¶

The --end stage runs once, so complex calculations are fine:

kelora -j \
    --exec 'track_count(e.service)' \
    --end 'let sorted = metrics.keys().sort(); for key in sorted { print(key + ": " + metrics[key]) }' \
    app.log

Parallel Processing¶

When using --parallel, scripting stages behave differently:

Begin and End¶

--begin and --end run once (not parallelized):

kelora -j --parallel \
    --begin 'conf.start = now_utc()' \
    --exec 'track_count(e.service)' \
    --end 'print("Duration: " + (now_utc() - conf.start))' \
    app.log

Exec Stage¶

--exec runs in parallel across worker threads:

Each thread has its own copy of conf (read-only)
Tracking functions aggregate across threads
Event modifications are isolated per thread

kelora -j --parallel \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'track_count(e.service)' \
    app.log

Thread Safety¶

Kelora handles thread safety automatically:

conf is cloned per thread (immutable)
metrics uses thread-safe aggregation
Event modifications are isolated

You don't need to worry about race conditions in scripts.

Troubleshooting¶

conf is Read-Only in --exec¶

Problem:

kelora -j --exec 'conf.value = 42' app.log
# Error: conf is read-only in exec stage

Solution: Initialize in --begin:

kelora -j --begin 'conf.value = 42' --exec 'e.result = conf.value * 2' app.log

metrics Not Available in --exec¶

Problem:

kelora -j --exec 'print(metrics["total"])' app.log
# Error: metrics not available in exec stage

Solution: Use --end:

kelora -j --exec 'track_count("total")' --end 'print(metrics["total"])' app.log

File Helpers Not Working¶

Problem:

kelora -j --exec 'append_file("out.txt", e.message)' app.log
# Error: filesystem writes not allowed

Solution: Add --allow-fs-writes:

kelora -j --allow-fs-writes --exec 'append_file("out.txt", e.message)' app.log

Script Stage Ordering¶

Understanding: --filter and --exec execute in exact CLI order, intermixed.

# Both --exec scripts run on all events
kelora -j --exec 'e.a = 1' --exec 'e.b = e.a + 1' app.log

# Filter runs between execs - second --exec only sees filtered events
kelora -j --exec 'e.a = 1' --filter 'e.level == "ERROR"' --exec 'e.b = e.a + 1' app.log

What happens in the second example: 1. First --exec adds field a=1 to all events 2. --filter removes non-ERROR events 3. Second --exec adds field b=2 to ERROR events only

This is expected behavior - later stages only process events that passed earlier filters. Each stage (filter or exec) processes the output of the previous stage sequentially.

Scripting Stages¶

Overview¶

Begin Stage¶

Purpose¶

The conf Map¶

Available Helpers¶

read_lines(path)¶

read_file(path)¶

read_json(path)¶

Examples¶

Load Lookup Table¶

Load IP Geolocation Data¶

Initialize Counters¶

Load Configuration¶

Filter Stage¶

Purpose¶

Boolean Expressions¶

Access to Event Data¶

Multiple Filters¶

Common Patterns¶

Basic Field Matching¶

String Operations¶

Regex Matching¶

Existence Checks¶

Complex Conditions¶

Using conf for Dynamic Filtering¶

Filtering with Metadata¶

Filter vs --levels¶

Filter Output¶

Exec Stage¶

Purpose¶

The Event Variable¶

Multiple Exec Scripts¶

Intermixing --filter and --exec¶

Atomic Execution¶

Common Patterns¶

Transform Fields¶

Add Computed Fields¶

Conditional Field Creation¶

Remove Events¶

Track Metrics¶

Fan-Out Arrays¶

Access to conf¶

Access to meta¶

End Stage¶

Purpose¶

The metrics Map¶

Available Data¶

Examples¶

Print Summary Statistics¶

Generate Report¶

Export Metrics to File¶

Calculate Percentages¶

Stage Interaction¶

Data Flow Between Stages¶

Complete Example¶

Using Exec Files¶

-E, --exec-file¶

-I, --include¶

Best Practices¶

Use --begin for Initialization¶

Keep --exec Scripts Simple¶

Use --end for Summaries¶

Leverage File Helpers¶

Performance Considerations¶

Begin Stage Overhead¶

Exec Stage Optimization¶

End Stage Overhead¶

Parallel Processing¶

Begin and End¶

Exec Stage¶

Thread Safety¶

Troubleshooting¶

conf is Read-Only in --exec¶

metrics Not Available in --exec¶

File Helpers Not Working¶

Script Stage Ordering¶

See Also¶

The `conf` Map¶

`read_lines(path)`¶

`read_file(path)`¶

`read_json(path)`¶

`-E, --exec-file`¶

`-I, --include`¶