Transform with Scripts¶

When options run out, --filter and --exec (-e) run small Rhai scripts on each event. Rhai reads like a mix of JavaScript and Rust; this page covers what log work needs. The Rhai cheatsheet has the full syntax, and kelora --help-functions KEYWORD searches the 150+ built-in functions.

kelora examples/simple_json.jsonl -l error \
  -e 'e.duration_s = e.get("duration_ms", 0) / 1000.0; e.service = e.service.to_upper()' \
  -k timestamp,service,duration_s,message

timestamp='2024-01-15T10:01:30Z' service='DATABASE' duration_s=5.0
  message='Query timeout'
timestamp='2024-01-15T10:03:30Z' service='AUTH' duration_s=0.0
  message='Account locked'
timestamp='2024-01-15T10:16:00Z' service='API' duration_s=0.0
  message='Service unavailable'

The event, `e`¶

Each event is a map of fields, available as e.

Do	Write
read a field	`e.status`, `e["user-agent"]` (names with `-`, `@`, `.`)
read nested data	`e.user.id`, `e.tags[0]`, `e.tags[-1]`
add or change a field	`e.duration_s = e.duration_ms / 1000.0`
remove a field	`e.password = ()`
drop the whole event	`e = ()`
copy into a new field	`e.path_only = e.path.split("?")[0]`
rename a field	`e.rename_field("ts", "timestamp")`

() is Rhai's "nothing", like null elsewhere: assigning it removes a field, and reading a missing field gives it.

Alongside e, scripts see meta: meta.line (the raw input line), meta.line_num, meta.filename, and meta.parsed_ts (the parsed timestamp, see Work with Time). The script variables reference lists everything.

Types¶

Most parsers give you strings. Only JSON keeps its types and logfmt detects numbers and booleans; other formats type at most a few known fields (status, bytes, pid), and csv, cols: and regex: fields need annotations. -d shows what each field holds; in this CSV file, even status is a string:

kelora examples/simple_csv.csv -d

Field        Type    Seen  Miss  Uniq  Examples
timestamp    string    25    0%    25  "2024-01-15T10:01:05Z", "2024-01-15T10:0…
method       string    25    0%     5  "GET", "POST", "PUT", "DELETE", "PATCH"
path         string    25    0%    22  "/api/health", "/api/users", "/api/stats…
status       string    25    0%     7  "200", "404", "201", "204", "401", "413"…
bytes        string    25    0%    17  "7890", "567", "234", "15678", "456", "1…
duration_ms  string    25    0%    17  "45", "98", "123", "34", "89", "234", "7…

25 events scanned · format: csv (auto-detected) · timestamp: timestamp → meta.parsed_ts

Comparing a string with a number never matches ("500" > 400 is false, with no error), so convert before you compare or do arithmetic:

Situation	Do
`csv`, `tsv`, `cols:`, `regex:`	annotate in the format spec: `-f 'csv status:int bytes:int'`, `cols:status:int`, `(?P<ms:float>...)` (how)
any other format, or a field you extracted yourself	convert once in an early `-e` and overwrite the field, so later stages see a number
values that may be junk (`"-"`, `""`)	`to_int_or(0)` / `to_float_or(0.0)` give a default; `to_int()` / `to_float()` / `to_bool()` give `()`
thousands separators (`"1,234"`)	`to_int(",")`; `to_float(",", ".")` also sets the decimal mark

kelora examples/simple_csv.csv -e 'e.status = e.status.to_int()' --filter 'e.status >= 500' -k path,status

path='/api/users' status=500

Integer division truncates: e.ms / 1000 is an integer, e.ms / 1000.0 a float.

Missing fields¶

Events in the same file often have different fields. A missing field reads as () — accessing it never fails by itself. What happens next depends on the operation:

Expression, when `dur` is missing	Result
`e.dur == "x"`, `e.dur > 5`	`false` — comparisons accept `()`
`e.dur + "ms"`	`"ms"` — string concatenation accepts `()`
`e.dur + 1`	error — no arithmetic on `()`
`e.dur.to_upper()`	error — no methods on `()`
`e.user.role`, when `user` is missing	error — can't look inside `()`

Guard first, or read with a default:

if e.has("dur") { e.dur_s = e.dur / 1000.0 }   // guard first
e.dur_s = e.get("dur", 0) / 1000.0              // or read with a default
e.role = e.get_path("user.role", "guest")       // same for nested paths

When a script fails on an event, Kelora rolls that --exec back (the event keeps its previous fields), counts the error, and continues. The summary on stderr names the line and the problem; --strict stops at the first error instead. See errors.

Strings¶

Want	Use
test for text	`.contains("x")`, `.starts_with("x")`, `.matches("re")`
change case	`.to_lower()`, `.to_upper()`
cut around delimiters	`.before(" ")`, `.after("=")`, `.between("[", "]")`
split	`.split(",")` (array), `.col(2)` (third whitespace column)
regex capture	`.extract_regex("id=(\\d+)", 1)`
replace	`.replace_regex("\\d+", "N")`
trim	`.strip()`, `.clip()` (also strips punctuation)
length	`.len()`

trim and replace return nothing

Rhai's built-in trim() and replace() change a string variable in place and return (), so e.y = e.x.trim() leaves y unset. Use strip() and replace_regex(), which return the new string.

before, after, between, and extract_regex return "" when there is nothing to return; add .or_empty() to get () instead, which skips the assignment: e.user = e.msg.after("user=").or_empty() creates user only on lines that have one. Test such a value with != (), not != "". Get Logs into Shape shows the extraction functions at work.

Logic¶

let ms = e.get("duration_ms", 0);                 // local variable
e.speed = if ms > 1000 { "slow" } else { "ok" };  // if is an expression
if e.status >= 500 { e.alert = true; } else if e.status == 404 { e.alert = false; }
e.class = switch e.status / 100 { 2 => "ok", 4 => "client", 5 => "server", _ => "other" };
for tag in e.tags { if tag == "pii" { e.redact = true; } }

Separate statements with ;. Strings take double quotes, so wrap the whole script in single quotes on the shell; #"…"# is a raw string, handy for regexes with backslashes.

Stages run in order¶

Every --filter and --exec is a stage, and stages run in the order you write them. Field changes carry from one stage to the next; let variables don't — each stage has its own scope:

kelora app.jsonl \
  -e 'e.ms = e.duration_ms.to_int_or(0)' \
  --filter 'e.ms > 1000' \
  -e 'e.secs = e.ms / 1000.0'       # sees e.ms, only for events that passed the filter

Put statements that share a let in one --exec. Splitting work across several stages has one advantage: if a later stage fails, the fields set by earlier stages survive.

Nested JSON¶

get_path and has_path walk dotted paths with array indexes, returning a default instead of failing:

kelora examples/fan_out_batches.jsonl \
  -e 'e.first_sku = e.get_path("orders[0].items[0].sku", "none"); e.n_orders = e.get("orders", []).len()' \
  -k batch_id,n_orders,first_sku

batch_id='batch_001' n_orders=2 first_sku='A001'
batch_id='batch_002' n_orders=1 first_sku='C001'
batch_id='batch_003' n_orders=3 first_sku='D001'
batch_id='batch_004' n_orders=0 first_sku='none'
batch_id='batch_005' n_orders=0 first_sku='none'

-k selects top-level fields only, so copy nested values up first, as above. e.flattened() returns the whole event as one level of keys like orders[0].order_id, ready for CSV export.

One event per array element¶

emit_each(array) replaces the current event with one event per element. The optional second argument adds fields to each, typically the parent's IDs:

kelora examples/fan_out_batches.jsonl \
  -e 'emit_each(e.orders, #{batch_id: e.batch_id})' \
  -e 'emit_each(e.items, #{batch_id: e.batch_id, order_id: e.order_id})' \
  -k batch_id,order_id,sku,qty,price -n 4

batch_id='batch_001' order_id='ord_1' sku='A001' qty=2 price=10.0
batch_id='batch_001' order_id='ord_1' sku='A002' qty=1 price=20.0
batch_id='batch_001' order_id='ord_2' sku='B001' qty=3 price=15.0
batch_id='batch_002' order_id='ord_3' sku='C001' qty=1 price=50.0

Each emit_each fans out one level, and later stages see the new events.

Lookups¶

--begin runs once before any event; whatever it stores in conf is read-only afterwards. Use it to load lookup tables:

kelora examples/simple_json.jsonl -l error,critical \
  --begin 'conf.owner = #{database: "dba-team", auth: "identity", disk: "infra"}' \
  -e 'e.owner = conf.owner.get(e.service) ?? "unassigned"' \
  -k service,owner,message

service='database' owner='dba-team' message='Query timeout'
service='auth' owner='identity' message='Account locked'
service='disk' owner='infra' message='Disk space critical'
service='api' owner='unassigned' message='Service unavailable'

In --begin, read_lines("blocked.txt") returns a file's lines as an array and read_file("map.json").parse_json() loads a JSON map. Memory across events (counters, deduplication, sessions) is different: see Cross-Event Logic.

KELORA_SECRET=team-key kelora examples/web_access_large.log.gz \
  -e 'e.ip = e.ip.mask_ip(2); if e.has("user") { e.user = pseudonym(e.user, "users") }' \
  -k ip,user,path -n 3

ip='120.180.0.0' user='dAfZqbdcoKyc5q7mDnSGQjGu'
  path='/reintermediate/e-commerce/cross-platform'
ip='83.14.0.0' user='QEk-iR9bGmeyLqtP280tHEsT'
  path='/infrastructures/customized/portals/enhance'
ip='233.208.0.0' user='mKIdGhJ2q_jd4zTOTDi-08gU'
  path='/enterprise/iterate/integrate/exploit'

mask_ip(n) zeroes the last n parts of an address. pseudonym(value, domain) replaces a value with a keyed alias: the same input and KELORA_SECRET always give the same alias, so you can still count and join on it. Keep the secret stable and private — without it, Kelora uses a random key and the aliases change on every run. hash() gives a plain SHA-256 digest, and normalized() replaces IPs, emails, UUIDs and the like inside free text. The cookbook has complete recipes.

Reuse scripts¶

Option	Use for
`-E transform.rhai`	an `--exec` stage read from a file
`-I helpers.rhai`	function definitions for the next stage: `-I helpers.rhai --filter 'is_noise(e)'`
`--save-alias NAME`	save the whole command line; rerun with `-a NAME` (Configuration)

Debug a script¶

-D profiles the fields as your script leaves them, with their types (-d shows them as parsed, before any script runs); -F inspect shows each event's fields and types.
-n 5 while you iterate.
print(...) and eprint(...) write to stdout/stderr from inside a script.
-v prints every error with its line; --strict stops at the first one.
--assert 'e.has("user_id")' reports events that break an expectation without filtering them.