Get Logs into Shape¶
Parsing has two jobs: split each line into named fields, and find a correct timestamp. Filters, scripts, and output work on fields; time ranges, windows, and ordering work on the timestamp — and a wrong timestamp fails silently.
For the fields, try these approaches in order and stop at the first one that works:
| Step | Situation | What to use |
|---|---|---|
| 1 | The format is common | nothing — auto-detection |
| 2 | You know the format, or detection guessed wrong | -f json, -f logfmt, -f syslog, … |
| 3 | One file mixes formats | cascade: -f json,line |
| 4 | Whitespace-separated columns | -f 'cols:ts(2) level *msg' |
| 5 | Anything with a pattern | -f 'regex:(?P<ts>\S+) …' |
| 6 | Fields are buried inside a text field | absorb_kv(), extract_regex(), … in --exec |
Then check the timestamp. If one event spans several lines, join them first.
How to tell whether parsing worked¶
Parsed events show several named fields. An unparsed event has a single line
field holding the whole line, and Kelora prints a hint:
💡 No input format detected; keeping whole lines as 'line'. The sampled lines look like 'timestamp LEVEL message' with a 2-token timestamp; extract fields with -f 'cols:ts(2) level *msg' (or a regex:). Mixed file? Cascade with repeated -f, e.g. -f json -f 'cols:ts(2) level *msg'. See --help-formats.
line='2024-01-15 10:00:00 [INFO] Server starting on port 8080'
line='2024-01-15 10:00:01 [DEBUG] Loading plugins from /etc/plugins'
For a field-by-field profile with types and sample values, use --discover
(-d); its last line also names the timestamp field. Use -v to see which
format was detected.
Split lines into fields¶
1. Let auto-detection do it¶
With no -f, Kelora samples the input and picks a parser. It recognizes JSON,
syslog, CEF, Apache/Nginx access logs, Kubernetes CRI, logfmt, CSV/TSV, and a
set of application-log layouts (log4j, python-logging, glog,
nginx-error, postgres, iso8601-level, …):
🔹 Auto-detected format: iso8601-level (from first 8 lines)
ts='2025-01-15 10:00:00' level='INFO' msg='Application started on :8080'
ts='2025-01-15 10:00:05' level='INFO' msg='Connected to database db-primary'
Detection runs once per run (on files, it samples the head plus a few probes deeper in). If a file mixes a dominant format with stray text lines, it switches to a cascade automatically (see 3 below). On stdin, it uses the first line only.
2. Name the format¶
Pass -f when detection guesses wrong, or in scripts, so a future detection
change can't surprise you. -j is short for -f json.
timestamp='2024-01-15T10:00:00Z' level='info' msg='Server started' service='web'
port=8080
timestamp='2024-01-15T10:00:05Z' level='debug' msg='Loading middleware'
service='web' count=5
Most formats keep their natural types: JSON numbers stay numbers, and logfmt values that look numeric become numbers. CSV/TSV values are strings unless you annotate them:
Format reference lists every format and the fields
it produces; kelora --help-formats prints the same in the terminal.
3. Mixed files: try several parsers per line¶
A comma-separated list is a cascade: each line goes to the first parser
that accepts it, and the event records the winner in _format. Typical case:
JSON logs interrupted by plain-text banners and stack traces.
_format='line'
line='Server starting up, plaintext logger active before JSON init'
_format='line' line='Loading config from /etc/checkout/config.yml'
_format='json' level='INFO' msg='Order 4412 captured for user alice'
_format='json' level='WARN' msg='Retrying upstream auth.svc after HTTP 503'
_format='line'
line='[legacy] 2024-01-15 10:00:04 ERROR gateway connection reset by peer'
Put strict parsers first and catch-alls (line, raw) last. cols: and
regex: can't go in a comma list (their specs contain commas) — repeat -f
instead:
cols: accepts any line, so it can only be last; a regex: declines lines it
doesn't match and can sit anywhere.
Different formats in different files (JSON from one service, logfmt from
another)? Use -f auto-per-file to detect each file separately.
4. Columns: cols:¶
For logs whose fields are separated by whitespace, describe the columns in order:
2024-01-15 10:00:00 INFO server Application started successfully
2024-01-15 10:00:05 DEBUG config Loading configuration from file
2024-01-15 10:00:10 WARN memory Memory usage at 85% threshold
ts='2024-01-15 10:00:00' level='INFO' msg='Application started successfully'
service='server'
ts='2024-01-15 10:00:05' level='DEBUG' msg='Loading configuration from file'
service='config'
ts='2024-01-15 10:00:10' level='WARN' msg='Memory usage at 85% threshold'
service='memory'
| Token | Meaning |
|---|---|
name |
one column |
name(3) |
three columns, joined (e.g. syslog-style Jan 15 10:00:00) |
- / -(2) |
skip one / two columns |
*name |
the rest of the line (must be last) |
name:int |
convert: int, float, bool |
Count the timestamp's whitespace tokens: ts(2) for 2024-01-15 10:00:00,
ts(3) for Jan 15 10:00:00. The no-format hint names the count when the
sampled lines agree on one. Use --cols-sep '|' for a different separator.
5. Patterns: regex:¶
When columns aren't enough — brackets, quotes, key= prefixes — write a regex
with named groups. Each group becomes a field; :int, :float, :bool
convert it.
2025-01-15T10:00:00Z [ERROR] code=500 duration=123.45ms msg="Database connection failed"
2025-01-15T10:00:05Z [INFO] code=200 duration=5.67ms msg="User login successful"
2025-01-15T10:00:10Z [WARN] code=429 duration=1.23ms msg="Rate limit exceeded"
kelora examples/regex_custom_format.log \
-f 'regex:(?P<ts>\S+) \[(?P<level>\w+)\] code=(?P<code:int>\d+) duration=(?P<ms:float>[\d.]+)ms msg="(?P<msg>[^"]*)"' \
-n 3
ts='2025-01-15T10:00:00Z' level='ERROR' msg='Database connection failed'
code=500 ms=123.45
ts='2025-01-15T10:00:05Z' level='INFO' msg='User login successful' code=200
ms=5.67
ts='2025-01-15T10:00:10Z' level='WARN' msg='Rate limit exceeded' code=429
ms=1.23
- The pattern must match the whole line (it is anchored automatically). Lines that don't match are counted as parse errors and skipped — or, in a cascade, handed to the next parser.
- Unnamed groups
(?:…)are allowed for structure; only named groups become fields. .also matches newlines, so a trailing(?P<msg>.*)captures a whole multi-line event.
kelora --help-regex has more patterns and common mistakes.
6. Finish the job in a script¶
Often the line has a clean prefix and a messy rest: free text with
key=value pairs, embedded JSON, a bracketed host. Parse the prefix, then pull
fields out of the remainder in --exec.
2024-06-19T12:31:52Z kube-controller host=cp-1 level=INFO action=rollout ns=payments detail="Scaling replicas"
2024-06-19T12:32:07Z kubelet[worker-2] level=ERROR Back-off restarting failed container
2024-06-19T12:32:08Z kubelet[worker-2] level=ERROR user=system:node detail="CrashLoopBackOff" pod=checkout-v2
2024-06-19T12:32:15Z kubelet[worker-3] level=WARN Container image not ready
2024-06-19T12:32:20Z kube-controller host=cp-1 level=INFO action=deploy ns=payments detail="Rollout progressing"
2024-06-19T12:32:30Z kubelet[worker-2] level=ERROR detail="Failed to pull image" pod=checkout-v2
The prefix is timestamp, source, and an optional [node]; the rest mixes
logfmt pairs, quoted values, and free text. A regex takes the prefix, and two
absorb_* calls handle the rest — absorb_logfmt for lines that are pure
logfmt, absorb_kv (which keeps the leftover text in msg) for the others:
kelora examples/incident_story.log \
-f 'regex:(?P<ts>\S+) (?P<source>[^\[ ]+)(?:\[(?P<node>[^\]]+)\])? (?P<msg>.*)' \
-e 'if e.absorb_logfmt("msg").status != "applied" { e.absorb_kv("msg") }'
ts='2024-06-19T12:31:52Z' level='INFO' source='kube-controller' action='rollout'
detail='Scaling replicas' host='cp-1' ns='payments'
ts='2024-06-19T12:32:07Z' level='ERROR'
msg='Back-off restarting failed container' source='kubelet' node='worker-2'
ts='2024-06-19T12:32:08Z' level='ERROR' source='kubelet' node='worker-2'
detail='CrashLoopBackOff' pod='checkout-v2' user='system:node'
ts='2024-06-19T12:32:15Z' level='WARN' msg='Container image not ready'
source='kubelet' node='worker-3'
ts='2024-06-19T12:32:20Z' level='INFO' source='kube-controller' action='deploy'
detail='Rollout progressing' host='cp-1' ns='payments'
ts='2024-06-19T12:32:30Z' level='ERROR' source='kubelet' node='worker-2'
detail='Failed to pull image' pod='checkout-v2'
There are two kinds of functions for this.
absorb_* take a field's name, in quotes, and do the whole job: parse the
text in that field, add what they find to the event as new fields, and remove
the source field once it is fully used. Here the key=value pairs in msg
become fields of their own:
ts='Jan 15 10:00:15' level='ERROR' msg='Payment timeout' duration='5s'
gateway='stripe' order='1234'
ts='Jan 15 10:00:22' level='ERROR' msg='Gateway unreachable' host='stripe.com'
| Function | Parses the named field as |
|---|---|
e.absorb_kv("msg") |
key=value tokens; leftover text stays in the field (not quote-aware) |
e.absorb_logfmt("msg") |
logfmt — the whole field (quote-aware, typed) |
e.absorb_json("msg") |
a JSON object stored as a string |
e.absorb_regex("msg", pattern) |
a regex; named groups become fields |
They return a status ("applied", "parse_error", …) you can branch on, as the
incident_story.log example does.
The other functions work on a field's value and return one result, which you assign to a field yourself:
kelora examples/quickstart.log -f 'cols:ts(3) level *msg' -l error -n 2 \
-e 'e.order = e.msg.extract_regex("order=(\\d+)", 1).or_empty()' -k msg,order
msg='Payment timeout order=1234 gateway=stripe duration=5s' order='1234'
msg='Gateway unreachable host=stripe.com'
| Function | Returns |
|---|---|
e.msg.extract_regex(pattern, 1) |
one capture group |
e.msg.between("[", "]"), .after(":"), .before(" ") |
text around delimiters |
e.msg.extract_json(), .extract_ip(), .extract_url() |
the first JSON / IP / URL in the text |
e.url.parse_url(), e.agent.parse_user_agent(), e.token.parse_jwt() |
a map of parts |
They return "" when they find nothing; .or_empty() turns that into "no
field", as on the second line above. The full list is in the
function reference.
The field doesn't have to come from a parser: on a file Kelora couldn't
parse, everything is in line, so -e 'e.absorb_kv("line")' turns any
KEY=VALUE pairs in it straight into fields.
The same approach parses a field that is itself a log line — a JSON payload
inside a Kubernetes CRI msg, a key=value list inside a syslog message:
ts='2024-07-17T12:12:05.120394823Z' stream='stdout' level='info' msg='listening'
ts='2024-07-17T12:12:09.448210001Z' stream='stdout' level='info' msg='request'
status=200
ts='2024-07-17T12:12:10.002931552Z' stream='stdout' level='warn'
msg='slow upstream'
ts='2024-07-17T12:12:10.913322873Z' stream='stderr'
msg='panic: runtime error: invalid memory address or nil pointer dereference'
ts='2024-07-17T12:12:10.913401201Z' stream='stderr'
msg='[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4b1c2a]'
Lines whose msg isn't JSON (the panic) keep their text; absorb_json just
returns a parse_error status for them.
Quick column grabs work too: e.line.col(2) returns the third whitespace column,
and e.line.split(" ") an array.
Get the timestamp right¶
Time ranges (--since), time windows (--span), merging files, and the time
shown in the output all depend on the timestamp. When it is wrong, nothing
errors: events just fall outside a range, land in the wrong window, or show
the wrong time. So check it once for every new log source. --stats says
which field Kelora used and how many values it could read:
kelora examples/cols_fixed.log -f 'cols:ts(2) level service *msg' --stats | grep -E 'Timestamp|Time span'
Timestamp: ts (auto-detected) - 10/10 parsed (100.0%) — access via meta.parsed_ts.
Time span: 2024-01-15T10:00:00+00:00 to 2024-01-15T10:00:45+00:00 (45s)
Kelora looks for common field names (ts, timestamp, time, @timestamp,
…) and recognizes most formats, including Unix epochs. Four things can go wrong:
Symptom in --stats |
Cause | Fix |
|---|---|---|
Timestamp: (none found …) |
unusual field name | --ts-field logged |
0/… parsed |
unusual format | --ts-format '%d.%m.%Y %H:%M:%S' |
| a time span shifted by whole hours | no zone in the log, and it isn't UTC | --input-tz Europe/Berlin |
| a guessed year (syslog-style dates) | the log has no year | --input-year 2024 |
A field Kelora doesn't recognize by name:
echo '{"logged":"2024-09-05 10:02:00","msg":"disk check ok"}' | kelora -j --stats | grep Timestamp
echo '{"logged":"2024-09-05 10:02:00","msg":"disk check ok"}' | kelora -j --ts-field logged --stats | grep Timestamp
Timestamp: (none found, 1 events) - 0/0 parsed (0.0%); 1 missing. Hint: Try --ts-field or --ts-format.
Timestamp: logged (--ts-field) - 1/1 parsed (100.0%) — access via meta.parsed_ts.
The quiet one is the time zone. This server logs Berlin local time without a zone. Read as UTC (the default), every timestamp is two hours off — 09:55 UTC instead of 07:55:
kelora examples/berlin_local.log -f 'cols:ts(2) *msg' --stats | grep 'Time span'
kelora examples/berlin_local.log -f 'cols:ts(2) *msg' --input-tz Europe/Berlin --stats | grep 'Time span'
Time span: 2024-09-05T09:55:00+00:00 to 2024-09-05T10:52:00+00:00 (57m)
Time span: 2024-09-05T07:55:00+00:00 to 2024-09-05T08:52:00+00:00 (57m)
Kelora prints a hint when a time filter or window relies on the UTC
assumption. Work with Time covers zones, years, and format codes in
detail. The parsed timestamp is available to scripts as meta.parsed_ts.
One event spans several lines¶
Stack traces, wrapped messages, and pretty-printed payloads occupy several
physical lines. -M (--multiline) joins them into one event before
parsing:
ts='2024-01-15 10:01:00' level='ERROR'
msg='Failed to process request Traceback (most recent call last): File "/app/server.py", line 42, in handle_request result = process_data(request.body) File "/app/processor.py", line 15, in process_data return json.loads(data) ValueError: Invalid JSON format at line 3'
| Strategy | A new event starts at | Good for |
|---|---|---|
timestamp |
a line beginning with a timestamp | most application logs with stack traces |
indent |
a line that is not indented | indented continuations without timestamps |
java, python, go |
anything that isn't a recognized trace line | raw stderr, container output, CI logs |
blank |
the line after a blank line | paragraph-style reports |
regex:match=^BEGIN[:end=^END] |
your start (and optional end) pattern | explicit record markers |
all |
— (the whole input is one event) | whole-file processing |
Lines are joined with spaces by default; add --multiline-join=newline to keep
the line structure (java, python, and go already join with newlines). Multiline grouping happens
before parsing, so the joined block must still match your format — free-text
parsers (line, raw, regex with a trailing .*, layouts that end in the
message such as log4j or glog) handle that. Details: kelora --help-multiline and
Multiline reference.
Prefixes added by other tools¶
docker compose logs and similar tools prepend name | to each line.
--extract-prefix moves it into a field before the rest is parsed:
kelora examples/prefix_docker.log --extract-prefix container \
-f 'regex:(?P<ts>\S+ \S+) \[(?P<level>\w+)\] (?P<msg>.*)' -n 3
ts='2024-01-15 10:00:00' level='INFO' msg='Starting web server on port 8080'
container='web_1'
ts='2024-01-15 10:00:05' level='INFO' msg='Listening for connections'
container='web_1'
ts='2024-01-15 10:00:10' level='INFO' msg='PostgreSQL starting up'
container='db_1'
The separator defaults to |; change it with --prefix-sep.
When lines don't parse¶
Kelora is resilient by default: a line that doesn't match the format is counted, reported in a summary on stderr, and skipped; the rest of the file is processed. The exit code stays 0 unless no line parsed at all.
--statsshows the counts;-vprints each failure.--strictstops at the first failure (exit 1) — use it in pipelines where bad input must not pass silently.- To keep unparsable lines instead of skipping them, end a cascade with
line:-f json,line.