Kelora

Scriptable log processor for the command line.

Parse messy logs into structured events, then filter, transform, and analyze them with embedded Rhai scripting.

Experimental Tool

Kelora is a vibe-coded experimental tool under active development. APIs and behaviour may change without notice.

Examples

Parse embedded formats inside syslog

CommandOutputLog Data

kelora -f syslog examples/simple_syslog.log \
  --exec 'if e.msg.contains("=") { e += e.msg.parse_logfmt() }' \
  --filter 'e.has_field("user")' \
  --keys timestamp,host,user,action,detail,message \
  -F json

kelora -f syslog examples/simple_syslog.log \
  --exec 'if e.msg.contains("=") { e += e.msg.parse_logfmt() }' \
  --filter 'e.has_field("user")' \
  --keys timestamp,host,user,action,detail,message \
  -F json

{"timestamp":"Oct 04 10:27:22","host":"bernier5641","user":"alice","action":"login","detail":"Authenticated via SSO","message":"User login accepted"}
{"timestamp":"Oct 04 10:27:22","host":"gusikowski6748","user":"service-bot","action":"restart","detail":"Process restarted automatically","message":"Service restarted"}
{"timestamp":"Oct 04 10:27:22","host":"gerlach2033","user":"pager-duty","action":"acknowledge","detail":"Acknowledged incident #4242","message":"Change request processed"}
{"timestamp":"Oct 04 10:27:22","host":"emard3831","user":"carol","action":"deploy","detail":"Pushed release v2024.10","message":"Deployment triggered"}

<96>Oct 04 10:27:22 bernier5641 dolore[7276]: user=alice action=login detail="Authenticated via SSO" message="User login accepted"
<131>Oct 04 10:27:22 gusikowski6748 eaque[4669]: user=service-bot action=restart detail="Process restarted automatically" message="Service restarted"
<132>Oct 04 10:27:22 gerlach2033 non[4398]: user=pager-duty action=acknowledge detail="Acknowledged incident #4242" message="Change request processed"
<30>Oct 04 10:27:22 emard3831 assumenda[4110]: user=carol action=deploy detail="Pushed release v2024.10" message="Deployment triggered"
<169>Oct 04 10:27:22 gibson6546 est[9787]: If we compress the program, we can get to the AI interface through the digital HTTP system!
<96>Oct 04 10:27:22 kunde1506 ipsa[7859]: You can't reboot the card without connecting the multi-byte PCI program!
<179>Oct 04 10:27:22 stamm6021 ipsum[4729]: I'Ll index the redundant PNG bus, that should program the IB driver!
<134>Oct 04 10:27:22 ward8280 corporis[5923]: Use the optical SMTP firewall, then you can program the bluetooth circuit!
<132>Oct 04 10:27:22 fadel7435 voluptates[8145]: If we calculate the circuit, we can get to the RAM array through the back-end HDD protocol!
<8>Oct 04 10:27:22 greenholt1125 amet[5061]: Try to copy the USB port, maybe it will synthesize the redundant driver!
<13>Oct 04 10:27:22 wilkinson7356 atque[7503]: We need to compress the open-source SMTP driver!
<32>Oct 04 10:27:22 schamberger8301 nulla[34]: If we override the bus, we can get to the JBOD monitor through the auxiliary JSON card!
<168>Oct 04 10:27:22 leuschke5412 ab[3807]: Try to back up the AGP feed, maybe it will back up the wireless application!
<99>Oct 04 10:27:22 heaney4231 quia[4833]: The SSL system is down, program the cross-platform sensor so we can hack the SSL microchip!
<76>Oct 04 10:27:22 wintheiser2248 earum[863]: The IB sensor is down, reboot the optical sensor so we can generate the RAM system!
<99>Oct 04 10:27:22 stokes8116 minima[2942]: The PCI port is down, connect the primary interface so we can copy the IB program!
<161>Oct 04 10:27:22 sauer1505 ut[465]: Use the online XML driver, then you can transmit the auxiliary port!
<147>Oct 04 10:27:22 lockman6842 illo[981]: You can't input the bandwidth without hacking the solid state SMS pixel!
<33>Oct 04 10:27:22 kunde2345 velit[7658]: Use the digital PNG capacitor, then you can hack the wireless program!
<68>Oct 04 10:27:22 medhurst3836 in[1617]: If we connect the alarm, we can get to the GB alarm through the optical THX application!
<98>Oct 04 10:27:22 keebler3487 aspernatur[4587]: Try to override the AGP firewall, maybe it will reboot the primary feed!
<31>Oct 04 10:27:22 bartell8525 qui[9098]: Synthesizing the circuit won't do anything, we need to calculate the primary SMS sensor!
<7>Oct 04 10:27:22 oreilly3006 iste[1827]: You can't parse the matrix without parsing the open-source CSS firewall!
<70>Oct 04 10:27:22 kutch3057 eos[1488]: Transmitting the sensor won't do anything, we need to copy the online SAS program!
<47>Oct 04 10:27:22 borer2101 dolore[7206]: The ADP firewall is down, bypass the optical bus so we can parse the SDD feed!

Keep stacktraces together

CommandOutputLog Data

kelora examples/multiline_stacktrace.log \
  --multiline timestamp \
  --filter 'e.line.lower().contains("valueerror")' \
  --before-context 1 --after-context 1

kelora examples/multiline_stacktrace.log \
  --multiline timestamp \
  --filter 'e.line.lower().contains("valueerror")' \
  --before-context 1 --after-context 1

/ line='2024-01-15 10:00:05 DEBUG Initializing database connection '
* line='2024-01-15 10:01:00 ERROR Failed to process request Traceback (most recent call last):   File "/app/server.py", line 42, in handle_request     result = process_data(request.body)   File "/app/processor.py", line 15, in process_data     return json.loads(data) ValueError: Invalid JSON format at line 3 '
\ line='2024-01-15 10:01:30 WARN Retrying with default configuration '

2024-01-15 10:00:00 INFO Starting application
2024-01-15 10:00:05 DEBUG Initializing database connection
2024-01-15 10:01:00 ERROR Failed to process request
Traceback (most recent call last):
  File "/app/server.py", line 42, in handle_request
    result = process_data(request.body)
  File "/app/processor.py", line 15, in process_data
    return json.loads(data)
ValueError: Invalid JSON format at line 3
2024-01-15 10:01:30 WARN Retrying with default configuration
2024-01-15 10:02:00 ERROR Database connection failed
java.sql.SQLException: Connection timeout
    at com.example.db.ConnectionPool.getConnection(ConnectionPool.java:123)
    at com.example.api.UserController.getUser(UserController.java:45)
    at com.example.api.RequestHandler.handle(RequestHandler.java:89)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
2024-01-15 10:02:30 INFO Switching to backup database
2024-01-15 10:03:00 ERROR Unhandled exception in worker thread
RuntimeError: Maximum retry attempts exceeded
  File "/app/worker.py", line 67, in run
    self.process_job(job)
  File "/app/worker.py", line 98, in process_job
    raise RuntimeError("Maximum retry attempts exceeded")
2024-01-15 10:03:30 CRITICAL System shutting down due to errors

Track container activity with metrics

CommandOutputLog Data

kelora examples/prefix_docker.log --extract-prefix container \
  --exec 'e.level = e.line.between("[", "]")' \
  --metrics \
  --exec 'track_count(e.container); track_count(e.level)' \
  --keys container,level,line \
  -F csv

kelora examples/prefix_docker.log --extract-prefix container \
  --exec 'e.level = e.line.between("[", "]")' \
  --metrics \
  --exec 'track_count(e.container); track_count(e.level)' \
  --keys container,level,line \
  -F csv

container,level,line
web_1,INFO,2024-01-15 10:00:00 [INFO] Starting web server on port 8080
web_1,INFO,2024-01-15 10:00:05 [INFO] Listening for connections
db_1,INFO,2024-01-15 10:00:10 [INFO] PostgreSQL starting up
db_1,INFO,2024-01-15 10:00:15 [INFO] Database system is ready
redis_1,INFO,2024-01-15 10:00:20 [INFO] Redis server starting
redis_1,INFO,2024-01-15 10:00:25 [INFO] Ready to accept connections
web_1,INFO,2024-01-15 10:00:30 [INFO] Connected to database
web_1,DEBUG,2024-01-15 10:00:35 [DEBUG] Cache connection established
nginx_1,INFO,2024-01-15 10:00:40 [INFO] Nginx starting
nginx_1,INFO,2024-01-15 10:00:45 [INFO] Ready to serve traffic
web_1,INFO,2024-01-15 10:01:00 [INFO] Received request GET /api/health
web_1,INFO,2024-01-15 10:01:05 [INFO] Health check passed
db_1,WARN,2024-01-15 10:01:10 [WARN] Connection pool 80% utilized
redis_1,WARN,2024-01-15 10:01:15 [WARN] Memory usage at 75%
worker_1,INFO,2024-01-15 10:01:20 [INFO] Background worker starting
worker_1,INFO,2024-01-15 10:01:25 [INFO] Processing job queue
web_1,ERROR,2024-01-15 10:01:30 [ERROR] Request timeout after 5s
nginx_1,ERROR,2024-01-15 10:01:35 [ERROR] Upstream server timeout
worker_1,INFO,2024-01-15 10:01:40 [INFO] Job completed: send_email
db_1,INFO,2024-01-15 10:01:45 [INFO] Checkpoint completed

kelora: Tracked metrics:
DEBUG        = 1
ERROR        = 2
INFO         = 15
WARN         = 2
db_1         = 4
nginx_1      = 3
redis_1      = 3
web_1        = 7
worker_1     = 3

web_1        | 2024-01-15 10:00:00 [INFO] Starting web server on port 8080
web_1        | 2024-01-15 10:00:05 [INFO] Listening for connections
db_1         | 2024-01-15 10:00:10 [INFO] PostgreSQL starting up
db_1         | 2024-01-15 10:00:15 [INFO] Database system is ready
redis_1      | 2024-01-15 10:00:20 [INFO] Redis server starting
redis_1      | 2024-01-15 10:00:25 [INFO] Ready to accept connections
web_1        | 2024-01-15 10:00:30 [INFO] Connected to database
web_1        | 2024-01-15 10:00:35 [DEBUG] Cache connection established
nginx_1      | 2024-01-15 10:00:40 [INFO] Nginx starting
nginx_1      | 2024-01-15 10:00:45 [INFO] Ready to serve traffic
web_1        | 2024-01-15 10:01:00 [INFO] Received request GET /api/health
web_1        | 2024-01-15 10:01:05 [INFO] Health check passed
db_1         | 2024-01-15 10:01:10 [WARN] Connection pool 80% utilized
redis_1      | 2024-01-15 10:01:15 [WARN] Memory usage at 75%
worker_1     | 2024-01-15 10:01:20 [INFO] Background worker starting
worker_1     | 2024-01-15 10:01:25 [INFO] Processing job queue
web_1        | 2024-01-15 10:01:30 [ERROR] Request timeout after 5s
nginx_1      | 2024-01-15 10:01:35 [ERROR] Upstream server timeout
worker_1     | 2024-01-15 10:01:40 [INFO] Job completed: send_email
db_1         | 2024-01-15 10:01:45 [INFO] Checkpoint completed

Mask sensitive fields in JSON logs

CommandOutputLogs Data

kelora -j examples/security_audit.jsonl \
  --exec 'if e.has_field("token") {
            let jwt = e.token.parse_jwt();
            e.role = jwt.get_path("claims.role", "guest")
          }' \
  --exec 'e.ip = e.ip.mask_ip(2)' \
  --keys timestamp,event,role,ip \
  -F json

kelora -j examples/security_audit.jsonl \
  --exec 'if e.has_field("token") {
            let jwt = e.token.parse_jwt();
            e.role = jwt.get_path("claims.role", "guest")
          }' \
  --exec 'e.ip = e.ip.mask_ip(2)' \
  --keys timestamp,event,role,ip \
  -F json

{"timestamp":"2024-01-15T10:00:00Z","event":"login","role":"user","ip":"192.168.X.X"}
{"timestamp":"2024-01-15T10:00:10Z","event":"login","ip":"203.0.X.X"}
{"timestamp":"2024-01-15T10:00:20Z","event":"api_call","role":"admin","ip":"10.0.X.X"}
{"timestamp":"2024-01-15T10:00:30Z","event":"file_upload","ip":"172.16.X.X"}
{"timestamp":"2024-01-15T10:00:40Z","event":"suspicious_activity","ip":"198.51.X.X"}
{"timestamp":"2024-01-15T10:00:50Z","event":"login","ip":"2001:db8::1"}
{"timestamp":"2024-01-15T10:01:00Z","event":"password_change","ip":"192.168.X.X"}
{"timestamp":"2024-01-15T10:01:10Z","event":"api_call","ip":"10.255.X.X"}
{"timestamp":"2024-01-15T10:01:20Z","event":"vpn_connect"}
{"timestamp":"2024-01-15T10:01:30Z","event":"data_export","ip":"192.168.X.X"}
{"timestamp":"2024-01-15T10:01:40Z","event":"firewall_block"}
{"timestamp":"2024-01-15T10:01:50Z","event":"certificate_renewal"}
{"timestamp":"2024-01-15T10:02:00Z","event":"malware_scan"}
{"timestamp":"2024-01-15T10:02:10Z","event":"ddos_attempt"}
{"timestamp":"2024-01-15T10:02:20Z","event":"ssh_login","ip":"192.168.X.X"}

{"timestamp":"2024-01-15T10:00:00Z","event":"login","user":"alice","ip":"192.168.1.10","success":true,"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkFsaWNlIiwicm9sZSI6InVzZXIiLCJpYXQiOjE1MTYyMzkwMjJ9.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"}
{"timestamp":"2024-01-15T10:00:10Z","event":"login","user":"admin","ip":"203.0.113.42","success":false,"reason":"invalid_password","attempts":3}
{"timestamp":"2024-01-15T10:00:20Z","event":"api_call","ip":"10.0.0.5","endpoint":"/api/users","method":"GET","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI5ODc2NTQzMjEiLCJuYW1lIjoiQm9iIiwicm9sZSI6ImFkbWluIiwiaWF0IjoxNTE2MjM5MDIyfQ.dQw4w9WgXcQ"}
{"timestamp":"2024-01-15T10:00:30Z","event":"file_upload","user":"charlie","ip":"172.16.0.10","file":"document.pdf","hash":"5d41402abc4b2a76b9719d911017c592","size":1048576}
{"timestamp":"2024-01-15T10:00:40Z","event":"suspicious_activity","ip":"198.51.100.50","type":"port_scan","ports":[22,23,80,443,3306,5432,8080],"blocked":true}
{"timestamp":"2024-01-15T10:00:50Z","event":"login","user":"diana","ip":"2001:db8::1","success":true,"ipv6":true}
{"timestamp":"2024-01-15T10:01:00Z","event":"password_change","user":"eve","ip":"192.168.1.20","old_hash":"5f4dcc3b5aa765d61d8327deb882cf99","new_hash":"098f6bcd4621d373cade4e832627b4f6"}
{"timestamp":"2024-01-15T10:01:10Z","event":"api_call","ip":"10.255.255.1","endpoint":"/api/admin/users","method":"DELETE","user":"frank","blocked":true,"reason":"insufficient_privileges"}
{"timestamp":"2024-01-15T10:01:20Z","event":"vpn_connect","user":"grace","client_ip":"203.0.113.100","vpn_ip":"10.8.0.5","protocol":"openvpn"}
{"timestamp":"2024-01-15T10:01:30Z","event":"data_export","user":"henry","ip":"192.168.2.30","records":10000,"hash":"e4d909c290d0fb1ca068ffaddf22cbd0","approved":true}
{"timestamp":"2024-01-15T10:01:40Z","event":"firewall_block","src_ip":"185.220.101.42","dst_ip":"192.168.1.1","dst_port":22,"reason":"blacklist","country":"unknown"}
{"timestamp":"2024-01-15T10:01:50Z","event":"certificate_renewal","domain":"api.example.com","issuer":"Let's Encrypt","fingerprint":"sha256:a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2"}
{"timestamp":"2024-01-15T10:02:00Z","event":"malware_scan","file":"/uploads/suspicious.exe","hash":"44d88612fea8a8f36de82e1278abb02f","threat":"trojan.generic","quarantined":true}
{"timestamp":"2024-01-15T10:02:10Z","event":"ddos_attempt","src_ips":["203.0.113.1","203.0.113.2","203.0.113.3"],"requests_per_second":5000,"mitigated":true}
{"timestamp":"2024-01-15T10:02:20Z","event":"ssh_login","user":"root","ip":"192.168.1.100","key_fingerprint":"SHA256:abcd1234efgh5678ijkl9012mnop3456","success":true}

See the Quickstart for a step-by-step tour with full output.

What It Does

• Parse JSON, logfmt, syslog, CSV/TSV, Apache/Nginx logs, or custom formats
• Filter with Rhai expressions - keep events matching your conditions
• Transform with 100+ built-in functions - enrich, redact, extract, restructure
• Analyze with built-in metrics - track counts, sums, averages, distributions
• Output as logfmt, JSON, or CSV

How It Works

Kelora processes logs through a streaming pipeline with composable stages:

Input → Parse → --exec → --filter → --exec → --filter → ... → Output
  ↓       ↓         ↓         ↓         ↓         ↓              ↓
Files   JSON   transform  narrow   enrich    narrow        logfmt
stdin   syslog                                              JSON
.gz     custom                                              CSV

Each --filter and --exec runs in the order specified, passing events forward. Chain them in any sequence to build multi-stage processing logic. Read the Pipeline Model for details.

Key Features

• Resilient Processing - Skip bad lines automatically, continue processing
• Parallel Mode - Process large archives using all CPU cores with --parallel
• Sliding Windows - Analyze events in context with --window
• 100+ Functions - Rich built-in library for transformation and analysis
• Format Conversion - Read any format, write any format
• Metrics Tracking - Built-in counters, sums, averages, distributions

Install

Download from GitHub Releases (macOS, Linux, Windows) or:

cargo install kelora

Learn More

• Quickstart - 5-minute tour
• Tutorials - Step-by-step guides
• How-To - Solve specific problems
• Concepts - Understanding how Kelora works
• Reference - Functions, formats, CLI options

Run kelora --help for comprehensive command-line help screens, or browse the example files on GitHub.

Works Well With

Kelora focuses on normalising noisy logs into structured data. Reach for it when you need programmable filtering, enrichment, or windowed analytics in one streaming pipeline—and pair it with the tools below when you need deeper visualisation or post-processing:

• jq — process Kelora's JSON output for complex transformations, filtering, or reformatting
• lnav — explore Kelora's output in an interactive TUI with live filtering, histograms, and ad-hoc SQL queries
• qsv — analyze Kelora's CSV output with statistical operations, joins, and aggregations
• SQLite/DuckDB — load Kelora's CSV/JSON output into a database for SQL queries and reporting
• miller — transform Kelora's CSV output for reshaping, aggregating, and format conversion

License

Kelora is open source software licensed under the MIT License.