Metadata-Version: 2.5
Name: zelos-log-lines
Version: 0.1.1
Summary: Text log lines to Zelos log rows and signals: log levels, device time and printed values.
Project-URL: Homepage, https://zeloscloud.io
Author-email: Zelos Cloud <info@zeloscloud.io>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: backports-zstd>=1.0; python_version < '3.14'
Requires-Dist: zelos-sdk<0.1,>=0.0.12
Description-Content-Type: text/markdown

# zelos-log-lines

Turns what a device or program prints into Zelos log rows and signals. Zelos extensions that read text, such as a serial console or a board's journal, share it.

## Use

```python
import time

import zelos_sdk
from zelos_log_lines.stream import LogStream

source = zelos_sdk.TraceSource("acme")
stream = LogStream(source, "board", notes="acme")

stream.feed(b"[00:00:12.340,000] <wrn> dcdc: rail=13.73V\n", time.time_ns())
stream.close()
```

That line becomes a `warn` row in `board/log`, named `dcdc`, placed by the board's own uptime, and the value `rail` becomes a signal in `board/dcdc`.

A `LogStream` has two ways in:

- **`feed(data, host_ns)`** takes bytes as they arrive from a port or a pipe. It splits them into lines with a terminal's rules: colors, `\r`, erased lines, and lines over 4 KB.
- **`write(text, host_ns, ...)`** takes a line that another framing produced, such as a journal entry's message, with what its source knows: `default_level` applies only when the line prints no level, while `name`, `file`, `line` and `time_ns` are facts that beat what the text says.

`note(level, message, host_ns)` writes the extension's own row, `state()` reports counters and signals, and `close()` writes what is still held. A reader whose connection closes calls `end()`.

A record printed over several lines, such as a traceback or a message continued under its header, is one row placed at its first line. The stream holds the last record until the next one starts, or until the source has been quiet for 100 ms, and writes part of a line after 100 ms with no bytes; a timer of the stream's own does both, so a caller never flushes. The `keep` and `on_line` callbacks run under the stream's lock, so on that timer's thread too for a line written after a pause. Part of a line, cut or ended by a pause, joins nothing. A blank line neither ends a record nor joins it, as a terminal shows it.

Two hooks let an extension take part. `keep` sees each fed line first and returns it, a changed line, or None to leave it out, such as a shell prompt. `on_line` sees each line as soon as it is read, with what that line says on its own, before `keep` sees the next. An unexpected error, in the stream or a hook, costs at most its line, and a line `keep` fails on is written as it came. The first error is logged, then one a minute, and `state().errors` counts each.

## What it reads

A level comes only from a line's header, never from a word in its sentence. A line in no known format is a plain `info` line, and its terminal color sets its level.

| | Formats |
|---|---|
| **Microcontrollers** | Zephyr (uptime, date and tick stamps, and its minimal mode, as MCUboot prints it), ESP-IDF, ESPHome, esp-println, Arduino on ESP32, the nRF5 SDK, FreeRTOS, NuttX, RIOT, Contiki-NG, Mynewt, Pigweed, Particle, CircuitPython and defmt |
| **Linux** | the kernel's printk, dmesg's own formats, and /dev/kmsg |
| **JVM and .NET** | logback, log4j and log4j2, Spring Boot, java.util.logging, tinylog, Microsoft.Extensions.Logging, Serilog, NLog, log4net |
| **Scripting runtimes** | Python logging, structlog, rich and loguru; pino-pretty, the bunyan CLI and winston; Ruby's Logger; Monolog; Elixir's Logger and Erlang's reports; swift-log |
| **Go** | `log`, `slog`, logrus, zap, zerolog, charmbracelet/log, hclog, glog and klog |
| **Rust** | env_logger, pretty_env_logger, tracing, simplelog, flexi_logger, log4rs |
| **C and C++** | spdlog, Boost.Log, glog, easylogging++, plog, loguru, quill, Qt, GLib, log4cplus, POCO |
| **Robotics and Android** | ROS 1, ROS 2, and Android logcat in each of its formats |
| **Configured layouts** | A stamp, bracketed fields, a level word, then the logger and the message, in the arrangements logback, log4j, Python logging, Serilog, NLog and their peers let a program choose |
| **Structured** | logfmt and JSON lines, with a level key and a message key, including those of Elastic ECS, GELF, MongoDB, PostgreSQL, Serilog's compact JSON, Google Cloud, hclog, slog, log15 and Microsoft.Extensions.Logging |
| **Syslog and the journal** | journalctl's outputs, its JSON export included, syslog files from rsyslog, syslog-ng and BusyBox, OpenWrt `logread`, macOS's styles, ESXi, a program's syslog echo on stderr, RFC 3164, RFC 5424 and RFC 6587 on the wire, and systemd's `<N>` prefix. The line a record carries is read again, and its own level wins over the record's. |
| **Servers** | Apache, nginx, Envoy, Flask, PostgreSQL, MySQL, Redis, MongoDB, Kafka, Elasticsearch and Logstash, NATS, Icinga, CUPS, SSSD |
| **Containers** | Docker's json-file, CRI, and `docker logs -t`; the line a container wrote is read again |
| **A level word** | `ERROR: …`, `[WRN] …` |
| **Crash output** | Python tracebacks, chained ones included, the uncaught exceptions of Java, .NET, Node, Ruby and PHP, Go and Rust panics, C and C++ aborts and failed asserts, fatal signals, AddressSanitizer, ThreadSanitizer and UBSan reports, and the Linux kernel's WARN, BUG, oops and panic reports are one row each. Stack frames (`at …`) under an exception a logger printed join it and make it an error. A microcontroller's panic, fault or failed assertion (ESP-IDF, ESP8266, Zephyr, Mbed OS, NuttX, embedded Rust) is one `critical` row. |

Printed values become signals, one event per module: `key=value`, Teleplot, the Arduino plotter, and a repeated `label: value`. Values come only from whole lines.

## Time

A printed uptime places a live line on the host clock, through a mapping that follows drift and notices restarts (`device_time=False` places every line by its arrival). A wall-clock stamp never places a live row. A source that knows a row's time, such as the journal, passes `time_ns`.

## Saved files

`files.convert(path)` writes a saved log file to a Zelos trace beside it, as `Logs/<file>/log` and its signals. `files.replay(path, stream, stop)` writes it into a live stream at the pace it was printed, its first line now. Plain text reads, and so does text compressed with gzip, bzip2, xz or zstd, in UTF-8 or in UTF-16 led by its mark, with lines that end as a terminal ends them. A conversion reads the same however long it takes: its records end at the next record, not after a pause.

A saved file's row goes where its line printed it:

- A stamp with a zone is placed as printed. A stamp with none is read in this computer's zone, or the `Timing` zone given.
- A stamp with no year takes the year of the file's last change, stepping back a year wherever the stamps wrap. A time with no date takes the date of the dated stamp before it, or else does the same with the date.
- A file that prints only its device's uptime ends at its last change, or starts at the `Timing` start given; after a restart, a run follows the one before.
- A line that prints no time follows the line before it by 1 µs.

## Develop

```bash
just install   # uv sync --locked
just check     # ruff, pyright, pytest
```

Most formats are one regex with named groups, a row in a `formats/` module. Syslog, logfmt, JSON and plain text are code. A row in `native.py` or `managed.py` names the characters its lines can start with, so a line meets only the rows that could read it. `formats.read(text)` returns the `LogRecord` for one line, with the format that read it.

`tests/oracles/` holds the sample lines of lnav's format catalog and of Elastic Filebeat's module tests, with the level each gives, under their own licenses. No line may read above its level, and `differ.json` names each line that reads differently, with why.

To add a format, add its row, then `tests/vectors/<format>.txt` with at least 15 lines it must read and 10 it must not. Each line reads `input ⇒ {expected}`, and a `---` block is one case whose lines share history. `tests/zoo/` holds lines that real logging libraries printed, with what each program logged.
