Metadata-Version: 2.5
Name: jsonl-tail
Version: 0.1.0
Summary: Tail JSONL files (eval logs, datasets) with pretty-printed JSON, filters, and live follow mode
Project-URL: Homepage, https://github.com/FreakyAdy/jsonl-tail
Project-URL: Repository, https://github.com/FreakyAdy/jsonl-tail
Project-URL: Issues, https://github.com/FreakyAdy/jsonl-tail/issues
Project-URL: Changelog, https://github.com/FreakyAdy/jsonl-tail/blob/main/CHANGELOG.md
Author-email: Aditya Suryavanshi <urban.void.69@gmail.com>
License: MIT
License-File: LICENSE
Keywords: cli,eval,json,jsonl,llm,logs,streaming,tail
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Logging
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Provides-Extra: windows
Requires-Dist: colorama>=0.4.6; (sys_platform == 'win32') and extra == 'windows'
Description-Content-Type: text/markdown

<div align="center">

# 📜 `jsonl-tail`
### A Resilient, Zero-Dependency `tail -f` for JSON Lines and LLM Eval Logs

**Stream, pretty-print, and filter live JSONL without crashing on corrupt lines.**

[![CI / Quality Gate](https://github.com/FreakyAdy/jsonl-tail/actions/workflows/ci.yml/badge.svg)](https://github.com/FreakyAdy/jsonl-tail/actions)
[![Tests Passing](https://img.shields.io/badge/tests-25%2F25%20passed%20(100%25)-brightgreen.svg)](tests/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://python.org)
[![Zero Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen.svg)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![PyPI version](https://img.shields.io/pypi/v/jsonl-tail.svg)](https://pypi.org/project/jsonl-tail/)
[![Contributing Guide](https://img.shields.io/badge/contributing-guide-blue.svg)](CONTRIBUTING.md)
[![Changelog](https://img.shields.io/badge/changelog-v0.2.0-brightgreen.svg)](CHANGELOG.md)

<p align="center">
  <a href="#-quick-demo"><b>⚡ Quick Demo</b></a> •
  <a href="#-why-jsonl-tail"><b>💡 Why jsonl-tail</b></a> •
  <a href="#-key-capabilities"><b>🎯 Key Capabilities</b></a> •
  <a href="#-system-architecture"><b>🏗️ Architecture</b></a> •
  <a href="#-quick-start"><b>🚀 Quick Start</b></a> •
  <a href="#-ecosystem-comparison"><b>⚖️ Comparisons</b></a>
</p>

<br>

<p align="center">
  <img src="docs/demo.gif" alt="jsonl-tail streaming evaluation logs with syntax colors and corrupt line resilience" width="100%" style="border-radius: 12px; box-shadow: 0 12px 40px rgba(0,0,0,0.5);">
</p>

<!-- 
RECORDING INSTRUCTIONS FOR HERO GIF:
Tool: VHS (https://github.com/charmbracelet/vhs) or Asciinema (https://asciinema.org)
Dimensions: 90 columns x 22 rows
Script:
  1. jsonl-tail examples/eval.log.jsonl
  2. jsonl-tail --regex '"(WARN|ERROR)"' examples/eval.log.jsonl
  3. jsonl-tail -k meta -c examples/eval.log.jsonl
Save output to docs/demo.gif
-->

> **⚡ Zero-Crash Stream Processing** — Unlike `tail -f file.jsonl | jq .` which terminates the entire pipeline on the first non-JSON log line and buffers stdout when piped, `jsonl-tail` provides atomic follow polling, fault-tolerant record pass-through, key projection, and bounded memory usage with zero external dependencies.

</div>

---

## ⚡ Quick Demo

Tailing an active LLM evaluation log with formatted JSON, metadata extraction, and safe handling of corrupt disk writes:

```bash
$ jsonl-tail examples/eval.log.jsonl
```

```text
{
  "timestamp": "2026-09-03T10:00:03Z",
  "level": "INFO",
  "run_id": "eval-001",
  "model": "claude-3-5-sonnet",
  "step": 4,
  "score": 0.97,
  "meta": {
    "dataset": "human-eval",
    "temperature": 0.0
  }
}
[invalid json] [corrupt-disk-write-header] 0xDEADBEEF <<< corrupt log line pass-through >>>
{
  "timestamp": "2026-09-03T10:00:04Z",
  "level": "ERROR",
  "run_id": "eval-001",
  "model": "claude-3-5-sonnet",
  "step": 5,
  "score": 0.0,
  "meta": {
    "dataset": "human-eval",
    "error": "rate_limit_exceeded"
  }
}
{
  "timestamp": "2026-09-03T10:00:05Z",
  "level": "INFO",
  "run_id": "eval-001",
  "model": "gemini-2.5-pro",
  "step": 6,
  "score": 0.91,
  "meta": {
    "dataset": "math500",
    "i18n": "日本語テスト (evaluation successful)"
  }
}
```

*Notice how the middle corrupt line is badged with `[invalid json]` and passes through cleanly without crashing the viewer.*

---

## 💡 Why `jsonl-tail`?

In modern AI engineering, agent trajectories, eval benchmarks (SWE-bench, GSM8K), and model training traces all stream as append-only `.jsonl` files. Developers typically inspect these logs using traditional Unix pipelines:

```bash
$ tail -f run.jsonl | jq .
```

While convenient, this pipeline breaks down in production for three critical reasons:

1. **The Fatal Syntax Crash:** `jq` expects 100% syntactically valid JSON. If an external library prints a raw traceback, a progress bar, or an unformatted warning directly to stdout, `jq` throws `parse error` and immediately aborts the pipeline.
2. **Standard Output Buffering:** When `jq` is piped into downstream commands (or monitored in subshells), its output is block-buffered by default unless explicitly invoked with `--unbuffered`, delaying live step updates.
3. **Partial Write Race Conditions:** Active logging processes periodically flush byte chunks before the trailing newline `\n` hits disk. Standard tools attempt to parse the half-written line, corrupting terminal output.

**`jsonl-tail` solves these edge cases out of the box.** It is a single, zero-dependency CLI utility that gives you native `tail -f` semantics specifically engineered for JSON Lines.

---

## 🎯 Key Capabilities

| Capability | Flag & Syntax | Behavior & Guarantees | Failure Resilience |
| :--- | :--- | :--- | :---: |
| **Fault-Tolerant Tail** | `jsonl-tail log.jsonl` | Pretty-prints the last $N$ records with 2-space indentation and ANSI coloring. | Non-JSON lines pass through raw with `[invalid json]` badge. |
| **Live Atomic Follow** | `jsonl-tail -f log.jsonl` | Emits the last records and polls for newly appended records (`tail -f`). | Incomplete lines lacking `\n` are held until fully flushed. |
| **Piped Stream Tail** | `cat log.jsonl \| jsonl-tail -n 5` | Bounded memory consumption; correctly preserves `-n` count on standard input. | Memory bounded to $O(N)$ via ring buffer. |
| **Compact Mode** | `jsonl-tail -c log.jsonl` | Compresses output to one JSON record per line (pipe-friendly). | Strips unnecessary whitespace without modifying values. |
| **Key Projection** | `jsonl-tail -k meta -c log.jsonl` | Extracts top-level keys; formats nested objects or primitives cleanly. | Emits `[no key '<key>']` if missing; never throws KeyErrors. |
| **Regex & Substring Filter** | `jsonl-tail --regex '"(WARN\|ERROR)"'` | Retains only matching records before formatting. | Compiles pattern once; clean syntax errors on invalid regex. |

---

## 🏗️ System Architecture

`jsonl-tail` follows a streaming, memory-bounded pipeline designed for continuous operation:

```mermaid
flowchart LR
    subgraph INGESTION["1. Ingestion & Buffering"]
        A["File Path(s) / Piped Stdin"] --> B["Bounded Ring Buffer (deque)"]
        A --> C["Atomic Follow Poller"]
    end

    subgraph RECOVERY["2. Fault-Tolerant Engine"]
        B --> D["Line Normalizer"]
        C --> D
        D --> E["JSON Decoder & Guard"]
        E -->|Valid Record| F["Filter & Key Projection"]
        E -->|Corrupt / Raw Text| G["Pass-Through Badging"]
    end

    subgraph EMISSION["3. Output Pipeline"]
        F --> H["Format Engine (Pretty / Compact)"]
        G --> I["ANSI Color Renderer"]
        H --> I
        I --> J["Immediate Unbuffered Flush (stdout)"]
    end
```

### Architectural Highlights
- **Bounded Memory Ingestion:** Files and stdin streams are consumed through a bounded FIFO ring buffer (`collections.deque(maxlen=N)`). Memory footprint remains constant regardless of whether the log is 10 KB or 50 GB.
- **Atomic Line Recovery:** The follow engine inspects trailing byte offsets for line feeds (`\n`, `\r`). Incomplete writes are buffered in memory and the file pointer is preserved until complete records land on disk.
- **Stream Auto-Reconfiguration:** Standard output and error streams are dynamically reconfigured to UTF-8 with character replacement on startup, guaranteeing consistent rendering on Windows terminals and non-UTF8 locales.

---

## 🚀 Quick Start

### Installation

Choose the installation method that fits your workflow:

```bash
# Method 1: Install as a standalone CLI tool via uv (Recommended)
uv tool install jsonl-tail

# Method 2: Install via standard pip
pip install jsonl-tail

# Method 3: Run instantly without installing
uvx jsonl-tail examples/eval.log.jsonl

# Method 4: Editable local development setup
git clone https://github.com/FreakyAdy/jsonl-tail.git
cd jsonl-tail
uv sync
```

---

### Basic Commands

```bash
# 1. View last 10 records, formatted with syntax colors
jsonl-tail examples/eval.log.jsonl

# 2. View all records in compact one-line JSON
jsonl-tail -a -c examples/eval.log.jsonl

# 3. Follow live appended logs as they land
jsonl-tail -f examples/eval.log.jsonl

# 4. Filter for specific severity levels using regex
jsonl-tail --regex '"(WARN|ERROR)"' examples/eval.log.jsonl

# 5. Extract a specific nested key in compact format
jsonl-tail -k meta -c examples/eval.log.jsonl

# 6. Pipe input from stdin while respecting tail counts
cat examples/eval.log.jsonl | jsonl-tail -n 3 -c
```

> **💡 Terminal Tip:** Set `NO_COLOR=1` in your environment to disable ANSI color codes when piping to files or plain text pagers.

---

### Runnable Examples

Explore the bundled [`examples/`](examples/) directory to test `jsonl-tail` against real evaluation records:

```bash
# Inspect the sample evaluation log
jsonl-tail examples/eval.log.jsonl

# Filter for failed steps
jsonl-tail --filter "rate_limit_exceeded" examples/eval.log.jsonl
```

---

## ⚖️ Ecosystem Comparison

How `jsonl-tail` compares to existing log viewers and command-line JSON tools:

| Feature / Metric | `jsonl-tail` | `tail -f \| jq .` | `lnav` | `fx` / `jless` |
| :--- | :---: | :---: | :---: | :---: |
| **Primary Scope** | **Stream & Follow JSONL** | General JSON processor | Interactive log TUI | Interactive JSON tree pager |
| **Fault Tolerance** | ✅ **100% (Pass-through)** | ❌ Aborts on corrupt line | ✅ Tolerant | ❌ Aborts on parse error |
| **Stream-Native (`tail -f`)** | ✅ Built-in | ⚠️ Fragile (buffering issues) | ✅ Built-in | ❌ Static inspection only |
| **Partial Write Safety** | ✅ Atomic line wait | ❌ Fails parse | ✅ Tolerant | ❌ Not applicable |
| **Pipe / Script Friendly** | ✅ Pure CLI stream | ✅ Pure CLI stream | ❌ Full-screen TUI | ❌ Full-screen TUI |
| **External Dependencies** | ✅ **Zero (Pure Python)** | Requires `jq` + coreutils | Requires C++ / libpcre | Requires Node.js / Rust |
| **Cross-Platform** | ✅ Linux, macOS, Windows | ⚠️ POSIX preferred | ⚠️ POSIX only | ✅ Cross-platform |

> *`jsonl-tail` is designed to be a lightweight, zero-dependency stream filter. When you need complex tree queries across deep schemas, use `jq`. When you need full-screen interactive log analysis, use `lnav`. When you want a crash-proof `tail -f` for JSON Lines, use `jsonl-tail`.*

---

## 🤝 Contributing & Community

`jsonl-tail` is an open-source community project. Contributions, bug reports, and ideas are warmly welcome!

* **[CONTRIBUTING.md](CONTRIBUTING.md)**: Development setup, local testing, and PR guidelines.
* **[CHANGELOG.md](CHANGELOG.md)**: Release history and version notes.
* **[Issue Tracker](https://github.com/FreakyAdy/jsonl-tail/issues)**: Report bugs or request enhancements.
* **[Pull Request Template](.github/pull_request_template.md)**: Standard PR review checklist.

---

## 📄 License

Distributed under the **[MIT License](LICENSE)**.

Copyright (c) 2026 Aditya Suryavanshi
