Metadata-Version: 2.3
Name: icho
Version: 0.8.0
Summary: Snapshot Testing for LLM Applications
Author: thetechnoadvisor
Author-email: thetechnoadvisor <thetechnoadvisor@gmail.com>
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: langchain-core>=0.1.0
Requires-Dist: langchain-groq>=0.3.0
Requires-Dist: openai>=1.0.0
Requires-Dist: anthropic>=0.18.0
Requires-Dist: nemoguardrails>=0.23.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: pytest>=8.0.0 ; extra == 'dev'
Requires-Dist: twine>=5.0.0 ; extra == 'dev'
Requires-Dist: langgraph>=0.2.0 ; extra == 'langgraph'
Requires-Dist: psycopg[binary]>=3.0.0 ; extra == 'postgres'
Requires-Python: >=3.12
Provides-Extra: dev
Provides-Extra: langgraph
Provides-Extra: postgres
Description-Content-Type: text/markdown

# Icho 📼

> **Deterministic testing for AI applications.**
>
> **Record once. Replay forever.**

[![PyPI](https://img.shields.io/pypi/v/icho)](https://pypi.org/project/icho/)
[![Python](https://img.shields.io/pypi/pyversions/icho)](https://pypi.org/project/icho/)
[![License](https://img.shields.io/github/license/thetechnoadvisor/llmcassette)](LICENSE)

---

## Stop paying for every AI test run.

Every time your AI application runs during testing, it probably:

- 💸 Calls the LLM again
- 🐢 Slows down your CI pipeline
- 🎲 Produces slightly different outputs
- 🌐 Depends on internet connectivity

**Icho records a real AI execution once and replays it locally during future test runs.**

### The result

- ⚡ Millisecond replay
- 💰 Zero replay API costs
- 🧪 Deterministic testing
- 💻 Works offline

---

# Before

```python
from langchain_groq import ChatGroq

model = ChatGroq(model_name="llama-3.1-8b-instant")

response = model.invoke(
    "Write a 3-word slogan for gravity."
)

# ⏱️ 2.3 seconds
# 🌐 Live API Call
```

---

# After

```python
from langchain_groq import ChatGroq
from icho import cassette

model = ChatGroq(model_name="llama-3.1-8b-instant")

with cassette("tests/cassettes"):
    response = model.invoke(
        "Write a 3-word slogan for gravity."
    )

# First Run
# ⏱️ 2.3 seconds
# 🌐 Live API Call
# 💾 Recorded

# Every Run After
# ⏱️ 12 ms
# ❌ No API Calls
# 📼 Replayed Locally
```

---

# Why Icho?

| Without Icho | With Icho |
|---------------|------------|
| Calls the LLM on every test | Record once, replay forever |
| Seconds of latency | Millisecond replay |
| API cost every execution | No replay API cost |
| Internet required | Works offline |
| Non-deterministic | Deterministic |

---

# Supported Frameworks

- ✅ OpenAI
- ✅ Anthropic
- ✅ LangChain
- ✅ LangGraph

---

# Installation

```bash
pip install icho
```

or

```bash
uv add icho
```

---

# Quick Start

```python
from langchain_groq import ChatGroq
from icho import cassette

model = ChatGroq(model_name="llama-3.1-8b-instant")

with cassette("tests/cassettes"):
    response = model.invoke("Hello Icho!")
```

That's it.

The first execution records the response.

Every matching execution after that replays it locally without calling the LLM.

---

# Features

- 🧪 **Phase 2 AI Regression Testing Engine**: Compare reference cassettes against new executions across 5 analysis dimensions: **Prompt Diff**, **Tool Diff**, **Semantic Diff**, **Cost Diff**, and **Latency Diff**.
- 🔎 **Searchable AI Executions & Instant Replay**: Find past executions with TF-IDF cosine similarity (`icho search "refund" --since yesterday`) and generate local replay code (`icho replay <hash>`).
- 📼 Record once, replay forever
- ⚡ Replay, Record, Auto, Live, and **Regression** execution modes
- 🧰 Tool Calling & Function Calling support
- 🌊 Streaming support (sync & async)
- 🔒 PII & Sensitive Information Masking
- 🛡️ NVIDIA NeMo Guardrails Integration
- 🧠 Deterministic request hashing
- 🎯 Custom ignored fields
- 🔧 Custom request normalizers
- 🗂️ File, Memory & PostgreSQL storage backends
- 🧹 CLI utilities (`regression`, `diff`, `search`, `replay`, `log`, `stats`, `inspect`, `clean`)

---

# Searchable AI Executions & Instant Replay

When a bug is reported (*"The AI gave the wrong answer yesterday"*), Icho makes it reproducible in seconds:

```bash
# 1. Search past executions by TF-IDF cosine similarity & relative time
icho search "refund request failed" --since yesterday -i

# 2. Inspect target execution and get instant Python replay code
icho replay 1cea06570793

# 3. View chronological Git-like execution history
icho log --path cassettes -n 5
```

Or query programmatically via Python API:

```python
from icho import search_cassettes

# Search recorded executions by natural language & metadata
results = search_cassettes(query="refund request", since="yesterday", provider="openai")

for res in results:
    print(f"Match Score: {res.score:.4f} | Hash: {res.hash[:12]}")
    print(f"  Input:  {res.input_snippet}")
    print(f"  Output: {res.output_snippet}")
```

---

# CLI Reference

Icho includes a built-in CLI to search, inspect, format, and debug your recorded cassettes:

| Command | Description | Example |
| :--- | :--- | :--- |
| `icho diff` | Compare two cassette executions (text, markdown, html formats) | `icho diff hash1 hash2 -f html -o diff.html` |
| `icho search` | Search executions by vector cosine similarity, time window, or provider/model | `icho search "refund" --since yesterday -i` |
| `icho log` | Show Git-like chronological execution log | `icho log --path cassettes -n 10` |
| `icho replay` | Inspect a target cassette and generate copy-paste Python replay code | `icho replay 1cea06570793` |
| `icho stats` | View total cassette count, disk size, and saved API latency | `icho stats --path cassettes` |
| `icho inspect` | List all saved cassettes with providers, models, and timestamps | `icho inspect --path cassettes` |
| `icho clean` | Redact volatile timestamps and latency before committing to Git | `icho clean --remove-latency --remove-timestamps` |

---

# Cassette Execution Diffing (`icho diff`) 🔍

Compare any two recorded cassettes by hash, ID, or file path to inspect prompt, model, parameter, or output differences:

```bash
# 1. Compare two cassette executions in terminal text mode
icho diff 1cea06570793 4b93d6e3 --path cassettes

# 2. Export diff comparison report to GitHub Markdown
icho diff 1cea06570793 4b93d6e3 -f markdown -o diff.md

# 3. Export standalone styled HTML diff report
icho diff 1cea06570793 4b93d6e3 -f html -o diff.html

# 4. Interactive Diff from Search Results
icho search "refund request" -i
# Enter result numbers to diff (e.g., '1,2' or 'diff 1 2'):
```

---

# Phase 2: AI Regression Testing Engine 🧪

Replay testing validates that an app runs deterministically against recorded cassettes. **Phase 2 Regression Testing** compares an **Old Cassette** (reference run) against a **New Execution** (live call or updated model) across **5 distinct analysis dimensions**:

$$\text{Old Cassette} \longrightarrow \text{New Execution} \longrightarrow \begin{cases} \text{1. 📝 Prompt Diff} \\ \text{2. 🛠️ Tool Diff} \\ \text{3. 🧠 Semantic Diff} \\ \text{4. 💰 Cost Diff} \\ \text{5. ⚡ Latency Diff} \end{cases}$$

### Python API

```python
from icho import compare_executions, cassette

# Programmatic 5-dimension comparison
report = compare_executions("tests/cassettes/ref_run.json", "tests/cassettes/new_run.json")

print(f"Semantic Similarity: {report.semantic_diff.similarity_score * 100:.1f}%")
print(f"Token Delta: {report.cost_diff.token_delta['total']:+d} tokens")
print(f"Latency Delta: {report.latency_diff.delta_ms:+.1f} ms")

# Enforce CI assertion rules
report.assert_no_regression(
    similarity_threshold=0.85,
    allow_tool_changes=False,
    max_cost_increase_pct=15.0,
    max_latency_increase_pct=25.0,
)

# Inline cassette execution with mode="regression"
with cassette(path="tests/cassettes/support_flow.json", mode="regression") as cas:
    response = model.invoke("How do I request a refund?")
    reg_report = cas.regression_report
    print(reg_report.render_text())
```

### CLI Command

```bash
# Run 5-dimension regression test comparing two cassette recordings
icho regression reference_run.json new_run.json

# Fail CI build if semantic output drifts below 85% threshold
icho regression reference_run.json new_run.json --fail-on-drift --threshold 0.85

# Export GitHub Markdown or HTML regression report
icho regression old_hash new_hash -f markdown -o regression_report.md
icho regression old_hash new_hash -f html -o regression_report.html
```

---

# AI Test Datasets (`icho test`) 🧪

Organize thousands of conversation scenarios into domain directories using `.yaml` or `.json` cassettes:

```text
tests/
├── customer_support/
│   ├── greeting.yaml
│   ├── refund.yaml
│   └── complaint.yaml
├── travel_agent/
│   ├── booking.yaml
│   └── cancellation.yaml
└── finance/
    └── invoices.yaml
```

### CLI Command
```bash
# Run batch test suite across a domain directory
icho test tests/customer_support/

# Run multi-domain dataset with parallel worker pool
icho test tests/ -j 4

# Run regression suite across datasets against reference cassettes
icho test tests/customer_support/ --mode regression --threshold 0.85 --fail-on-drift
```

### Python API
```python
from icho import discover_suite, SuiteRunner

# Discover nested test suite hierarchy
suite = discover_suite("tests/customer_support")

# Execute batch test suite
runner = SuiteRunner(mode="replay")
report = runner.run(suite)

print(report.render_text())
```

---

# Pytest Integration 🧪

Icho includes built-in Pytest support via the `icho` plugin.

### Markers & Fixtures

Use `@pytest.mark.icho` (or alias `@pytest.mark.sequa` / `@pytest.mark.cassette`) or inject the `icho_cassette` fixture:

```python
import pytest
from langchain_groq import ChatGroq

@pytest.mark.icho(mode="auto")
def test_llm_feature():
    model = ChatGroq(model_name="llama-3.1-8b-instant")
    response = model.invoke("Say hello")
    assert "hello" in response.content.lower()

def test_with_fixture(icho_cassette):
    model = ChatGroq(model_name="llama-3.1-8b-instant")
    response = model.invoke("Hello world")
```

### Pytest CLI Flags

| Flag | Description | Default |
| :--- | :--- | :--- |
| `--icho-mode=<mode>` | Globally override cassette mode (`auto`, `record`, `replay`, `live`) | Marker / `auto` |
| `--icho-path=<path>` | Set base cassette directory | `tests/cassettes` |
| `--icho-mask-pii` | Enable PII masking across all test cassette recordings | `False` |

---

# GitHub Actions Integration 🐙

Run deterministic LLM snapshot tests in CI/CD with zero API costs using the official Icho GitHub Action.

### Quick Workflow Setup

Add `.github/workflows/ai-tests.yml` to your repository:

```yaml
name: AI Snapshot Tests

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Icho LLM Tests
        uses: thetechnoadvisor/llmcassette@main
        with:
          mode: replay
          cassette-path: tests/cassettes
          post-summary: true
```

### Action Options

| Input | Description | Default |
| :--- | :--- | :--- |
| `mode` | Execution mode (`replay`, `auto`, `record`, `live`) | `replay` |
| `cassette-path` | Path to saved cassette files directory | `tests/cassettes` |
| `python-version` | Python version for setup | `3.12` |
| `pytest-args` | Additional arguments passed to pytest | `""` |
| `post-summary` | Post cassette stats report to `$GITHUB_STEP_SUMMARY` | `true` |

---

# Common Use Cases

### 🚀 Speed up AI integration tests

Run your test suite in milliseconds instead of waiting for repeated LLM calls.

---

### 💰 Reduce API costs

Replay previously recorded executions without paying for another API request.

---

### 🧪 Deterministic testing

Replay the exact same execution every time.

---

### 💻 Offline development

Develop and test AI applications without internet connectivity.

---

### 🐞 Reproduce bugs

Replay the exact LLM interaction that caused the issue.

---

# Storage Backends

Icho supports multiple storage backends:

- 📁 File Storage
- 🧠 In-Memory Storage
- 🐘 PostgreSQL Storage

Choose whichever fits your workflow.

---

# Contributing

Contributions are always welcome.

- ⭐ Star the repository
- 🐞 Report bugs
- 💡 Suggest new features
- 🔧 Open a Pull Request

---

# License

MIT License.