Metadata-Version: 2.4
Name: kalytera
Version: 0.1.5
Summary: Real-time quality monitoring and failure detection for production AI agents
Project-URL: Homepage, https://kalytera.ai
Project-URL: Documentation, https://github.com/priyamathur/kalytera-server#readme
Project-URL: Repository, https://github.com/priyamathur/kalytera-server
Project-URL: Bug Tracker, https://github.com/priyamathur/kalytera-server/issues
Author-email: Kalytera <priya@kalytera.ai>
License: MIT
Keywords: agents,ai,evaluation,llm,monitoring,observability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Requires-Dist: aiohttp>=3.9.0
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest>=8.1.0; extra == 'dev'
Requires-Dist: requests>=2.31.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: server
Requires-Dist: alembic==1.13.1; extra == 'server'
Requires-Dist: anthropic>=0.40.0; extra == 'server'
Requires-Dist: fastapi==0.110.0; extra == 'server'
Requires-Dist: google-generativeai>=0.8.0; extra == 'server'
Requires-Dist: numpy>=1.24.0; extra == 'server'
Requires-Dist: openai>=1.0.0; extra == 'server'
Requires-Dist: pandas==2.0.3; extra == 'server'
Requires-Dist: plotly==5.19.0; extra == 'server'
Requires-Dist: psycopg2-binary==2.9.9; extra == 'server'
Requires-Dist: pydantic==2.5.3; extra == 'server'
Requires-Dist: python-dotenv==1.0.1; extra == 'server'
Requires-Dist: python-multipart==0.0.27; extra == 'server'
Requires-Dist: sqlalchemy==2.0.27; extra == 'server'
Requires-Dist: streamlit>=1.37.0; extra == 'server'
Requires-Dist: stripe>=8.0.0; extra == 'server'
Requires-Dist: uvicorn[standard]==0.27.1; extra == 'server'
Description-Content-Type: text/markdown

# Kalytera

Know when your AI agent fails — before your users do.

```bash
pip install kalytera
```

```python
import kalytera

kalytera.configure(api_key="kly_live_...")

kalytera.trace(
    session_id="session-001",
    step_number=1,
    step_name="classify_intent",
    input="I need to cancel my subscription",
    output="I can help with that. What's the reason?",
)
```

That's it. Kalytera scores every step with an LLM judge and surfaces failure patterns — what's breaking, at which workflow step, and why.

![Kalytera dashboard — quality scores, failure patterns, and trace viewer](https://raw.githubusercontent.com/priyamathur/kalytera-server/main/site/demo.gif)

→ **[Live demo dashboard](https://app.kalytera.dev)** · **[Get an API key](#get-started)**

---

## Get started

**1. Get a free API key** (no credit card):

```bash
curl -s -X POST https://api.kalytera.dev/signup \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com", "name": "my-agent"}' | python3 -m json.tool
```

Copy the `api_key` from the response (looks like `kly_live_...`).

**2. Install and trace:**

```python
import kalytera

kalytera.configure(
    api_key="kly_live_...",
    api_endpoint="https://api.kalytera.dev",
    agent_id="my-agent",
)

# Call after every step in your agent workflow
kalytera.trace(
    session_id="session-001",   # same ID groups steps of one conversation
    step_number=1,
    step_name="retrieve_policy",
    input="What is the cancellation policy?",
    output="You can cancel anytime. No refunds for partial months.",
    tool_calls=[{"name": "policy_api", "success": True, "latency_ms": 210}],
)
```

`trace()` returns in under 5ms. It never raises. If the server is unreachable, your agent keeps running.

**3. Open the dashboard** — quality scores appear within 30 seconds:

```
https://app.kalytera.dev
```

---

## What Kalytera detects

Every step is scored across six dimensions by an LLM judge (Claude Haiku):

| Dimension | Default weight | What it measures |
|---|---|---|
| Accuracy | 25% | Contextually correct answer for the situation |
| Goal alignment | 25% | Agent stayed on what the user actually needed |
| Decision quality | 15% | Reasoning was sound, right tools were chosen |
| Completeness | 15% | Request fully resolved end-to-end |
| Helpfulness | 10% | Response had practical value the user could act on |
| Factuality | 10% | All claims grounded and true — no hallucinations |

Steps scoring below 70% are flagged. Seven failure types are detected automatically:

`wrong_answer` · `tool_failure` · `goal_drift` · `hallucination` · `context_loss` · `incomplete` · `loop`

Weights and pass threshold are configurable per agent in the dashboard. You can also add custom metrics (e.g. `helpfulness`, `tone`, `compliance`) with their own weights.

---

## Calibrate the judge

Every judge has bias. Calibration measures whether Kalytera's LLM judge agrees with your own judgement.

In the Trace Viewer, label any session with **👍 passed** or **👎 failed**. The dashboard shows judge accuracy against your labels and flags when weights need adjustment:

| Agreement | Status |
|---|---|
| ≥ 90% | Calibrated ✓ |
| 80–89% | Good |
| < 80% | Review your scoring weights |

Label 10–20 sessions to get a reliable calibration reading. The more you label, the more precisely Kalytera can tell you if the judge is drifting.

---

## `@watch` decorator

Zero-config alternative — wraps a function and captures input, output, and latency:

```python
@kalytera.watch
def handle_request(user_input: str) -> str:
    return your_agent_logic(user_input)
```

---

## Self-host

For teams that want data on their own infrastructure:

```bash
git clone https://github.com/priyamathur/kalytera-server
cd kalytera-server
cp .env.example .env        # add ANTHROPIC_API_KEY and DATABASE_URL
docker compose up
```

API at `http://localhost:8000`, dashboard at `http://localhost:8501`.

---

## Roadmap

- [x] LLM-as-judge scoring (4 built-in dimensions)
- [x] Custom metrics (helpfulness, tone, compliance, …)
- [x] Golden set calibration — label sessions, measure judge accuracy
- [ ] Cross-model agreement — flag low-confidence evals by comparing two judges
- [ ] Auto-calibration — suggest weight adjustments when agreement drops below threshold
- [ ] Team review queue — route low-confidence evals to a human reviewer

---

## License

MIT — [priya@kalytera.ai](mailto:priya@kalytera.ai)
