Metadata-Version: 2.4
Name: llm-analyzer
Version: 0.1.0
Summary: Universal, provider-agnostic local LLM response analyzer with SQLite storage and a local dashboard.
Author: Pabolu Venkata Subrahmanya Dheeraj
License: MIT
Keywords: llm,llm-observability,llm-monitoring,llm-evaluation,ai,generative-ai,rag,local-dashboard
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# LLM-Analyzer

Universal, provider-agnostic local LLM response analyzer.

LLM-Analyzer captures LLM responses, extracts available metadata, stores telemetry locally in SQLite, and provides a local dashboard.

## Features

- Provider-agnostic response capture
- Model and provider metadata extraction when available
- Token usage extraction when available
- Latency measurement
- Error capture
- Temporary local SQLite storage
- Automatic local dashboard at `http://127.0.0.1:1918`
- Optional response evaluation
- CSV export
- Python API and CLI

The analyzer does not require an OpenAI, Gemini, Groq, Claude, or other provider SDK.

## Install

```bash
pip install llm-analyzer
```

## Basic usage

```python
from llm_analyzer import analyze

response = my_llm_call(prompt)

analyze(response)

return response
```

Or measure the complete call automatically:

```python
from llm_analyzer import analyze_call

response = analyze_call(my_llm_call, prompt)

return response
```

The first analysis call automatically starts the local dashboard.

Open:

```text
http://127.0.0.1:1918
```

## Local storage

By default telemetry is stored in:

```text
.llm-analyzer/telemetry.db
```

Captured data is temporary by default and uses a 7-day retention period.

You can change the retention period:

```python
from llm_analyzer import Analyzer

analyzer = Analyzer(retention_days=3)
```

Or with an environment variable:

```text
LLM_ANALYZER_RETENTION_DAYS=3
```

## Disable automatic dashboard startup

Set:

```text
LLM_ANALYZER_AUTO_DASHBOARD=0
```

## CLI

Start the dashboard manually:

```bash
llm-analyzer dashboard
```

Print the database path:

```bash
llm-analyzer path
```

## Evaluation

Evaluation is optional. The package can use supplied prompt/reference/context evidence or a custom evaluator.

Quality metrics such as accuracy, hallucination, bias, and fairness should be treated as evaluation results, not as metadata automatically inferred from an arbitrary LLM response.

## License

MIT
