Metadata-Version: 2.4
Name: llm-observer-proxy
Version: 0.1.0
Summary: Run-scoped LLM observability control plane backed by LiteLLM
Author-email: Shuqiao Zhang <stevenjoezhang@gmail.com>, Yue Xiong <xxyyyue@gmail.com>
Maintainer-email: Shuqiao Zhang <stevenjoezhang@gmail.com>, Yue Xiong <xxyyyue@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/xxyyue/llm-observer-proxy
Project-URL: Documentation, https://github.com/xxyyue/llm-observer-proxy/blob/main/docs/control-plane-api.md
Project-URL: Repository, https://github.com/xxyyue/llm-observer-proxy.git
Project-URL: Issues, https://github.com/xxyyue/llm-observer-proxy/issues
Project-URL: Changelog, https://github.com/xxyyue/llm-observer-proxy/blob/main/docs/CHANGELOG.md
Keywords: llm,litellm,observability,proxy,agents
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: Proxy Servers
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: aiohttp==3.14.1
Requires-Dist: apscheduler==3.11.3
Requires-Dist: backoff==2.2.1
Requires-Dist: boto3==1.43.38
Requires-Dist: cryptography==46.0.7
Requires-Dist: fastapi==0.138.2
Requires-Dist: fastapi-sso==0.21.1
Requires-Dist: litellm==1.90.2
Requires-Dist: orjson==3.11.9
Requires-Dist: pydantic-settings==2.14.2
Requires-Dist: python-multipart==0.0.32
Requires-Dist: pyyaml==6.0.3
Requires-Dist: redis==8.0.1
Requires-Dist: uvicorn==0.49.0
Requires-Dist: uvloop==0.22.1; sys_platform != "win32"
Requires-Dist: websockets==16.0
Provides-Extra: dev
Requires-Dist: build<2,>=1.2.2; extra == "dev"
Requires-Dist: pytest<10,>=9.0.3; extra == "dev"
Dynamic: license-file

# LLM Observer Proxy

[![CI](https://github.com/xxyyue/llm-observer-proxy/actions/workflows/ci.yml/badge.svg)](https://github.com/xxyyue/llm-observer-proxy/actions/workflows/ci.yml)

LLM Observer Proxy runs a temporary LiteLLM data plane for each evaluation or agent run and stores its observable LLM traffic under a stable `run_id`. It records requests, responses, streaming output, tool calls, token usage, cost estimates, errors, and provider-visible reasoning content.

It is independently installable and can be embedded into any evaluation platform, agent runner, or orchestration service through its HTTP API.

## How It Works

There are two HTTP layers:

1. The **control plane** listens on port `8790` by default. An orchestrator uses it to create, inspect, and stop run-scoped proxies.
2. Each **data plane** listens on a separate, automatically allocated port. An agent uses the returned URL as an OpenAI-compatible or Anthropic-compatible LLM endpoint.

One control-plane process can manage multiple concurrent data planes. Stopping a data plane preserves its history on disk, and history remains readable after the control plane restarts.

The observer can only record information exposed by the upstream API. It cannot obtain hidden chain-of-thought that a provider does not return.

## Requirements

- Python 3.11, 3.12, or 3.13
- A provider API key or a custom LiteLLM configuration
- Local permission to start subprocesses and bind data-plane ports

## Install

Install the command in the current Python environment:

```bash
python -m pip install llm-observer-proxy
```

Or install it as an isolated command with `uv`:

```bash
uv tool install llm-observer-proxy
```

Run it directly without a permanent installation:

```bash
OPENAI_API_KEY=your-openai-api-key uvx llm-observer-proxy
```

Because the distribution name and console command are both `llm-observer-proxy`, `uvx` resolves the package and starts the server in one command. It caches the isolated environment for subsequent runs.

For a source checkout, install the locked development environment instead:

```bash
uv sync --extra dev
```

LiteLLM is pinned because the observer callback adapts specific response and streaming internals. The package declares the subset of LiteLLM Proxy runtime dependencies needed by the tested LLM data plane rather than installing the full `litellm[proxy]` extra.

## Quick Start

The bundled catalog includes `gpt-5.5` and reads the OpenAI credential from the environment:

```bash
export OPENAI_API_KEY=your-openai-api-key
llm-observer-proxy
```

In another terminal, check the control plane:

```bash
curl http://127.0.0.1:8790/healthz
```

Create a data plane for a run:

```bash
curl -X POST http://127.0.0.1:8790/api/runs/run-001/proxy \
  -H 'Content-Type: application/json' \
  -d '{}'
```

The data plane serves every model in the selected catalog or custom LiteLLM configuration. Each LLM request selects a route through its own `model` field. The create response includes a dynamically allocated bare `base_url`, for example `http://127.0.0.1:45678`, plus `openai_base_url` (`http://127.0.0.1:45678/v1`) and `anthropic_base_url` (`http://127.0.0.1:45678`). Use the explicit URL for the client SDK, or call an endpoint directly:

```bash
curl http://127.0.0.1:45678/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}]
  }'
```

Read the normalized history:

```bash
curl http://127.0.0.1:8790/api/runs/run-001/history
```

Stop the data plane without deleting its history:

```bash
curl -X DELETE http://127.0.0.1:8790/api/runs/run-001/proxy
```

## Configuration

The command accepts these options:

```text
--host HOST                 Control-plane bind host (default: 127.0.0.1)
--port PORT                 Control-plane port (default: 8790)
--env-file PATH             KEY=VALUE file loaded before startup (default: .env)
--artifact-root PATH        Artifact storage root
--model-catalog PATH        Model catalog JSON
--log-level LEVEL           Python log level
```

Paths are resolved in this order:

1. `--artifact-root` and `--model-catalog`
2. `LLM_OBSERVER_PROXY_ARTIFACT_ROOT` and `LLM_OBSERVER_PROXY_MODEL_CATALOG_PATH`
3. `./artifacts` and `./config/models.json` in the working directory
4. The package's secret-free bundled model catalog

Relative paths are resolved against the process working directory. `LITELLM_BIN` can select a specific LiteLLM executable.

### Model Catalog

A model catalog has this shape:

```json
{
  "llm_observer_proxy": {
    "base_url": "https://api.openai.com",
    "api_key": "os.environ/OPENAI_API_KEY",
    "litellm_settings": {
      "drop_params": true
    }
  },
  "models": [
    {
      "id": "gpt-5.5",
      "upstream_api": "openai"
    }
  ]
}
```

`id` is the public model alias exposed by the data plane. `upstream_api` selects LiteLLM's `openai` or `anthropic` provider adapter. Optional input and output token prices are used for local cost estimates. Other catalog properties, including `name` and `provider`, are accepted but ignored. Environment references use LiteLLM's `os.environ/NAME` syntax, which prevents a literal credential from being written to the generated configuration. See `config/models.example.json` for a complete example.

For advanced provider routing, pass a server-local LiteLLM YAML path as `litellm_config_path` in the create request. The observer callback is added to that configuration automatically.

## Control API

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/healthz` | Check control-plane health |
| `GET` | `/api/proxies` | List process-local running and archived proxies |
| `POST` | `/api/runs/{run_id}/proxy` | Create a run-scoped data plane |
| `GET` | `/api/runs/{run_id}/proxy` | Read proxy state |
| `DELETE` | `/api/runs/{run_id}/proxy` | Stop a data plane |
| `GET` | `/api/runs/{run_id}/history` | Read normalized history |
| `GET` | `/api/runs/{run_id}/conversation` | Read the latest conversation |

The create payload may be empty. Optional fields are `port`, `bind_host`, `litellm_config_path`, `litellm_env`, and `strip_callbacks`. Unknown fields are ignored. The complete request and response contract is documented in `docs/control-plane-api.md`.

## Artifacts

Each run writes:

```text
artifacts/runs/<run_id>/platform/llm/
  config.yaml
  conversation_snapshot.json
  events.jsonl
  litellm.log
```

The generated YAML is retained as a run artifact. Keep provider credentials in environment variables rather than writing literal values into a model catalog or custom YAML.

## Supported Scope

The supported data-plane surface is LiteLLM's OpenAI-compatible LLM API and Anthropic `/v1/messages`, plus the control and history APIs documented above. LiteLLM MCP routes, its administration UI, database persistence, and enterprise gateway features are not part of this project's compatibility contract. In particular, the `mcp` package is intentionally not installed because MCP routes are not used by the observer and are lazy-loaded by the pinned LiteLLM version.

## Troubleshooting

- **The control plane starts, but proxy creation returns `502`:** inspect the returned `log_path` or the run's `litellm.log`. Missing provider variables and invalid custom YAML are the most common causes.
- **LiteLLM cannot be found:** install the package and command in the same environment, or set `LITELLM_BIN` to an executable path.
- **Proxy creation returns `409`:** the `run_id` already has a running data plane, a requested port is unavailable, or the selected configuration is invalid.
- **History is empty:** confirm the agent is using the returned data-plane URL, not the provider URL directly.

## Security

The control API is an administrative interface: it can start subprocesses, select listening ports, pass child environment values, and read server-local LiteLLM configurations. Version 0.1.x has no built-in authentication. Bind it to loopback or a trusted private network and read `.github/SECURITY.md` before remote deployment.

History can contain prompts, model output, tool output, and credentials. Apply appropriate filesystem permissions, access control, and retention policies to the artifact root.

## Development

```bash
uv sync --extra dev
uv run pytest
LLM_OBSERVER_PROXY_RUN_INTEGRATION=1 uv run pytest tests/test_integration_proxy.py
uv build
uvx twine check dist/*
```

## License

Apache-2.0. See `LICENSE` and `NOTICE`.

## Authors

- Shuqiao Zhang <stevenjoezhang@gmail.com>
- Yue Xiong <xxyyyue@gmail.com>
