Metadata-Version: 2.4
Name: stubllm
Version: 0.1.0
Summary: Deterministic mock server for LLM APIs. Test your AI code without spending tokens.
Author: mockllm contributors
License: MIT
License-File: LICENSE
Keywords: anthropic,gemini,llm,mock,openai,stub,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: anyio>=4.0.0
Requires-Dist: click>=8.1.0
Requires-Dist: fastapi>=0.110.0
Requires-Dist: httpx>=0.26.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: uvicorn[standard]>=0.27.0
Provides-Extra: dev
Requires-Dist: httpx>=0.26.0; extra == 'dev'
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: openai>=1.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.3.0; extra == 'dev'
Provides-Extra: pytest
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'pytest'
Requires-Dist: pytest>=7.0; extra == 'pytest'
Description-Content-Type: text/markdown

# stubllm

**Deterministic mock server for LLM APIs. Test your AI code without spending tokens.**

[![CI](https://github.com/your-org/stubllm/actions/workflows/ci.yml/badge.svg)](https://github.com/your-org/stubllm/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/stubllm)](https://pypi.org/project/stubllm/)
[![Python](https://img.shields.io/pypi/pyversions/stubllm)](https://pypi.org/project/stubllm/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Works with: **OpenAI** · **Anthropic** · **Google Gemini**

---

## 30-second quickstart

```bash
# 1. Install
pip install stubllm

# 2. Create a fixture
mkdir fixtures
cat > fixtures/chat.yaml << 'EOF'
fixtures:
  - name: "greeting"
    match:
      provider: openai
      messages:
        - role: user
          content:
            contains: "hello"
    response:
      content: "Hello! How can I help you today?"
EOF

# 3. Start the server
stubllm serve --port 8765

# 4. Point your code at it
export OPENAI_BASE_URL=http://localhost:8765/v1/
python your_app.py  # no real API calls, no tokens spent
```

---

## Why stubllm?

| | stubllm | Real API | Ollama |
|---|---|---|---|
| Cost | Free | Paid | Free |
| Speed | <1ms | 1-30s | 5-30s |
| Deterministic | ✅ | ❌ | ❌ |
| Works offline | ✅ | ❌ | ✅ |
| No GPU needed | ✅ | ✅ | ❌ |
| Pytest integration | ✅ | ❌ | ❌ |
| Fixtures / record-replay | ✅ | ❌ | ❌ |
| CI-friendly | ✅ | Slow/expensive | Heavy |

---

## Installation

```bash
pip install stubllm

# With pytest support
pip install "stubllm[pytest]"
```

---

## Fixture format

Fixtures are YAML (or JSON) files that map request patterns to responses.

### Basic text response

```yaml
fixtures:
  - name: "greeting"
    match:
      provider: openai          # openai | anthropic | gemini | any
      endpoint: /v1/chat/completions   # optional
      model: "gpt-4o"           # optional
      messages:
        - role: user
          content:
            contains: "hello"   # exact | contains | regex
    response:
      content: "Hello! How can I help you today?"
      usage:
        prompt_tokens: 10
        completion_tokens: 12
        total_tokens: 22
```

### Tool call response

```yaml
fixtures:
  - name: "weather_tool"
    match:
      provider: openai
      messages:
        - role: user
          content:
            contains: "weather"
      tools_present: true        # only match when tools are provided
    response:
      tool_calls:
        - id: "call_abc123"
          type: function
          function:
            name: "get_weather"
            arguments: '{"location": "Amsterdam"}'
```

### Streaming with delay

```yaml
fixtures:
  - name: "slow_story"
    match:
      messages:
        - role: user
          content:
            contains: "story"
    response:
      content: "Once upon a time..."
      stream_chunk_delay_ms: 50   # simulate realistic streaming speed
```

### Error response

```yaml
fixtures:
  - name: "rate_limit"
    match:
      messages:
        - role: user
          content:
            contains: "trigger_error"
    response:
      content: '{"error": {"message": "Rate limit exceeded", "type": "rate_limit_error"}}'
      http_status: 429
```

### Content match strategies

```yaml
# Exact match (highest priority)
content:
  exact: "Hello, world!"

# Substring match (case-insensitive)
content:
  contains: "weather"

# Regular expression
content:
  regex: "tell me.*joke"
```

### Matching priority

Higher specificity = higher priority. When multiple fixtures match, the most specific wins:

1. `exact` message content (score: +10)
2. `contains` message content (score: +5)
3. `regex` message content (score: +4)
4. `model` specified (score: +2)
5. `tools_present` specified (score: +2)
6. `provider` specified (score: +1)
7. Fallback (no match criteria)

---

## Pytest plugin

### Basic setup

```python
# conftest.py — nothing needed, stubllm auto-registers as a pytest plugin
# The `stubllm_server` fixture is available automatically after installing stubllm
```

```python
# test_my_app.py
import openai
from stubllm.pytest_plugin import use_fixtures

@use_fixtures("fixtures/chat.yaml")
def test_greeting(stubllm_server):
    client = openai.OpenAI(
        base_url=stubllm_server.openai_url,  # includes /v1/ — openai SDK needs this
        api_key="test-key"
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "hello world"}]
    )
    assert "Hello" in response.choices[0].message.content
    assert stubllm_server.call_count == 1
```

### Assertion helpers

```python
def test_with_assertions(stubllm_server):
    # ... make some calls ...

    # Assert specific prompt was sent
    stubllm_server.assert_called_with_prompt("hello")

    # Assert number of calls
    assert stubllm_server.call_count == 2

    # Inspect all calls
    for call in stubllm_server.calls:
        print(call["path"], call["body"])

    # Assert last call path
    stubllm_server.assert_last_call_path("/v1/chat/completions")
```

### Multiple fixture files

```python
@use_fixtures("fixtures/chat.yaml", "fixtures/tools.yaml")
def test_combined(stubllm_server):
    ...  # both fixture files are active for this test
```

---

## Multi-provider support

### OpenAI

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:8765/v1/",  # note: /v1/ required — the OpenAI SDK does not add it
    api_key="test-key"
)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "hello"}]
)
```

### Anthropic

```python
import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:8765",  # Anthropic SDK adds /v1/ itself
    api_key="test-key"
)
message = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "hello"}]
)
```

### Google Gemini

```python
import google.generativeai as genai

genai.configure(
    api_key="test-key",
    client_options={"api_endpoint": "localhost:8765"}
)
model = genai.GenerativeModel("gemini-pro")
response = model.generate_content("hello")
```

---

## Streaming

All providers support streaming. Fixtures work identically — streaming is controlled by the `stream: true` parameter in the request, not the fixture.

```python
# OpenAI streaming
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "hello"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content, end="", flush=True)
```

Control streaming speed in fixtures:
```yaml
response:
  content: "A long streaming response..."
  stream_chunk_delay_ms: 20   # default: 20ms between chunks
```

---

## Record and replay

Record real API interactions for later replay:

```bash
# Start in record mode (proxies to real OpenAI, saves fixtures)
stubllm record \
  --target https://api.openai.com \
  --fixture-dir ./recorded_fixtures

# Run your app against the recording proxy
OPENAI_BASE_URL=http://localhost:8765/v1/ python your_app.py

# Fixtures are saved to ./recorded_fixtures/
ls recorded_fixtures/
# recorded_hello_world_1706000000.yaml
# recorded_weather_query_1706000001.yaml
```

Recorded fixtures are sanitized (API keys removed) and can be committed to your repo.

---

## CLI reference

```bash
# Start server (auto-loads ./fixtures/ if it exists)
stubllm serve

# Custom port and fixture directory
stubllm serve --port 9000 --fixture-dir ./my-fixtures

# Multiple fixture directories
stubllm serve --fixture-dir ./fixtures/openai --fixture-dir ./fixtures/anthropic

# Individual fixture files
stubllm serve --fixture-file chat.yaml --fixture-file tools.yaml

# Record mode
stubllm record --target https://api.openai.com --fixture-dir ./recorded

# Version
stubllm --version
```

---

## Structured output (JSON schema)

When `response_format: { type: "json_schema" }` is set, stubllm validates that the fixture response is valid JSON. If it's not, it wraps the content automatically.

```yaml
fixtures:
  - name: "structured"
    match:
      provider: openai
    response:
      content: '{"name": "Alice", "age": 30}'  # must be valid JSON
```

---

## Project structure

```
stubllm/
├── src/stubllm/
│   ├── fixtures/       # YAML/JSON loading, Pydantic models, matching engine
│   ├── providers/      # OpenAI, Anthropic, Gemini endpoint handlers
│   ├── streaming/      # SSE streaming simulation
│   ├── recorder/       # Record-and-replay proxy
│   ├── pytest_plugin/  # pytest fixtures and @use_fixtures decorator
│   ├── server.py       # FastAPI app factory
│   └── cli.py          # click CLI
├── tests/              # >80% coverage
└── examples/           # Working examples (basic + advanced)
```

---

## Contributing

```bash
git clone https://github.com/your-org/stubllm
cd stubllm
pip install -e ".[dev]"
pytest tests/ -v
```

---

## License

MIT
