Metadata-Version: 2.4
Name: chatsee-connector-testing
Version: 0.1.0
Summary: Shared offline test kit for ChatSee connectors (fake tracker + contract assertions).
Maintainer-email: ChatSee <contact@chatsee.ai>
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"

# chatsee-connector-testing

Shared offline test kit for ChatSee connectors.

A connector hooks one framework and drives `chatsee.ChatseeTracker`. This package
lets you test all of that with **no network, no API key, no framework account and
no dev tenant** — so capture bugs are found in seconds, not in a live 2×2 run.

```bash
pip install -e python/connectors/_shared
```

## The idea

Don't mock ChatSee. **Mock the framework.**

Your connector's test supplies a small stub of the framework that replays a
canned turn, instruments it with a `FakeTracker`, and then asserts what reached
the tracker.

```python
from chatsee_connector_testing import FakeTracker, assert_contract

tracker = FakeTracker()
chatsee = instrument_google_adk(agent_id="a", tenant_id="t", _tracker=tracker)
run_stub_agent(chatsee.plugin)

assert_contract(tracker, expected_turns=1,
                expected_tools=["get_order_status"],
                framework="google-adk")
```

Every connector accepts a private `_tracker=` keyword so tests can inject the
fake. Customers never see it — they pass the normal identity arguments and the
connector builds a real tracker itself.

## What `assert_contract` checks

| Rule | The bug it catches |
| --- | --- |
| One `start_turn`/`end_turn` per turn, correctly nested | Pydantic AI's `run_sync` delegating to `run`; Vercel emitting 10 for 4; CrewAI capturing only the first kickoff |
| `bot_message` non-empty | OpenAI Agents streaming — final text in no span at all |
| Each tool once, with args and a result **or** an error | Tool captured but its outcome lost |
| `log_model_call` once per model call | Connectors that sum tokens themselves and destroy the per-call grain |
| Absent is never `0` or `""` | An unmeasured turn looking like a free one |
| Required metadata present, tokens/model/tools **not** in metadata | Tokens buried in metadata are invisible to the analytics screens |

## Behavioural checks

These take callables rather than a tracker, because they're about how the
connector behaves rather than what it recorded.

```python
assert_fails_open(run_one_turn)                  # ChatSee must never break the agent
assert_instrument_is_idempotent(run_turns)       # instrumenting twice is a no-op
assert_concurrent_runs_isolated(run_one_turn)    # state keyed per run, not per instance
```

`assert_concurrent_runs_isolated` is the one most likely to fail first: Google
ADK, LangChain and Strands all run concurrent invocations against a single
instrumented object, and the ADK POC plugin holds one shared accumulator.

## Running its own tests

The suite has a self-test: a correct toy connector that must satisfy every rule,
and a deliberately broken one per rule that must trip it. An assertion nobody has
seen fail is not evidence of anything.

```bash
pip install -e "python/connectors/_shared[dev]"
pytest python/connectors/_shared/tests -q
```
