Metadata-Version: 2.4
Name: polarity-keystone
Version: 0.2.5
Summary: Python SDK client for the Keystone agent evaluation + sandboxed-execution platform
Author: Polarity
License: MIT
Project-URL: Homepage, https://github.com/Polarityinc/keystone-sdk-python
Project-URL: Repository, https://github.com/Polarityinc/keystone-sdk-python
Project-URL: Issues, https://github.com/Polarityinc/keystone-sdk-python/issues
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 4 - Beta
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Dynamic: author
Dynamic: requires-python

# Keystone SDK for Python

Python client for the [Keystone](https://keystone.polarity.cc) agent
evaluation + sandboxed-execution platform. Shares a single source of truth
with the [TypeScript](https://github.com/Polarityinc/keystone-sdk-js) and
[Go](https://github.com/Polarityinc/keystone-sdk-go) SDKs — byte-identical
cost estimates and prompt rendering across all three.

## Install

```bash
pip install polarity-keystone
```

Zero external runtime dependencies — uses only `urllib` from the standard
library.

## 60-second quick start: `Eval()`

The shortest path from "I have an agent" to "I have an evaluation":

```python
from polarity_keystone import Eval, Factuality, AnswerRelevancy

result = Eval(
    "summarisation-quality",
    data=[
        {"input": "Long article about whales...", "expected": "Whales are mammals."},
        {"input": "Article about Java GC...",     "expected": "Java GC reclaims memory."},
    ],
    task=lambda input: my_agent(input),                # your agent / prompt
    scores=[
        Factuality(model="paragon-fast"),
        AnswerRelevancy(),
    ],
    max_concurrency=4,
)

print(result.summary)                                   # p50/p95/mean per scorer
```

If `KEYSTONE_API_KEY` is set, the run is also recorded to your dashboard;
otherwise it stays purely local. Same shape in TypeScript and Go.

## Sandbox-as-a-tool ergonomics

`create()` / `get()` / `list()` return a bound **`SandboxHandle`** so an agent
loop can call the sandbox without threading the ID:

```python
sb = ks.sandboxes.create(spec_id="spec-123")

sb.exec("python script.py")
sb.write("/tmp/input.json", json.dumps(payload))
out = sb.read("/tmp/output.json")
diff = sb.diff()

sb.destroy()
```

Same pattern on **`ExperimentHandle`** and **`AgentSnapshotHandle`**:

```python
exp = ks.experiments.create("nightly", spec_id="s")
results = exp.run_and_wait(scores=[Factuality(), ExactMatch(expected_key="expected")])
cmp = exp.compare(other_exp)                            # handle or string ID
m   = exp.metrics()

snap = ks.agents.upload("codex", "./bundle", entrypoint=["python3", "agent.py"])
snap.delete()
```

The handles delegate field access to the underlying dataclass, so reading
`sb.id`, `exp.status`, `snap.version` keeps working unchanged. The old
service-level methods (`ks.sandboxes.run_command(id, …)`,
`ks.experiments.run(id)`) stay too — handle methods just call them.

## Auto-instrument every LLM client at once

```python
from polarity_keystone import auto_instrument

auto_instrument(sandbox_id=os.environ.get("KEYSTONE_SANDBOX_ID"))
# Patches every installed provider in-place. Subsequent OpenAI/Anthropic/
# Mistral/Google/LiteLLM/Claude Agent SDK/DSPy/LangChain calls auto-trace.
```

Wraps **OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK,
DSPy, LangChain** in one call — every prompt, token count, and tool call
shows up in your dashboard with no other code changes.

## Manual tracing when you want it

```python
from polarity_keystone import traced

# 1. Decorator
@traced
def fetch_user(uid: str) -> dict:
    return db.users.find(uid)

# 2. Named decorator
@traced(name="planning-phase")
def plan(prompt: str) -> str:
    return llm.complete(prompt)

# 3. Context manager
with traced(name="embed-doc"):
    openai.embeddings.create(...)
```

Spans automatically nest using contextvar-based parent linking — no need to
plumb a context object through your code.

## What's in the SDK

- **9 client services** — `sandboxes`, `specs`, `experiments`, `alerts`, `agents`, `datasets`, `scoring`, `export`, `prompts`
- **3 bound handles** — `SandboxHandle`, `ExperimentHandle`, `AgentSnapshotHandle` with delegated methods
- **29 built-in scorers** across 5 families:
  - **Heuristic** (6): `ExactMatch`, `Levenshtein`, `NumericDiff`, `JSONDiff`, `JSONValidity`, `SemanticListContains`
  - **LLM-judge** (9): `Factuality`, `Battle`, `ClosedQA`, `Humor`, `Moderation`, `Summarization`, `SQLJudge`, `Translation`, `Security`
  - **RAG** (8): `ContextPrecision`, `ContextRecall`, `ContextRelevancy`, `ContextEntityRecall`, `Faithfulness`, `AnswerRelevancy`, `AnswerSimilarity`, `AnswerCorrectness`
  - **Embedding** (1): `EmbeddingSimilarity`
  - **Sandbox invariants** (5): `FileExists`, `FileContains`, `CommandExits`, `SQLEquals`, `LLMJudge`
- **`@Scorer` decorator** — wrap any `(scenario) -> score` function
- **`Eval(name, data, task, scores)`** — Braintrust-style one-call eval primitive
- **`@traced` decorator** + context manager — auto-capture spans with contextvar-based propagation
- **Client wrapping** — `ks.wrap(openai_client, sandbox_id=...)` adds transparent trace reporting (sync / async / streaming)
- **`auto_instrument()`** — single call patches every installed provider (OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain)
- **Prompt management** — `ks.prompts.create(slug, template)` / `ks.prompts.get(slug)` / `Prompt.render(**vars)` with server-side audit + byte-identical local renderer
- **Bulk export** — `ks.export.traces(...)`, `.spans(...)`, `.scenarios(...)`, `.scores(...)` auto-paginate; `ks.export.experiment(id)` for full JSON or NDJSON dumps
- **OpenTelemetry bridge** — `wrap()` emits `gen_ai.*` metadata; `register_otel_flush()` piggy-backs on program shutdown

## Versioning

Semver. `0.x` = alpha/beta while the API shape settles.

## License

MIT.
