Metadata-Version: 2.4
Name: polarity-keystone
Version: 0.2.1
Summary: Python SDK client for the Keystone agent evaluation + sandboxed-execution platform
Author: Polarity
License: MIT
Project-URL: Homepage, https://github.com/Polarityinc/keystone-sdk-python
Project-URL: Repository, https://github.com/Polarityinc/keystone-sdk-python
Project-URL: Issues, https://github.com/Polarityinc/keystone-sdk-python/issues
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 4 - Beta
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Dynamic: author
Dynamic: requires-python

# Keystone SDK for Python

Python client for the [Keystone](https://keystone.polarity.cc) agent
evaluation + sandboxed-execution platform. Shares a single source of truth
with the [TypeScript](https://github.com/Polarityinc/keystone-sdk-js) and
[Go](https://github.com/Polarityinc/keystone-sdk-go) SDKs — byte-identical
cost estimates and prompt rendering across all three.

## Install

```bash
pip install polarity-keystone
```

Zero external runtime dependencies — uses only `urllib` from the standard
library.

## Quick start

```python
from polarity_keystone import (
    Keystone, traced,
    Factuality, ExactMatch, FileExists,
)

ks = Keystone(api_key="ks_live_...")

# Create an experiment and run it with three client-side scorers.
exp = ks.experiments.create("nightly-regression", spec_id="spec-123")
results = ks.experiments.run_and_wait(
    exp.id,
    scores=[
        Factuality(model="paragon-fast"),
        ExactMatch(expected_key="expected", case_sensitive=False),
        FileExists("output.txt", gate=True),
    ],
)
print(f"pass rate: {results.metrics.pass_rate:.0%}")

# Stream every trace for the experiment.
for trace in ks.export.traces(experiment_id=exp.id):
    print(trace["tool"], trace["cost"])

# Wrap an OpenAI/Anthropic client for auto-trace ingest
from openai import OpenAI
client = ks.wrap(OpenAI(), sandbox_id="sb-xxx")
client.chat.completions.create(model="gpt-4o", messages=[...])  # auto-reports
```

## What's in the SDK

- **9 client services** — `sandboxes`, `specs`, `experiments`, `alerts`, `agents`, `datasets`, `scoring`, `export`, `prompts`
- **28 built-in scorers** across 5 families:
  - **Heuristic** (6): `ExactMatch`, `Levenshtein`, `NumericDiff`, `JSONDiff`, `JSONValidity`, `SemanticListContains`
  - **LLM-judge** (9): `Factuality`, `Battle`, `ClosedQA`, `Humor`, `Moderation`, `Summarization`, `SQLJudge`, `Translation`, `Security`
  - **RAG** (8): `ContextPrecision`, `ContextRecall`, `ContextRelevancy`, `ContextEntityRecall`, `Faithfulness`, `AnswerRelevancy`, `AnswerSimilarity`, `AnswerCorrectness`
  - **Embedding** (1): `EmbeddingSimilarity`
  - **Sandbox invariants** (5): `FileExists`, `FileContains`, `CommandExits`, `SQLEquals`, `LLMJudge`
- **`@Scorer` decorator** — wrap any `(scenario) -> score` function
- **`@traced` decorator** + context manager — auto-capture spans with AsyncLocalStorage-equivalent context propagation
- **Client wrapping** — `ks.wrap(openai_client, sandbox_id=...)` adds transparent trace reporting for OpenAI + Anthropic (sync / async / streaming)
- **`auto_instrument()`** — single call patches every installed provider (OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain)
- **Prompt management** — `ks.prompts.create(slug, template)` / `ks.prompts.get(slug)` / `Prompt.render(**vars)` with server-side audit + byte-identical local renderer
- **Bulk export** — `ks.export.traces(...)`, `.spans(...)`, `.scenarios(...)`, `.scores(...)` auto-paginate; `ks.export.experiment(id)` for full JSON or NDJSON dumps
- **OpenTelemetry bridge** — `wrap()` emits `gen_ai.*` metadata; `register_otel_flush()` piggy-backs on program shutdown

## Versioning

Semver. `0.x` = alpha/beta while the API shape settles.

## License

MIT.
