Metadata-Version: 2.5
Name: raindrop-dspy
Version: 0.0.13
Summary: Raindrop observability integration for DSPy
Project-URL: Homepage, https://raindrop.ai
Project-URL: Documentation, https://docs.raindrop.ai
Author-email: Raindrop AI <sdk@raindrop.ai>
License-Expression: MIT
License-File: LICENSE
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: dspy>=2.0.0
Requires-Dist: raindrop-ai>=0.0.77
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: requests>=2.28; extra == 'dev'
Description-Content-Type: text/markdown

# raindrop-dspy

[![PyPI version](https://img.shields.io/pypi/v/raindrop-dspy.svg)](https://pypi.org/project/raindrop-dspy/)
[![Python](https://img.shields.io/pypi/pyversions/raindrop-dspy.svg)](https://pypi.org/project/raindrop-dspy/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Raindrop observability integration for [DSPy](https://dspy.ai/). Wraps DSPy modules (`Predict`, `ChainOfThought`, etc.) to automatically capture input fields, output fields, model name, and token usage.

## Installation

```bash
pip install raindrop-dspy dspy
```

## Quick Start

```python
import dspy
from raindrop_dspy import RaindropDSPy

raindrop = RaindropDSPy(api_key="rk_...", user_id="user-123")

lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)

predict = dspy.Predict("question -> answer")
wrapped = raindrop.wrap(predict)

result = wrapped(question="What is the capital of France?")
print(result.answer)

raindrop.shutdown()
```

### Projects

Route events to a specific [project](https://docs.raindrop.ai/platform/projects) by passing its slug as `project_id`:

```python
raindrop = RaindropDSPy(
    api_key="rk_...",
    project_id="support-prod",
)
```

`project_id` sets the `X-Raindrop-Project-Id` header on every event. Omit it (or pass `"default"`) to use your org's default **Production** project, which is the existing behavior. The same option is accepted by the `create_raindrop_dspy(...)` factory. Invalid slugs are ignored with a warning and no header is sent.

### Debug Mode

Enable verbose logging to troubleshoot issues:

```python
raindrop = RaindropDSPy(api_key="rk_...", debug=True)
```

### Factory Function (Legacy)

The `create_raindrop_dspy` factory function is still available for backwards compatibility:

```python
from raindrop_dspy import create_raindrop_dspy

raindrop = create_raindrop_dspy(
    api_key="rk_...",
    user_id="user-123",
    tracing_enabled=True,
    bypass_otel_for_tools=True,
)
wrapped = raindrop.wrap(predict)
```

## What Gets Captured

| Data | Description |
|------|-------------|
| **Input fields** | All keyword arguments passed to `forward()` |
| **Output fields** | Extracted from the DSPy `Prediction` result |
| **Model name** | From the active `dspy.settings.lm` configuration |
| **Token usage** | Prompt and completion tokens (when available) |
| **Errors** | Exception type and message captured in event properties |
| **Finish reason** | Completion finish reason (e.g. `stop`, `length`) from the LM history |

## API Reference

### `RaindropDSPy(api_key=None, user_id=None, convo_id=None, project_id=None, tracing_enabled=True, bypass_otel_for_tools=True, disable_auto_instrument=True, debug=False)`

Create a new Raindrop DSPy integration instance.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str \| None` | `None` | Raindrop API key. If omitted, telemetry is disabled. |
| `user_id` | `str \| None` | `None` | Associate all events with a user |
| `convo_id` | `str \| None` | `None` | Group events into a conversation |
| `project_id` | `str \| None` | `None` | Route events to a specific [project](https://docs.raindrop.ai/platform/projects) (slug); omit for the default **Production** project |
| `tracing_enabled` | `bool` | `True` | Enable OpenTelemetry tracing |
| `bypass_otel_for_tools` | `bool` | `True` | Bypass OTEL instrumentation for tool calls |
| `disable_auto_instrument` | `bool` | `True` | Library auto-instrumentation is opt-in (see below) |
| `debug` | `bool` | `False` | Enable debug logging (sets logger to `DEBUG` level) |

### Library auto-instrumentation is opt-in

As of `0.0.4`, `disable_auto_instrument` defaults to `True`: the
integration no longer lets Traceloop monkey-patch every LLM client library
it recognizes in your process (including the OpenAI client DSPy drives through LiteLLM). The
wrapper captures input/output, token usage, and model name
directly from the wrapped module's `forward()` calls and DSPy LM history, so no library patching is needed for full
dashboards.

Model spans need no library instrumentation: the wrapper records its own
(see [Model spans](#model-spans)). If you specifically want the
instrumentor's spans instead and have verified compatibility in your
environment, opt back in with `disable_auto_instrument=False`; the wrapper
then adds no span for calls that instrumentation already records.

### Methods

| Method | Description |
|--------|-------------|
| `wrap(module)` | Instrument a DSPy module (sync and async `forward` supported) |
| `flush()` | Flush buffered events to Raindrop |
| `shutdown()` | Flush and shut down the client |
| `identify(user_id, traits=None)` | Identify a user with optional traits |
| `track_signal(event_id, name, signal_type="default", ...)` | Track a signal on an AI event (e.g. thumbs up/down, feedback) |
| `last_event_id` | `event_id` of the most recent event this wrapper captured, or `None` |
| `resume_interaction(event_id=None)` | Reopen an event (default: `last_event_id`) as a core `Interaction` to update it; `None` when there is nothing to resume |

### Per-call properties and identity

A module's `forward()` has no channel for your own event data, so wrap the call in
`event_context`. Its `properties` merge into every event captured inside the
block, and `user_id` / `convo_id` replace the constructor values for those
events:

```python
from raindrop_dspy import event_context

with event_context(user_id="user-123", convo_id="conv-9", properties={"plan": "pro"}):
    wrapped(question="What is the capital of France?")
```

### Updating the last event

`last_event_id` is the `event_id` of the most recent event this wrapper
captured. Use it to attach a signal, or reopen the event with
`resume_interaction()` (defaults to `last_event_id`) and update it through the
core SDK's `Interaction`:

```python
interaction = raindrop.resume_interaction()
if interaction is not None:
    interaction.set_properties({"reviewed": True})
    interaction.finish()
```

**Key clashes.** If one of your properties has the same key as one the
wrapper derives itself, your value wins, whichever update carries it (a
debug-level log line names the key). The wrapper's own keys are `ai.*` (for
example `ai.usage.prompt_tokens`), `error`/`error.*` and `dspy.*`; avoid them
unless you mean to replace the wrapper's value.

Property-only updates need `raindrop-ai` 0.0.77 or newer; older core
versions drop an update that carries no input or output.

`last_event_id` is per wrapper instance; under concurrent calls it is whichever
event was captured last, so pass `event_id=` when you need to correlate
concurrent calls.

### Identifying Users

```python
raindrop.identify("user-123", traits={"name": "Alice", "plan": "pro"})
```

### Tracking Signals

```python
raindrop.track_signal(
    event_id="evt-abc",
    name="thumbs_up",
    signal_type="feedback",
    sentiment="POSITIVE",
    comment="Great response!",
)
```

### Flushing and Shutdown

```python
raindrop.flush()     # flush pending data
raindrop.shutdown()  # flush + release resources
```

## Async Support

DSPy modules with async `forward` methods are automatically detected and wrapped:

```python
import dspy
from raindrop_dspy import RaindropDSPy

raindrop = RaindropDSPy(api_key="rk_...")

class AsyncPredict(dspy.Module):
    async def forward(self, question: str) -> dspy.Prediction:
        ...

wrapped = raindrop.wrap(AsyncPredict())
result = await wrapped(question="Hello")
```

## Tool catalog capture (`ai.prompt.tools`)

Every model span records the native tools DSPy passed to the LM on that call as `ai.prompt.tools`: an OTel string array, one JSON document per tool with `name`, `description` and the full JSON Schema as `inputSchema` (`{"type": "function", ...}`). The capture is **exact, per LM call**: while a wrapped module's `forward` runs, the wrapper installs a DSPy callback (through `dspy.context(callbacks=...)`) whose `on_lm_start` receives the LM call's `kwargs["tools"]`, the OpenAI-shaped list `ChatAdapter(use_native_function_calling=True)` built for that request, and binds it through the core's `prompt_tools_scope` mechanism until `on_lm_end`. An LM call without native tools (the default adapter, which describes tools in the prompt text) records `[]`.

With `disable_auto_instrument=False`, model spans are `litellm.chat` spans from the OpenLLMetry LiteLLM instrumentor. OpenLLMetry 0.62 writes the same list to `gen_ai.tool.definitions`, not `llm.request.functions.*`; `raindrop-ai` 0.0.70 / 0.0.71 do not rebuild `ai.prompt.tools` from that attribute and 0.0.72 does, so the wrapper-side scope is what puts the attribute on the span on every core version, and it takes precedence over the rebuilt list when both are present. With the default `disable_auto_instrument=True`, the wrapper's own model span for each LM call (see below) carries it. `wrap()` instruments `forward` (sync, or an `async def forward`); `acall()` / `aforward` are not instrumented by this wrapper, so no override reaches LM calls made that way: with `raindrop-ai` >= 0.0.72 they carry the exact request list the core rebuilds from the instrumentor, on 0.0.70 / 0.0.71 they carry no attribute.

Override:

```python
rd = RaindropDSPy(api_key="...", user_id="u1", disable_auto_instrument=False,
                  tools=[{"type": "function", "name": "get_weather", "inputSchema": {...}}])
module = rd.wrap(dspy.Predict(Sig))               # constructor default applies to every LM call
module = rd.wrap(dspy.Predict(Sig), tools=[...])  # per-wrap value replaces the default
```

An override is bound for the whole `forward` call and the per-LM-call callback is not installed. `tools=[]` at either level records an empty catalog. An enclosing `raindrop.prompt_tools(...)` / `begin(..., tools=...)` scope is also treated as an override and is never shadowed by the inferred list.

Semantics shared with every Raindrop SDK:

- **Attribute absent**: the wrapper could not see the tool list (or there is no model span, see below).
- **`[]`**: the model had no tools.
- **Override replaces, never merges.** A `tools=` you pass wins over anything inferred; `tools=[]` records an empty catalog.
- **Content gate.** Tool definitions are prompt content. With `TRACELOOP_TRACE_CONTENT=false` nothing is recorded, overrides included.
- Malformed entries are skipped with a debug log; nothing here raises into your application.

Accepted shapes for `tools=`: canonical `{"type": "function", "name", "description", "inputSchema"}`, OpenAI `{"type": "function", "function": {...}}`, Anthropic `{"name", "description", "input_schema"}`, and `{"type": "provider-defined", "name", ...}`. The core normalises them; the JSON Schema is kept exactly as given.

Needs `raindrop-ai>=0.0.70`; on an older core the wrapper logs at debug and records nothing.

## Double-Wrap Protection

Calling `wrap()` on an already-wrapped module is a no-op — the module is returned unchanged.

## Application Git metadata

`RaindropDSPy(...)` and `create_raindrop_dspy(...)` accept the keyword-only `app_git` option. It defaults to `True`: explicit Raindrop Git environment or deployment context is applied immediately, and the base SDK may perform one bounded background local-Git lookup from the process working directory. Event capture, flush, and shutdown never wait for that lookup. Pass `False` to disable enrichment, or pass an `AppGitOptions` mapping with `commit_sha`, `commit_dirty`, `branch`, `source_directory`, `detect_branch`, and/or `auto_detect`. Automatic branch discovery remains opt-in through `detect_branch=True` (or `RAINDROP_GIT_DETECT_BRANCH=true`).

For an ordinary in-process application, the process working directory is treated as the application-under-test checkout. A remote, coding, workflow, or observer process must not rely on its own checkout: pass `app_git=False`, provide explicit revision values, or set `source_directory` to the actual application checkout. Canonical per-operation properties remain authoritative. When supplying `client=`, configure `app_git` while constructing that `Raindrop` client; the supplied client is authoritative and the wrapper's `app_git` argument does not reconfigure it.

Release order is deliberate: first publish the base SDK feature, then publish the wrapper feature release with its minimum dependency coordinated to that base release. The existing `raindrop-ai` lower bound remains compatible, but application Git metadata is unavailable on an older core and must not be claimed complete until the base is upgraded. Until coordination assigns a released version, the wrapper checks for an explicit base `app_git` parameter and omits the option when unsupported. Explicit non-default configuration is debug-logged and omitted. Unsupported `app_git` is determined by signature inspection before construction, not by retrying initialization after a `TypeError`; Git configuration adds no initialization attempts and does not change any existing framework-specific initialization fallback.

## User id

Every event needs a real user id: set `user_id=` on the wrapper or per call with `event_context(user_id=...)`. Without one (missing, blank, or a placeholder such as `"unknown"` or `"anonymous"`), the event is **not sent** and the wrapper logs one warning; it never raises. Updating an event you resumed with `resume_interaction(event_id=...)` needs no user id, since ingest keeps the one it stored.

## Model spans

With tracing on (the default), the wrapper records each LM call a wrapped module makes (observed through DSPy's `on_lm_start` / `on_lm_end` callbacks) as its own span on the Raindrop client's tracing, with that call's model and usage. A module call that makes no LM call records no model span; an LM call DSPy serves from its cache records no usage, and so do concurrent calls on one LM with the same prompt and the same answer, which cannot be told apart (never another call's usage). A wrapped module called inside another owns its own LM calls. Each span carries:

- the model: `gen_ai.provider.name` (canonical name, such as `openai` or `amazon-bedrock`), `gen_ai.request.model` and `gen_ai.response.model`;
- the token usage the provider reports: `gen_ai.usage.input_tokens`, `output_tokens`, `cache_read_input_tokens`, `cache_write_input_tokens` and `reasoning_tokens`;
- the event it belongs to: `ai.telemetry.metadata.raindrop.eventId` and the `traceloop.association.properties.*` keys.

The module call's total token usage also stays on the event as `ai.usage.*` properties. When an OpenTelemetry instrumentation that sees the call already records it (the provider SDK's instrumentation, for example with `auto_instrument=True` on the client, when the call goes through LiteLLM; the LiteLLM or DSPy instrumentation; only the DSPy instrumentation when DSPy sends the call through its own HTTP client, its default for known providers), the wrapper adds no span of its own, so a call is never counted twice. An instrumentation for another provider or library does not suppress the wrapper's span.

## Tool calls

Each tool call in a run becomes a tool span on the run's event, with its
arguments, result, duration and any error. The wrapper collects `dspy.Tool` calls (for example `dspy.ReAct` tools) through DSPy's tool callbacks; ReAct's internal `finish` step is skipped. Spans need tracing
enabled (the default); input and output follow the core SDK's content setting
(`TRACELOOP_TRACE_CONTENT`).

## Standard options and flush results

When the wrapper builds its own client, it accepts the core SDK's standard
options by the same names and passes them on:

```python
raindrop = RaindropDSPy(
    api_key="rk_...",
    user_id="user-123",
    redact_pii=True,        # local PII redaction of event input/output
    flush_timeout=5,        # seconds; deadline for flush() and shutdown()
    debug=True,             # also raises the core SDK's log verbosity
    max_queue_size=5_000,   # any other option `Raindrop(...)` declares
)

result = raindrop.flush()   # the core's FlushResult; safe to ignore
```

- An option the installed `raindrop-ai` does not support is ignored with a
  debug log, never an error. `flush_timeout` and the `FlushResult` return
  value need `raindrop-ai` 0.0.77 or later; on older releases `flush()` and
  `shutdown()` return `None`.
- With `client=` (a `Raindrop` you built), configure these options on that
  client; the wrapper ignores them (debug log).
- `flush()` and `shutdown()` never raise. The result is truthy when
  everything was delivered, and reports `pending`, `dropped` and `timed_out`.

## Development

```bash
cd packages/dspy-python
pip install -e ".[dev]"
python -m pytest tests/ -v   # unit tests (no external services)
```

End-to-end behavior is verified by the cross-SDK **conformance harness**. This
package ships a thin conformance driver at
[`conformance/driver.py`](conformance/driver.py) that maps the shared scenario
corpus onto the wrapper's public API; known gaps are tracked as ticket-linked
entries in [`conformance/failures.txt`](conformance/failures.txt). The **fault
lane** runs on every PR touching `packages/*-python/**`
(`.github/workflows/conformance-wrappers-python.yml`) against a local capture
server; the **prod lane** verifies delivery by reading back through the public
[Query API](https://docs.raindrop.ai/api-reference/overview). The harness is
pinned by commit SHA (`HARNESS_REF`). See the harness docs:
[HOW-IT-WORKS](https://github.com/invisible-tools/raindrop-sdk-harness/blob/main/docs/HOW-IT-WORKS.md) ·
[AGENTS](https://github.com/invisible-tools/raindrop-sdk-harness/blob/main/AGENTS.md) ·
[README](https://github.com/invisible-tools/raindrop-sdk-harness/blob/main/README.md).

## License

MIT

[Full documentation](https://docs.raindrop.ai/integrations/dspy)
