Metadata-Version: 2.4
Name: needlepath-litellm
Version: 0.1.0
Summary: Needlepath context selection as a LiteLLM proxy CustomGuardrail.
Project-URL: Homepage, https://nextmoca.com
Author: Next Moca Global, Inc.
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: context,guardrail,litellm,llm,needlepath,proxy
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: litellm[proxy]<2.0.0,>=1.70.0
Requires-Dist: needlepath<1.0.0,>=0.1.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# `needlepath-litellm`

Needlepath context selection as a **LiteLLM proxy** `CustomGuardrail`. Every
client behind the proxy gets context selection with no client-side change at
all — and when the selection service is slow or down, every one of them still
gets their completion.

```bash
pip install needlepath-litellm     # pulls litellm[proxy]
```

## Configure

```yaml
# config.yaml
guardrails:
  - guardrail_name: "needlepath"
    litellm_params:
      guardrail: needlepath_litellm.NeedlepathGuardrail
      mode: "pre_call"
      default_on: true
      operating_point: "np-2026-07-r2"     # required; immutable label
      history_max_tokens: 8000
      preserve_recent: 2
```

Set `NEEDLEPATH_API_KEY` in the proxy's environment (or pass `api_key:` above,
which puts a secret in your config file — prefer the environment).

Everything under `litellm_params` other than `guardrail`, `mode` and
`default_on` is forwarded to the constructor, so every knob below is settable
from config with no code.

| Parameter | Default | What it does |
|---|---|---|
| `operating_point` | — | **Required.** An immutable label. Also `NEEDLEPATH_OPERATING_POINT`. |
| `history_max_tokens` | `8000` | Both the trigger and the budget. Under it, no call is made. |
| `preserve_recent` | `2` | Trailing messages never rewritten. |
| `shadow` | `false` | Measure and report; never apply. |
| `enabled` | `true` | Kill switch. Needs no credentials when `false`. |
| `include_assistant` / `include_user` | `false` | Widen beyond tool replies. |
| `placeholder` | see source | Replaces an unselected tool reply. |
| `base_url`, `api_key`, `timeout` | env / defaults | Core-client settings. |

With `default_on: false`, clients opt in per request:

```json
{"model": "gpt-4o", "messages": [...], "guardrails": ["needlepath"]}
```

## What it does

Before the request is routed, tool replies in the message array are selected
against the current user turn. A reply the current step needs is replaced by the
part of it that answers; a reply it does not need collapses to a short
placeholder. The `system` message, assistant messages carrying `tool_calls`, and
the last `preserve_recent` messages are never touched.

**A message is never removed, only rewritten.** Dropping an assistant message
that carries `tool_calls` while keeping its `tool` reply — or the reverse — is
rejected by every provider. Rewriting content in place makes that impossible by
construction, at the cost of the tokens in the message envelopes.

## Hook coverage — read this before you deploy

Mutation on LiteLLM is proxy-only in practice, and **exactly one hook can
replace a request**: `async_pre_call_hook`. Its returned `dict` becomes the
request data. Everything else on the base class is observability, rejection, or
post-processing.

### Which routes this guardrail acts on

`mode: "pre_call"` fires on a long list of routes, but only one of them carries
an OpenAI-format `messages` array we can safely rewrite. The rest are **left
strictly alone**, with `reason: "unsupported"` recorded and no call made:

| Route | `call_type` | Body shape | This guardrail |
|---|---|---|---|
| `/chat/completions` | `acompletion` | `data["messages"]`, OpenAI format | ✅ **selects** |
| `/v1/messages` (native Anthropic) | `anthropic_messages` | Anthropic content blocks **plus a separate top-level `data["system"]`** | ⏭️ skipped |
| `/responses` | `aresponses` | `data["input"]`, Responses-API shape, **no reverse transform provided by litellm** | ⏭️ skipped |
| `/completions` | `atext_completion` | `data["prompt"]`, a raw string | ⏭️ skipped |
| `/embeddings` | `aembedding` | `data["input"]` | ⏭️ skipped |
| `/anthropic/*` pass-through | `pass_through_endpoint` | raw provider body | ⏭️ skipped |
| MCP tool call | `call_mcp_tool` | a different event type (`pre_mcp_call`) — a `mode: "pre_call"` guardrail **never fires** | ⏭️ n/a |
| rerank, audio, images, realtime, moderations | various | not message-shaped | ⏭️ skipped |
| `/v1/models`, `/health*`, `/key/*`, `/user/*`, `/team/*`, admin | — | **no `pre_call` hook at all** | ⏭️ n/a |
| `/v1/files`, `/v1/batches`, `/v1/fine_tuning/*` | — | **post-call only**, no `pre_call` | ⏭️ n/a |

Skipping is not an oversight. Rewriting an Anthropic content-block body or a
Responses-API `input` as if it were a chat message array would corrupt the
request; the safe move is to do nothing and say so.

**If your traffic is mostly `/v1/messages` or `/responses`, this guardrail saves
you nothing today.** Route-specific support is additive work that does not
exist yet — which is a better thing to read here than to discover from a
savings number that turns out to be zero.

### Other caveats worth knowing

- **`apply_guardrail` is never overridden here, and you must not add it.** If a
  subclass defines it, LiteLLM routes the call through its `unified_guardrail`
  singleton and `async_pre_call_hook` never runs. Silent, and total.
- **`data` is mutated in place and returned.** Returning a fresh dict works on
  the proxy but loses non-`messages` keys on LiteLLM's SDK path, where only
  `result["messages"]` is copied back.
- **The SDK path is not purely observability.** A `CustomGuardrail` in
  `litellm.callbacks` *will* fire in an SDK process on `completion`/`acompletion`
  when the caller passes `guardrails=[...]`, because `CustomGuardrail` bridges
  the SDK-side `async_pre_call_deployment_hook` to `async_pre_call_hook`. This
  guardrail is safe there for the same reason: in-place mutation.
- **Metadata goes in `metadata` / `litellm_metadata`, never a new top-level
  key.** Unknown top-level keys are forwarded toward providers on some routes
  and rejected. A debug marker must not become an outage.
- **This hook never raises.** An exception here is raised to the client, which
  would turn a selection-service hiccup into a failed LLM request for everyone
  behind the proxy. The whole body is wrapped; worst case, the request goes
  through unchanged.
- **Streaming egress is untouched.** This guardrail is ingress-only. If you ever
  add egress control, use `async_post_call_streaming_iterator_hook`, not
  `async_post_call_streaming_hook` — the latter computes its payload only for
  `ModelResponse`-typed chunks and is a silent no-op on Anthropic and
  pass-through SSE, which emit raw `str`/`bytes`.

## Observability

Every request the guardrail touches carries a metadata-only blob:

```json
{"metadata": {"needlepath": {
  "applied": true, "reason": "ok", "request_id": "np-…",
  "operating_point": "np-2026-07-r2", "tokens_saved": 6800,
  "rewrite_tokens_before": 7100, "rewrite_tokens_after": 900,
  "gate_reason": "engage:needle"}}}
```

`rewrite_tokens_*` is what this adapter measured; `tokens_*` is the service's own
accounting. They are reported separately and never blended.

`guardrail.stats.as_dict()` aggregates the same counters per process.

Nothing derived from message text is in the blob. One consequence is non-obvious:
the service's `selection_error` is built as `f"{type(exc).__name__}: {exc}"` and
can carry fragments of the request, so only its *type* is reported.

## Lifecycle

A proxy builds its guardrails once at startup, so this rarely matters. If you
construct guardrails repeatedly, `await guardrail.aclose()` releases the client's
connection pool; a client you passed in with `client=` is never closed for you.

## Shadow mode

`shadow: true` makes every call, reports every saving, and changes nothing. It is
the day-one deployment: put it in front of real traffic, read the numbers, decide
afterwards.

## Failure behaviour

| What happens | What the client gets |
|---|---|
| Selection service times out, 5xx, throttles | their original request, unchanged |
| The gate stands down, or selects nothing | their original request, unchanged |
| A route we do not support | their original request, unchanged |
| A defect in this guardrail | their original request, unchanged |

There is no configuration in which a Needlepath failure becomes an LLM failure.

## Tested against

`litellm==1.94.1` (`litellm[proxy]`), capped at `<2.0.0`. CI runs against the
newest 1.x minor; see `.github/workflows/sdk-python.yml`.
