Metadata-Version: 2.5
Name: nirnay
Version: 0.2.0
Summary: Fast, cheap decisions with honest confidence on the Sarvam API: classify, route and escalate when unsure
Project-URL: Homepage, https://github.com/Dobbyisus/nirnay
Project-URL: Source, https://github.com/Dobbyisus/nirnay
Project-URL: Issues, https://github.com/Dobbyisus/nirnay/issues
Author: Dobbyisus
License-Expression: MIT
License-File: LICENSE
Keywords: calibration,classification,code-mixed,confidence,hindi,hinglish,india,indic,intent,llm,routing,sarvam
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Bengali
Classifier: Natural Language :: English
Classifier: Natural Language :: Hindi
Classifier: Natural Language :: Tamil
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: python-dotenv>=1.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Description-Content-Type: text/markdown

# nirnay

**Fast, cheap decisions with honest confidence on the Sarvam API.**

nirnay (Hindi: *decision*) is a small Python library for the questions inside your product that
have a fixed set of answers: which team should handle this ticket, what does this user want, is
this review positive. It returns the answer, a confidence score you can act on, and the exact ₹
cost, and it hands genuinely hard cases to Sarvam's thinking mode automatically.

Benchmarked on 1,616 test decisions across 5 tasks in English, Hindi, Hinglish, Bengali and Tamil,
against `sarvam-105b` with Sarvam's default settings (nirnay v0.2, logprobs confidence):

| | Sarvam default settings | nirnay |
|---|---|---|
| Accuracy (with label descriptions) | 89.7% | **91.5%** |
| Cost per 1,000 decisions | ₹43.51 | **₹16.14** (37%, 2.7× cheaper) |
| Typical response time | 2.2–3.5 s | **0.22–0.40 s** (9–14× faster) |
| Decisions with no usable answer | 1.2–5.4% | **0%** |
| Confidence score | — | **yes** |

> **Fine print:** most of the accuracy gap comes from the default sometimes giving no answer
> (it thinks until it hits the token limit). On the decisions it did answer, the default was
> slightly more accurate on 4 of 5 tasks. Read it as **similar accuracy, much lower cost and
> latency, and no failed decisions.** Full results and caveats are [below](#benchmark).

![Accuracy by task](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/01_accuracy.png)

---

## Install

```bash
pip install nirnay
```

Requires Python 3.10+ and a Sarvam API key ([dashboard.sarvam.ai](https://dashboard.sarvam.ai)),
read from the `SARVAM_API_KEY` environment variable.

## Quickstart

```python
from nirnay import Decider

decider = Decider("sarvam-105b")

result = decider.decide(
    question="Which team should handle this customer support message?",
    choices={
        "billing": None,
        "refund": "refund policy, getting a refund, or refund status",
        "technical": None,
        "other": None,
    },
    context="Mera paisa do baar kat gaya, refund kab milega?",
)

result.choice          # "refund"
result.confidence      # 0.999 (the model's probability; calibrated if you pass a calibrator)
result.escalated       # False (True when the model was unsure and thinking mode decided)
result.probabilities   # {"refund": 0.999, "billing": 0.0003, ...}
result.usage.cost_inr  # ≈ 0.003
result.latency_ms      # ≈ 260 typical
```

## How it works

1. **One call, the model's own probabilities.** The model answers with a single letter on
   Sarvam's V2 endpoint, which also returns how likely the model found each option (token
   log-probabilities). nirnay keeps the probabilities of the allowed letters, rescales them to
   add up to 1, and picks the most likely option. Its probability is the confidence. This is
   the default for models that support it (`sarvam-105b`, `gemma4`).
   For models without it (e.g. `sarvam-105b-conversations`), or with `confidence="votes"`,
   nirnay instead asks for 10 answers in one call (Sarvam's `n`; the prompt is billed once) and
   uses the share that agree. Votes are cheaper but a weaker confidence signal (see
   [v0.1 vs v0.2](#votes-v01-vs-logprobs-v02)).
2. **Escalate when unsure.** If the confidence is below 0.8, nirnay asks again with Sarvam's
   default settings (thinking on) and uses that answer. In the benchmark this happened for
   2–27% of decisions, depending on the task. If thinking runs out of tokens without
   answering, nirnay keeps the fast answer instead of failing.
3. **Honest confidence.** Optionally fit a calibrator on a few hundred labelled examples so that
   "90% confident" means right about 90% of the time. Set `threshold=` to get `choice=None`
   (route to a human) when confidence is too low.
4. **Label descriptions.** Pass `{label: description}` instead of a list to say what a category
   means. This was the biggest single accuracy lever in our tests (+17 points on support routing,
   measured with v0.1).
5. **Robust parsing.** Replies such as `"\nB"`, `"b."`, `"refund"`, or a letter spelt in Hindi,
   Bengali or Tamil script (`"बी"`, `"বি"`, `"சி"`) are all understood.

### Details

- The fast path runs with thinking off and a 4-token answer cap; thinking is used only for
  escalated decisions. Sarvam's two endpoints switch thinking off differently (`"none"` on V2,
  `null` on V1); nirnay sends the right one.
- Choices are shown to the model as letters (A, B, C…) and mapped back, so answers are one token.
- The fixed part of the prompt (question, options, descriptions) comes first so Sarvam's prompt
  cache can bill it at the cached rate (reported on V1; V2 reported no cache hits in our tests).
- Rate limits (429) and transient errors are retried with backoff; `min_interval_s` paces calls.

![Label descriptions (measured with v0.1)](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/08_label_descriptions.png)

## Calibration

```python
from nirnay import Decider, IsotonicCalibrator

# raw confidences (result.raw_confidence) and whether the answer was right, from a labelled
# dev set run with the same confidence method
calibrator = IsotonicCalibrator().fit(dev_raw_confidences, dev_was_correct)
calibrator.save("calibrator.json")

decider = Decider(
    "sarvam-105b",
    calibrator=IsotonicCalibrator.load("calibrator.json"),
    threshold=0.7,  # below this, choice is None: send it to a human
)
```

Calibrate when you have a few hundred labelled examples and the raw confidence looks
overconfident. With small or noisy dev sets the raw confidence is often already reasonable:
v0.2's raw logprobs confidence had a calibration error of 0.032 across the benchmark's test
sets without any calibrator (see the benchmark).

## API

**`Decider(model="sarvam-105b", *, confidence="auto", samples=None, temperature=None,
escalate_below=AUTO, escalation_max_tokens=None, threshold=None, calibrator=None,
fallback=None, reasoning_effort=None, max_tokens=4, client=None, endpoint=None)`**

- `confidence`: `"logprobs"` (one V2 call, the model's probabilities), `"votes"` (several
  answers in one V1 call), or `"auto"`: logprobs for `sarvam-105b` and `gemma4`, votes
  otherwise or when you pass `samples` or a V1 `endpoint`.
- `samples`: votes per decision in votes mode (1–128, default 10). `1` turns voting off (no
  confidence).
- `escalate_below`: escalate when the winner's raw confidence is below this. Defaults to
  0.8 on models with thinking (both methods); `None` turns escalation off.
- `threshold`: non-escalated decisions with confidence below this return `choice=None`.
- `calibrator`: any `float -> float` callable, e.g. a fitted `IsotonicCalibrator`.
- `fallback`: returned as `choice` if nothing usable came back at all.

**`decider.decide(question, choices, context) -> Decision`**: `choices` is a list of labels, or a
mapping `{label: description or None}`.

**`Decision`** fields: `choice`, `best_guess`, `confidence`, `raw_confidence`,
`confidence_method`, `probabilities`, `votes` (votes mode), `escalated`, `escalation_answer`, `abstained`, `valid`, `invalid_samples`,
`retried`, `usage` (`input_tokens`, `cached_input_tokens`, `output_tokens`, `cost_inr`, `calls`),
`latency_ms`, `model`.

**Calibration helpers:** `IsotonicCalibrator` (`fit`, `save`, `load`), `expected_calibration_error`,
and in `nirnay.calibration`: `reliability_bins`, `risk_coverage`.

**`SarvamClient(api_key=None, *, timeout=30, max_retries=3, min_interval_s=0)`** if you need to
share a client or pace requests.

## Benchmark

Two systems, same model (`sarvam-105b`), same prompt, same categories and descriptions, same
answer parser:

- **Sarvam default settings:** only the model and messages are sent (thinking on, one answer).
- **nirnay:** the library as shipped (v0.2: logprobs confidence, escalation below 0.8).

| Task | Data | Languages | Categories | Test n |
|---|---|---|---|---|
| Support routing | 352 realistic support messages (AI-drafted, see limitations) | Hindi, Hinglish | 11 | 242 |
| Voice-assistant domain | Google Hinglish-TOP (same request in two forms) | English, Hinglish | 8 | 240 |
| Voice-assistant scenario | Amazon MASSIVE (same request in four languages) | English, Hindi, Bengali, Tamil | 18 | 504 |
| Review sentiment | AI4Bharat IndicSentiment (parallel reviews) | English, Hindi, Bengali, Tamil | 2 | 480 |
| Tweet sentiment | SemEval-2020 SentiMix (real code-mixed tweets) | Hinglish | 3 | 150 |

### Results with label descriptions

| Task | Default | nirnay | Diff (95% CI) | Default failed | ₹/1k default → nirnay | p50 | Escalated |
|---|---|---|---|---|---|---|---|
| Support routing | 90.5% | 90.5% | 0.0 (−3.3, +3.3) | 3.3% | 42.90 → 18.03 | 2.19 → 0.22 s | 17.8% |
| Voice domain | 95.8% | 95.8% | 0.0 (−1.2, +1.2) | 1.2% | 35.20 → 8.19 | 2.50 → 0.23 s | 5.8% |
| Voice scenario | 86.1% | 89.9% | **+3.8** (+1.6, +6.2) | 5.4% | 47.84 → 29.33 | 3.21 → 0.23 s | 22.6% |
| Review sentiment | 97.5% | 99.4% | **+1.9** (+0.8, +3.1) | 1.5% | 39.45 → 4.58 | 2.86 → 0.26 s | 2.3% |
| Tweet sentiment | 65.3% | 66.7% | +1.3 (−4.7, +6.7) | 5.3% | 56.22 → 18.45 | 3.55 → 0.40 s | 26.7% |
| **All (weighted)** | 89.7% | 91.5% | **+1.9** (+0.8, +2.9) | 3.3% | 43.51 → 16.14 | 2.84 → 0.25 s | 13.7% |

Bold = statistically significant (paired; exact McNemar p < 0.05, 95% bootstrap CI). Typical =
median (p50). Slow-case (p95) times improve less, 1.4–5× (8.5 s → 5.7 s overall), because
escalated decisions wait for thinking mode.

![Cost per 1,000 decisions](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/02_cost_per_1000.png)
![Typical response time](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/03_latency_typical.png)
![Failed answers](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/05_failed_answers.png)
![Default accuracy when it answered](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/06_default_answered_only.png)

### Caveats

- **Failed answers.** The default's unanswered decisions count as wrong. On the decisions it did
  answer it scored 93.6 / 97.0 / 91.0 / 98.9 / 69.0% (above nirnay on four of five tasks).
- **Support routing data is synthetic:** drafted by an AI and judged realistic by a native
  speaker, with labels from the drafter. The other four tasks use public human-labelled data.
- **One run per configuration;** Sarvam sampling isn't deterministic (the logprobs themselves
  were identical in our few repeat checks). The 0.8 escalation threshold was chosen on the dev splits;
  the test sets are out-of-sample for it.
- **The parser fixes favoured the default:** teaching the parser native-script letters raised the
  default's scores by up to 11 points, and all of its replies were re-scored that way.
- **Only the with-descriptions condition was run for v0.2.** Without label descriptions, v0.1
  was within noise of the default and saved less; expect the same for v0.2.
- Prices are Sarvam's published rates as of 30 Sep 2026 (₹29.28 input / ₹10.98 cached /
  ₹73.20 output per million tokens). Sarvam's V2 endpoint reported no prompt-cache hits, so
  v0.2's input is billed at the full rate.

### Votes (v0.1) vs logprobs (v0.2)

v0.1 measured confidence by asking for 10 answers and counting; v0.2 reads the model's
probabilities from one answer. Same 1,616 test decisions, with descriptions:

| Test, n=1,616 | Votes (v0.1) | Logprobs (v0.2) |
|---|---|---|
| Final accuracy | 91.2% | 91.5% |
| Separates right from wrong (AUROC of the fast-path confidence, higher is better) | 0.794 | **0.887** |
| Calibration error of the fast-path confidence (ECE, lower is better) | 0.052 | **0.032** |
| Confident (≥ 0.9) but wrong on the fast path | 76 | **45** |
| Escalated to thinking mode | 9.9% | 13.7% |
| Cost per 1,000 decisions | **₹11.97** | ₹16.14 |
| Typical / slow-case (p50 / p95) time | **0.18** / 5.2 s | 0.25 / 5.7 s |

- **v0.2's confidence is clearly better:** higher AUROC on all five tasks, and about 40% fewer
  "very sure but wrong" answers. That is why it is the default: the confidence is what you act
  on (escalate, abstain, send to a human).
- **v0.1 is cheaper and a little faster.** v0.2 escalates more often, and its V2 calls get no
  prompt caching, so it costs ~35% more. Final accuracy is the same (difference not
  significant, p = 0.54). If cost matters more than confidence quality, use
  `Decider(confidence="votes")`.
- On dev data, the 0.8 threshold sent decisions to thinking mode that the fast path got right
  only 47% of the time; the rest were right 92% of the time.

![Confidence quality: v0.2 vs v0.1](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/09_confidence_quality.png)
![Confidence honesty](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/10_confidence_honesty.png)
![Escalation rate: v0.2 vs v0.1](https://raw.githubusercontent.com/Dobbyisus/nirnay/main/assets/benchmarks/v2/11_escalation_rate.png)

The v0.1 charts are kept in [`assets/benchmarks/`](https://github.com/Dobbyisus/nirnay/tree/main/assets/benchmarks).

### Reproducing

```bash
pip install -e ".[dev]"
python experiments/prepare_bench.py                       # builds data/bench/ from the public datasets in data/raw/
python experiments/bench_run.py --system default --splits test   # Sarvam default; needs SARVAM_API_KEY
python experiments/bench_v2.py                            # nirnay v0.2 on the test sets (~₹20, ~1 h)
python experiments/analyze_bench_v2.py                    # v0.2 vs default vs v0.1
python experiments/make_v2_charts.py                      # v0.2 charts
python experiments/logprobs_dev.py                        # logprobs on the dev splits (threshold choice)
python experiments/analyze_logprobs_dev.py
python experiments/bench_run.py --system nirnay           # v0.1 (votes)
python experiments/bench_analyze.py                       # v0.1 tables
python experiments/audit_bench.py                         # v0.1 independent recomputation + significance
```

Dataset downloads (place under `data/raw/`): Hinglish-TOP (Apache-2.0), MASSIVE 1.1 (CC BY 4.0),
IndicSentiment (licence not stated on its dataset card), SemEval-2020 SentiMix (CC BY 4.0). The
support-routing set is private.

## Roadmap

- Lower v0.2's cost: a cheaper escalation threshold chosen on fresh dev data, and prompt caching
  on V2 if Sarvam enables it.
- "Auto mode": detect decision-shaped calls inside an agent and route them through nirnay.
- Async client and a response cache.

## Related work

The escalate-when-unsure idea builds on LLM-cascade research, notably *Mixture-of-Thought
cascades* (ICLR 2024). nirnay applies it to Sarvam using Sarvam's own thinking switch as the two
tiers, so everything stays on one India-hosted provider.

## Development

Copy `.env.example` to `.env` and add your Sarvam API key.

```bash
pip install -e ".[dev]"
pytest
ruff check .
```

## License

Released under the MIT License: see the [LICENSE][license] file.

[license]: https://github.com/Dobbyisus/nirnay/blob/main/LICENSE
