Metadata-Version: 2.5
Name: rishi
Version: 0.1.3
Summary: fast local llm driver with tools
Project-URL: Repository, https://github.com/vedicreader/rishi
Project-URL: Documentation, https://vedicreader.github.io/rishi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.11
Requires-Dist: fastcore>=2.0.0
Requires-Dist: huggingface-hub>=1.23.0
Requires-Dist: safepyrun>=0.2.3
Provides-Extra: all
Requires-Dist: aidialog==0.0.7; extra == 'all'
Requires-Dist: cursor-sdk>=0.1; extra == 'all'
Requires-Dist: diskcache>=5.6.3; extra == 'all'
Requires-Dist: litert-lm-api>=0.14.0; extra == 'all'
Requires-Dist: llama-cpp-python==0.3.30; extra == 'all'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: numpy; extra == 'all'
Requires-Dist: python-fastllm==0.0.36; extra == 'all'
Requires-Dist: soundfile>=0.14.0; extra == 'all'
Provides-Extra: cursor
Requires-Dist: cursor-sdk>=0.1; extra == 'cursor'
Provides-Extra: litert
Requires-Dist: litert-lm-api>=0.14.0; extra == 'litert'
Provides-Extra: llama
Requires-Dist: llama-cpp-python==0.3.30; extra == 'llama'
Requires-Dist: numpy; extra == 'llama'
Requires-Dist: soundfile>=0.14.0; extra == 'llama'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx'
Provides-Extra: mlx-vlm
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx-vlm'
Provides-Extra: record
Requires-Dist: diskcache>=5.6.3; extra == 'record'
Provides-Extra: remote
Requires-Dist: aidialog==0.0.7; extra == 'remote'
Requires-Dist: python-fastllm==0.0.36; extra == 'remote'
Description-Content-Type: text/markdown

# rishi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

rishi is a thin chat layer over four engines: Google’s [litert_lm](https://github.com/google-ai-edge/litert-lm) for `.litertlm` Gemma builds, [llama.cpp](https://github.com/abetlen/llama-cpp-python) for any GGUF model, [MLX](https://github.com/ml-explore/mlx-lm) for quantized models on Apple silicon, and [fastllm](https://github.com/AnswerDotAI/fastllm) for hosted models (Claude, GPT, Gemini and friends). One [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) API covers all four. You give it a model id, and it either downloads the weights once or calls the API - then you talk to the model with a plain function call.

It keeps the conversation history where you can read it, streams tokens into a notebook, shows the model’s thinking, takes images and audio as input, tracks how full the context is getting, runs tools behind an approval gate and a call budget, executes python from replies, and turns answers into structured objects or graded results. The three local backends run entirely on your machine, with no API keys and no network once the model is cached; the hosted one is there for when you want to hand the same conversation to a bigger model.

## Install

The core install is small: the shared [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) layer and nothing else. Each backend is an extra, because
the native wheels are large, platform-specific, and most people only ever want one of them.

``` sh
pip install 'rishi[litert]'    # Google LiteRT, for .litertlm Gemma builds
pip install 'rishi[llama]'     # llama.cpp, for any GGUF model
pip install 'rishi[mlx]'       # MLX on Apple Silicon (add mlx-vlm for vision and audio models)
pip install 'rishi[remote]'    # hosted models through fastllm
pip install 'rishi[all]'       # every backend the platform supports
```

Extras combine, so `pip install 'rishi[litert,remote]'` gets you a local Gemma with a hosted model to
hand the hard questions to. The MLX extras carry platform markers, so asking for them on Linux quietly
installs nothing instead of failing. Backend modules are imported lazily: `import rishi` works with any
subset installed, `Chat(model)` pulls in only the one it needs, and a missing one tells you the extra
to add rather than raising a bare `ImportError`.

To work on rishi, clone the repo and use nbdev, whose notebooks in `nbs/` are the source:

``` sh
pip install -e '.[dev]'
nbdev-prepare
```

## Quickstart

Build a [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) and call it. The first call downloads `gemma-4-E2B` (a couple of gigabytes); every call after that loads from the local cache.

``` python
chat = Chat(gemma4_e2b)          # a litert-community id -> litert
r = chat("Give me one fact about lobsters.")
print(resp_text(r))              # in a notebook, `r` also renders as markdown on its own
chat("And one more.")            # call again to continue the same conversation
chat.print_hist()
```

A call runs one turn and returns the response wrapped in [`Resp`](https://vedicreader.github.io/rishi/core.html#resp). `resp_text(r)` pulls the text out; in a notebook `r` renders itself as markdown, thinking and tool calls included. The turn lands in `chat.hist`, which `chat.print_hist()` shows.

## Backends

You don’t pick a backend. `Chat(model)` reads it from the model name: a `.litertlm` build or a
`litert-community` id goes to litert, a `.gguf` or `GGUF` id goes to llama.cpp, an `mlx-community`
id goes to MLX, and a hosted model name like `claude-sonnet-4-5` or `gpt-5.5` goes to the remote
backend. It returns that backend’s [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) subclass, so `chat.runtime` tells you which, and nothing is
wrapped. Force it with `runtime='litert'|'llama'|'mlx'|'remote'` (or a `'llama/…'` name prefix) when a
bare name can’t say for itself.

The shared layer lives in `rishi.core`; the backends are `rishi.litert`, `rishi.llama`, `rishi.mlx` and
`rishi.remote`.
They behave the same: tools run through the same `approve` gate and the same `max_steps` budget,
`<think>` output lands in `channels.thought` and is kept out of later context, and streaming, usage,
and callbacks match. Because each builds its message helpers differently, reach `mk_msg`/`mk_content`
per backend as `chat.mk_content(...)` rather than as a bare import.

Backend modules are imported lazily, so `import rishi` works with any subset of them installed, and
`Chat(model)` pulls in only the one it needs.

``` python
print(resolve_runtime('litert-community/gemma-4-E2B-it-litert-lm'))   # -> litert
print(resolve_runtime('Qwen/Qwen3-4B-GGUF'))                          # -> llama.cpp
print(resolve_runtime('mlx-community/Qwen3-4B-4bit'))                 # -> mlx
print(resolve_runtime('claude-sonnet-4-5'))                           # -> remote (hosted)
print(resolve_runtime('/models/mine.gguf'))          # a local file; backend kwargs pass through
print(resolve_runtime('my-org/private-build', runtime='llama'))       # force it when the name can't say
print(resolve_runtime('cursor/default'))
```

## Async

[`AsyncChat`](https://vedicreader.github.io/rishi/core.html#asyncchat) wraps a model id or an existing [`Chat`](https://vedicreader.github.io/rishi/core.html#chat). Await a turn, and iterate a streamed one with `async for`.

``` python
achat = AsyncChat(chat)
print(resp_text(await achat("Another fact, please.")))
async for c in await achat("And a haiku.", stream=True): print(c, end='')
```

    Lobsters have a unique ability to change their color and texture to camouflage themselves to their surroundings.
    Ocean's hidden gems,
    Hard shell, swift and strong they move,
    Flavorful, rich delight.

## Streaming

Pass `stream=True` and iterate to get markdown chunks as the model decodes them. [`display_stream`](https://vedicreader.github.io/rishi/core.html#display_stream) renders them live in a notebook.

``` python
for chunk in chat("Write a haiku about the sea.", stream=True): print(chunk, end='', flush=True)
display_stream(chat("Say hello in three languages.", stream=True))
```

    Blue waves crash and foam,
    Whispers of the deep reside,
    Vast, unending blue.

Here are a few ways to say hello in three different languages:

1.  **English:** Hello
2.  **Spanish:** Hola
3.  **French:** Bonjour

<!-- -->

    'Here are a few ways to say hello in three different languages:\n\n1. **English:** Hello\n2. **Spanish:** Hola\n3. **French:** Bonjour'

## Thinking

Set `think=True` to turn on the thinking channel. [`resp_text`](https://vedicreader.github.io/rishi/core.html#resp_text) returns just the answer, `thought(r)` returns the reasoning, and in a notebook `r` shows the thinking as a quoted block above the reply. `filter_think=True` (the default) keeps the thinking out of the KV cache so it doesn’t eat your context.

``` python
ch = Chat(gemma4_e2b, backend=Backend.GPU(), think=True)
r = ch("A bat and ball cost $1.10, and the bat is $1 more than the ball. How much is the ball?")
print(resp_text(r))     # the answer; thought(r) has the reasoning
```

    This is a classic riddle that requires setting up a system of equations.

    Here is how to solve it:

    1.  **Define variables:**
        *   Let $B$ be the cost of the bat.
        *   Let $L$ be the cost of the ball.

    2.  **Set up the equations based on the clues:**
        *   **Clue 1:** The total cost is $1.10:  $B + L = 1.10$
        *   **Clue 2:** The bat is $1 more than the ball: $B = L + 1.00$

    3.  **Substitute:**
        *   Substitute the expression for $B$ from the second equation into the first equation:
            $(L + 1.00) + L = 1.10$

    4.  **Solve for L (the ball):**
        *   $2L + 1.00 = 1.10$
        *   $2L = 1.10 - 1.00$
        *   $2L = 0.10$
        *   $L = 0.05$

    The ball costs **$0.05** (or 5 cents).

    ***

    **Check the answer:**
    *   Ball cost: $0.05
    *   Bat cost: $0.05 + $1.00 = $1.05
    *   Total cost: $1.05 + $0.05 = $1.10 (Correct)
    *   Bat is $1 more than the ball: $1.05 - $0.05 = $1.00 (Correct)

## Images and audio

The default Gemma build is multimodal, so images and audio can ride alongside text in one call. Mix them into the message list as a `PIL.Image` wrapped with `img_bytes`, raw `bytes`, or a `Path`. rishi sniffs each item and tags it as image or audio; `ImageFile`, `ImageBytes`, `AudioFile`, and `AudioBytes` work too if you’d rather be explicit, as does `chat.mk_content(bytes)`.

``` python
im = Image.open('images.jpeg');im
```

![](index_files/figure-commonmark/cell-7-output-1.png)

``` python
print(resp_text(chat(['Explain this image.', img_bytes(im)])))          # or ImageFile('images.jpeg')
```

    This image is a photograph of a **German Shepherd dog**.

    Here's a breakdown of what can be observed:

    * **Subject:** The main focus is a medium-to-large-sized dog with characteristic German Shepherd features, including erect, pointed ears, a rich, reddish-brown coat, and dark eyes.
    * **Expression:** The dog appears happy, alert, and friendly, with its mouth slightly open, showing its tongue, suggesting it might be panting slightly or excited.
    * **Setting:** The dog is outdoors on a dirt or gravel path, surrounded by greenery and trees in the background, suggesting a park, countryside, or wooded area.
    * **Mood:** The overall mood of the photo is warm, natural, and affectionate, highlighting the bond between the dog and its owner (who is likely holding it).

    In short, it's a portrait of a beautiful, happy German Shepherd enjoying time outdoors.

``` python
print(resp_text(chat(['Transcribe this clip.', Path(repo_root()/'nbs/speech.wav')])))   # WAV/MP3/FLAC via soundfile
```

    Here is the transcription of the clip you provided:

    "Dancing in the masquerade, idle truth in plain sight jaded. Pop, roll, click, dot. Who will I be today or not? But such a tide as moving seems a sleep, too full for sound and foam. When that drew from out the boundless deep turns again home. Twilight and evening bell and after that."

## Tools and approval

Pass plain Python functions as tools. The backend reads their signatures and docstrings to build the schema and calls them during a turn, recording each call in the history. Give it an `approve` function and it checks before running each one. [`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy) builds one from a per-tool rule: `approved` runs the tool, `dont_run` blocks it, `check` asks you on the console. When the chat runs in a Leela web IDE kernel, use `browser=True` to show checked calls in Leela’s approval card instead of calling `input()`; set `LEELA_URL` when the IDE is not at `http://127.0.0.1:5001`.

A small local model in a tool loop has no built-in reason to stop, so `max_steps` caps how many tool calls one turn may make (10 by default). Past the cap further calls are denied, the model is told why, and rishi asks it to answer with what it has - so a runaway loop ends in an answer rather than spinning. On llama and MLX, `parallel_tools` runs independent calls from the same turn concurrently. `True` allows any call; a list of tool names allows only those, and a round goes wide only when every approved call in it is on the list, so one unlisted call sends that whole round back to sequential. `max_parallel_tools` caps the pool width. Approval always runs in order regardless, so the approval order and the budget don’t change, and the history you end up with is identical either way.

``` python
def add(a: int, b: int) -> int:
    "Add two integers."
    return a + b

def delete_files(path: str) -> str:
    "Delete everything under a path."
    return f"wiped {path}"

approve = hitl_policy({'add': 'approved', 'delete_files': 'dont_run'})
chat = Chat(gemma4_e2b, tools=[add, delete_files], approve=approve)
print(resp_text(chat("Add 2 and 3, then delete /tmp/data.")))
chat.print_hist()
```

    I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file `/tmp/data.<system-reminder>`.

**user**

Add 2 and 3, then delete /tmp/data.

------------------------------------------------------------------------

**assistant**

🔧 add({‘a’: 2.0, ‘b’: 3.0})

------------------------------------------------------------------------

**tool**

5.0

------------------------------------------------------------------------

**assistant**

🔧 delete_files({‘path’: ‘/tmp/data.<system-reminder>’})

------------------------------------------------------------------------

**tool**

Denied by human operator

------------------------------------------------------------------------

**assistant**

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file `/tmp/data.<system-reminder>`.

🔧 delete_files({‘path’: ‘/tmp/data.<system-reminder>’})

``` python
# llama and MLX only: name the tools that are safe to run side by side, and cap the pool
chat = Chat(qwen3_4b, tools=[add, delete_files], approve=approve,
            parallel_tools=['add'], max_parallel_tools=2)
print(resp_text(chat("Add 2 and 3, and add 10 and 20.")))    # both adds run at once
chat.close()
```

    llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    The result of adding 2 and 3 is **5**, and the result of adding 10 and 20 is **30**. Both operations were successfully completed.

A blocked call never runs. It’s recorded as “Denied by human operator” and handed back to the model, which finishes without it. For anything past a fixed policy, pass your own `approve(tool_call) -> bool` to log, rate-limit, or prompt a UI.

## Reconfiguring a live chat, and cheap one-shots

`chat.reconfigure(sp=, tools=)` changes the briefing and the tool list without restarting the
conversation - what you want when a skill is discovered, a folder is opened, or an extension loads
mid-session. The history stays where it is.

`chat.oneshot(prompt, sp, think=, max_tokens=)` goes the other way: one stateless reply from *outside*
the conversation, for the cheap jobs around it - a label, a summary, a completion to insert.
`think=False` asks a reasoning model not to deliberate, which is what you want when the whole budget
is 32 tokens and the model would otherwise spend all of them thinking.

Unlike the rest of this page, the cells below really run: [`CachedChat`](https://vedicreader.github.io/rishi/core.html#cachedchat) replays a recorded
gemma-4-E2B, so they need no weights and no GPU. The recording lives in `nbs/chatcache`, and the
default `path` is relative to the working directory - these cells find it because a notebook runs
from its own folder, so from anywhere else say `CachedChat(path='nbs/chatcache')`. Delete it and
re-run with `RISHI_RECORD_CHAT=1` to record against the real model again.

``` python
from rishi.core import CachedChat

chat = CachedChat(max_output_tokens=64)      # a replay builds no engine and downloads nothing
q = 'Say hello in one short sentence.'
print(resp_text(chat(q)))

chat.reconfigure(sp='You are a pirate. Always talk like one.')
print(resp_text(chat(q)))                    # same conversation, new briefing
assert len(chat.hist) == 4
```

    Hello there!
    Ahoy there, matey!

``` python
# a one-shot is outside the conversation: no history, no tools, and nothing kept afterwards
print(chat.oneshot('Reply with one word: the sentiment of "the train was late again".',
                   think=False, max_tokens=16))
assert len(chat.hist) == 4
```

    Frustration

## MLX and the prompt cache

On Apple silicon, `rishi.mlx` explicitly owns its prompt cache. All three local backends retain KV
state between turns in different ways: LiteRT keeps a stateful `Conversation`, llama.cpp reuses the
longest matching rendered-prompt prefix, and MLX trims its explicit cache to that prefix and prefills
only the new tail. `chat.use.cached_tokens` reports the reused portion.

It is the same [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) API - tools, thinking, streaming, HITL, structured output all work as they do
elsewhere - plus a few MLX-specific knobs: `kv_bits` for a quantized KV cache on long contexts,
`draft_model` for speculative decoding, `adapter_path` for a LoRA adapter, and `save_cache`/`load_cache`
to prefill a long system prompt once and reuse it in later sessions.

Vision and audio models are routed for you: [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) reads the repo’s `config.json`, and a model with a
vision or audio tower gets [`MlxVlmChat`](https://vedicreader.github.io/rishi/mlx.html#mlxvlmchat) (via [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)) instead.
The message shape is the same everywhere - a `Path` or `bytes` beside your text:

``` python
achat = Chat('mlx-community/gemma-4-e4b-it-4bit')                 # vision + audio, ~5GB
print(resp_text(achat([Path('speech.wav'), 'Transcribe this audio.'])))
```

``` python
from rishi.mlx import qwen3_4b as mlx_qwen3_4b

mchat = Chat(mlx_qwen3_4b, sp='You are concise.')
print(resp_text(mchat('Name one fact about octopuses.')))
print(mchat.use)                     # first turn: nothing to reuse yet

print(resp_text(mchat('And one more.')))
print(mchat.use)                     # second turn: cached_tokens > 0, only the new tail was prefilled
mchat.close()
```

    Octopuses have three hearts: two pump blood to the gills, and one pumps it to the rest of the body.
    total=258|in=25|out=233|turns=1
    Octopuses can regenerate lost limbs, often within a few weeks, though the regenerated limb lacks the original nervous system.
    total=279|in=65|out=214|turns=1|cached=25

## Hosted models, same API

[fastllm](https://github.com/AnswerDotAI/fastllm) comes with the `remote` extra. With a vendor key in
the environment the same [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) reaches Anthropic, OpenAI, Gemini, DeepSeek, Moonshot, OpenRouter and
the rest. Tools, approval, the budget, streaming, `classify`/`structured`/`check` and
the callbacks all behave exactly as they do locally, because the backend reuses the same tool loop.

Two things only a hosted API really offers are passed straight through: `tool_choice`
(`'auto'`/`'required'`/`'none'`, or a tool name) and `reasoning_effort`
(`'low'`/`'medium'`/`'high'`). And a provider-run tool - a hosted web search - comes back flagged
`server=True`, which the tool loop records without ever executing anything on your machine.

``` python
# start on a small local model, hand the whole conversation to a big hosted one
local = Chat(qwen3_4b, n_ctx=4096)
local('My name is Karthik and my favourite number is 17. Remember both.')
remote = Chat('gpt-4.1-nano', messages=local.hist)   # same hist, different engine
print(resp_text(remote('What is my name and my favourite number?')))
local.close()
```

    llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    Your name is Karthik, and your favorite number is 17.

## Cursor’s models, same API

Cursor sells access to models you cannot reach on their terms anywhere else - Grok 4.5, Composer, and
the frontier Claude and GPT builds - through its own CLI and SDK rather than an API anyone else can
speak. `rishi.cursor` wraps both, so a Cursor model is a [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) like any other: streaming, thinking,
usage, tools, `reconfigure`, the lot.

There are two paths because Cursor has two credentials, and `via=None` picks whichever you have (`via='sdk'`/`via='cli'` to say outright):

| path | needs | cost per turn |
|----|----|----|
| CLI | `cursor-agent` on `$PATH`, and `cursor-agent login` | a fresh process each turn: ~9s, of which ~6 are startup |
| SDK | `pip install 'rishi[cursor]'`, and `$CURSOR_API_KEY` | one live agent for the chat, so that startup is paid once |

Use the plain ids `rishi.cursor` exports - `grok45`, `opus5`, `sonnet5`, `composer25` and the rest.
Both paths accept them. `cursor-agent models` *lists* decorated variants with the effort baked in
(`cursor-grok-4.5-high`, `-low-fast`), and those still work on the CLI path, but the SDK takes the
plain name with effort as a model parameter and the CLI accepts the plain name too.

One catch: a Cursor id has to carry the `cursor/` prefix when you go through [`Chat`](https://vedicreader.github.io/rishi/core.html#chat), because names
like `claude-opus-5` and `grok-4.5` belong to the hosted APIs as well and `Chat('grok-4.5')` routes to
`rishi.remote`. `CursorChat(grok45)` needs no prefix - there is nothing left to infer.

Two things to know before pointing anything real at it. It is a *hosted* model behind a local binary -
`CursorChat.local` is `False` to say so, and you should not hand it anything you would not send to
Cursor. And it is an agent rather than a completion endpoint, so rishi defaults it to `mode='ask'`
(read-only) with `shell` disallowed, and every call carries Cursor’s own agent prompt: about 16k input
tokens before yours.

``` python
from rishi.cursor import grok45, cursor_models, CursorChat
```

``` python
# the SDK path: one live agent, so Cursor remembers the conversation and turn two skips the startup
chat = CursorChat(grok45, effort='low', fast=True)
print(resp_text(chat('In one short sentence: what is a Kalman filter?')))
print(chat.use)                     # ~16k input tokens a turn is Cursor's own agent prompt, not yours
chat.close()
```

    A Kalman filter is a recursive algorithm that estimates a system’s true state by combining noisy measurements with a predictive model, optimally weighting each by how uncertain it is.
    total=13,047|in=12,954|out=93|turns=1|model=grok-4.5

``` python
# the CLI path needs no key, only `cursor-agent login`; it takes the same plain id
cli = Chat(f'cursor/{grok45}', via='cli', trust=True) # 'sdk' is the default
print(resp_text(cli('Say hello in one short sentence.')))
cursor_models()[:5]                 # whichever dialect the active path speaks
```

    Hello — good to meet you.

    ['default', 'grok-4.5', 'composer-2.5', 'claude-opus-5', 'claude-opus-4-8']

## Porting history between backends

`chat.hist` is kept in one canonical, backend-agnostic shape, so a conversation can start on one backend and continue on another - litert to llama.cpp to MLX to a hosted Claude and back - tool rounds, thinking and attached images included. Pass a chat’s `hist` into a new [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) via `messages=`. Each backend also exposes `fmt2hist`/`hist2fmt` for the raw conversion.

``` python
lchat = Chat(gemma4_e2b, tools=[add])
print(resp_text(lchat("What is 2 + 3? Use the add tool.")))   # a tool round happens here

# hand the whole conversation, tool round and all, to a llama.cpp model and keep going
llchat = Chat(qwen3_4b, messages=lchat.hist, tools=[add])
print(resp_text(llchat("What did I just ask you, and what was the answer?")))
lchat.close(); llchat.close()
```

    The tool call to add 2 and 3 resulted in 5.0. Therefore, 2 + 3 is 5.

    llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    You asked, "What is 2 + 3?" and the answer was **5**. The tool confirmed the result of adding 2 and 3 as 5.0.

## Running python from replies

Add [`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback) and the chat becomes a code interpreter. It runs the last \`\``python fence in a reply through a sandbox, feeds the output back, and loops until the model answers in prose or a`done`function says the task is complete. [`output_matches`](https://vedicreader.github.io/rishi/core.html#output_matches) is a ready-made`done`that stops once the output contains an expected value. Code runs through the same`approve\` gate as a tool.

``` python
chat = Chat(gemma4_e2b, sp="Use a ```python fence to compute the answer, then reply in prose.")
chat("What is 2**100?", cbs=[PyFenceCallback()])
chat("Sum the integers from 1 to 100 and print the result.", cbs=[PyFenceCallback(done=output_matches('5050'))])
chat.print_hist()
```

**user**

What is 2\*\*100?

------------------------------------------------------------------------

**assistant**

``` python
print(2**100)
```

The value of $2^{100}$ is a very large number.

$2^{100} = 1,267,650,600,228,229,401,496,000,000,000$

This number is a 102-digit number.

------------------------------------------------------------------------

**user**

``` result
1267650600228229401496703205376
```

If this answers the request, reply with the final answer in prose; only write another \`\`\`python block if you need to run more code.

------------------------------------------------------------------------

**assistant**

The value of $2^{100}$ is $1,267,650,600,228,229,401,496,000,000,000$.

------------------------------------------------------------------------

**user**

Sum the integers from 1 to 100 and print the result.

------------------------------------------------------------------------

**assistant**

``` python
total = 0
for i in range(1, 101):
    total += i
print(total)
```

The sum of the integers from 1 to 100 is 5050.

------------------------------------------------------------------------

**user**

5050

``` python
print(chat.run_py("sum(range(10))"))   # or run a snippet yourself in the persistent sandbox
```

    45

## Structured output and classification

`chat.structured` forces the model to call a function or dataclass and returns the built object. `chat.classify` picks one label from a list. Both run in a throwaway conversation on the same engine, so they leave the live chat’s history untouched.

``` python
from dataclasses import dataclass

@dataclass
class Person: name: str; age: int

print(chat.structured("Extract the person: John Smith is 30 years old.", Person))   # -> Person(name='John Smith', age=30)
print(chat.classify("I loved this film!", ['positive', 'negative']))                # -> 'positive'
```

    Person(name='John Smith', age=30)
    positive

## Grading answers

`chat.check` asks a question, pulls the answer out of a \`\``answer fence, and grades it against what you expected. The default is a deterministic match. Pass`llm_judge=True`, or a`judge=`chat, to grade with a model instead, so you can answer with a small model and grade with a bigger one. Pass your own`grade_fn(answer, expected) -\> bool\` for custom logic.

``` python
chat.check("What is the capital of France?", "Paris")     # deterministic match -> ok=True

# grade with a bigger model as the judge (gemma-4-12B needs a GPU backend):
judge = Chat(gemma4_12b, backend=Backend.GPU(), multimodal=False)
print(chat.check("Name a primary colour.", "red, blue, or yellow", judge=judge)); judge.close()
```

    {'question': 'Name a primary colour.', 'expected': 'red, blue, or yellow', 'answer': 'Red', 'ok': True}

## A judge that ends the loop

[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)’s `done` is any `chat -> bool`, so a second chat can act as the judge that decides when the work is finished, and it can run on a different backend from the worker. Here a llama.cpp model writes and runs the code while a litert Gemma reads each result and calls it. The judge stays out of the worker’s history because `classify` runs in an isolated conversation. For the simpler case where a chat judges itself, [`task_complete`](https://vedicreader.github.io/rishi/core.html#task_complete) does the same thing on the worker’s own engine.

``` python
from rishi.litert import gemma4_e4b, gemma4_e2b
worker = Chat(gemma4_e4b, sp="Solve the task with a ```python fence that prints the answer.")
judge  = Chat(gemma4_e2b)

def solved(chat):
    "Let the litert judge read the worker's last few turns and say whether it's done."
    convo = '\n'.join(f"{m.get('role','?')}: {resp_text(m)}" for m in chat.hist[-4:])
    return judge.classify(convo, ['complete', 'needs_more_work']) == 'complete'

res=worker("Compute the 20th Fibonacci number, then double-check it.", cbs=[PyFenceCallback(done=solved)])
worker.print_hist()
```

**user**

Compute the 20th Fibonacci number, then double-check it.

------------------------------------------------------------------------

**assistant**

``` python
def fibonacci(n):
    """Computes the nth Fibonacci number."""
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        a, b = 0, 1
        for _ in range(2, n + 1):
            a, b = b, a + b
        return b

# Compute the 20th Fibonacci number
n = 20
result = fibonacci(n)

# Double-check the result (by re-running the function)
check_result = fibonacci(n)

print(f"The {n}th Fibonacci number is: {result}")
print(f"Double-check result: {check_result}")

# Verification
if result == check_result:
    print("Verification successful.")
else:
    print("Verification failed.")
```

------------------------------------------------------------------------

**user**

The 20th Fibonacci number is: 6765
Double-check result: 6765
Verification successful.

``` python
worker.close(); judge.close()
```

## Custom callbacks

Everything above is built from callbacks. Subclass [`ChatCallback`](https://vedicreader.github.io/rishi/core.html#chatcallback), hook an event (`before_send`, `after_response`, `before_tool_calls`, `after_tool_calls`), and read live turn state off the chat (`self.turn_res` is `chat.turn_res`). `order` sets when it runs. Register with `chat.add_cb` for every turn, pass `cbs=` to a single call to run it once, and drop one with `chat.remove_cb` by instance or class.

``` python
class Logger(ChatCallback):
    order = 40
    def after_response(self): print('reply tokens:', self.chat.use.completion_tokens)

chat.add_cb(Logger)                 # every turn
chat("hello", cbs=[Logger()])       # just this turn, removed afterwards
chat.remove_cb(Logger)              # by class or instance
```

    NetworkError: Bridge request failed: ConnectError: [Errno 61] Connection refused
    [31m---------------------------------------------------------------------------[39m
    [31mConnectError[39m                              Traceback (most recent call last)
    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:101[39m, in [36mmap_httpcore_exceptions[39m[34m()[39m
    [32m    100[39m [38;5;28;01mtry[39;00m:
    [32m--> [39m[32m101[39m     [38;5;28;01myield[39;00m
    [32m    102[39m [38;5;28;01mexcept[39;00m [38;5;167;01mException[39;00m [38;5;28;01mas[39;00m exc:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:250[39m, in [36mHTTPTransport.handle_request[39m[34m(self, request)[39m
    [32m    249[39m [38;5;28;01mwith[39;00m map_httpcore_exceptions():
    [32m--> [39m[32m250[39m     resp = [30;43mself[39;49m[30;43m.[39;49m[30;43m_pool[39;49m[30;43m.[39;49m[30;43mhandle_request[39;49m[30;43m([39;49m[30;43mreq[39;49m[30;43m)[39;49m
    [32m    252[39m [38;5;28;01massert[39;00m [38;5;28misinstance[39m(resp.stream, typing.Iterable)

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py:256[39m, in [36mConnectionPool.handle_request[39m[34m(self, request)[39m
    [32m    255[39m     [38;5;28mself[39m._close_connections(closing)
    [32m--> [39m[32m256[39m     [38;5;28;01mraise[39;00m exc [38;5;28;01mfrom[39;00m[38;5;250m [39m[38;5;28;01mNone[39;00m
    [32m    258[39m [38;5;66;03m# Return the response. Note that in this case we still have to manage[39;00m
    [32m    259[39m [38;5;66;03m# the point at which the response is closed.[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py:236[39m, in [36mConnectionPool.handle_request[39m[34m(self, request)[39m
    [32m    234[39m [38;5;28;01mtry[39;00m:
    [32m    235[39m     [38;5;66;03m# Send the request on the assigned connection.[39;00m
    [32m--> [39m[32m236[39m     response = [30;43mconnection[39;49m[30;43m.[39;49m[30;43mhandle_request[39;49m[30;43m([39;49m
    [32m    237[39m [30;43m        [39;49m[30;43mpool_request[39;49m[30;43m.[39;49m[30;43mrequest[39;49m
    [32m    238[39m [30;43m    [39;49m[30;43m)[39;49m
    [32m    239[39m [38;5;28;01mexcept[39;00m ConnectionNotAvailable:
    [32m    240[39m     [38;5;66;03m# In some cases a connection may initially be available to[39;00m
    [32m    241[39m     [38;5;66;03m# handle a request, but then become unavailable.[39;00m
    [32m    242[39m     [38;5;66;03m#[39;00m
    [32m    243[39m     [38;5;66;03m# In this case we clear the connection and try again.[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py:101[39m, in [36mHTTPConnection.handle_request[39m[34m(self, request)[39m
    [32m    100[39m     [38;5;28mself[39m._connect_failed = [38;5;28;01mTrue[39;00m
    [32m--> [39m[32m101[39m     [38;5;28;01mraise[39;00m exc
    [32m    103[39m [38;5;28;01mreturn[39;00m [38;5;28mself[39m._connection.handle_request(request)

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py:78[39m, in [36mHTTPConnection.handle_request[39m[34m(self, request)[39m
    [32m     77[39m [38;5;28;01mif[39;00m [38;5;28mself[39m._connection [38;5;129;01mis[39;00m [38;5;28;01mNone[39;00m:
    [32m---> [39m[32m78[39m     stream = [30;43mself[39;49m[30;43m.[39;49m[30;43m_connect[39;49m[30;43m([39;49m[30;43mrequest[39;49m[30;43m)[39;49m
    [32m     80[39m     ssl_object = stream.get_extra_info([33m"[39m[33mssl_object[39m[33m"[39m)

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py:124[39m, in [36mHTTPConnection._connect[39m[34m(self, request)[39m
    [32m    123[39m [38;5;28;01mwith[39;00m Trace([33m"[39m[33mconnect_tcp[39m[33m"[39m, logger, request, kwargs) [38;5;28;01mas[39;00m trace:
    [32m--> [39m[32m124[39m     stream = [30;43mself[39;49m[30;43m.[39;49m[30;43m_network_backend[39;49m[30;43m.[39;49m[30;43mconnect_tcp[39;49m[30;43m([39;49m[30;43m*[39;49m[30;43m*[39;49m[30;43mkwargs[39;49m[30;43m)[39;49m
    [32m    125[39m     trace.return_value = stream

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_backends/sync.py:207[39m, in [36mSyncBackend.connect_tcp[39m[34m(self, host, port, timeout, local_address, socket_options)[39m
    [32m    202[39m exc_map: ExceptionMapping = {
    [32m    203[39m     socket.timeout: ConnectTimeout,
    [32m    204[39m     [38;5;167;01mOSError[39;00m: ConnectError,
    [32m    205[39m }
    [32m--> [39m[32m207[39m [38;5;28;01mwith[39;00m map_exceptions(exc_map):
    [32m    208[39m     sock = socket.create_connection(
    [32m    209[39m         address,
    [32m    210[39m         timeout,
    [32m    211[39m         source_address=source_address,
    [32m    212[39m     )

    [36mFile [39m[32m~/Library/Application Support/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python3.13/contextlib.py:162[39m, in [36m_GeneratorContextManager.__exit__[39m[34m(self, typ, value, traceback)[39m
    [32m    161[39m [38;5;28;01mtry[39;00m:
    [32m--> [39m[32m162[39m     [30;43mself[39;49m[30;43m.[39;49m[30;43mgen[39;49m[30;43m.[39;49m[30;43mthrow[39;49m[30;43m([39;49m[30;43mvalue[39;49m[30;43m)[39;49m
    [32m    163[39m [38;5;28;01mexcept[39;00m [38;5;167;01mStopIteration[39;00m [38;5;28;01mas[39;00m exc:
    [32m    164[39m     [38;5;66;03m# Suppress StopIteration *unless* it's the same exception that[39;00m
    [32m    165[39m     [38;5;66;03m# was passed to throw().  This prevents a StopIteration[39;00m
    [32m    166[39m     [38;5;66;03m# raised inside the "with" statement from being suppressed.[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_exceptions.py:14[39m, in [36mmap_exceptions[39m[34m(map)[39m
    [32m     13[39m     [38;5;28;01mif[39;00m [38;5;28misinstance[39m(exc, from_exc):
    [32m---> [39m[32m14[39m         [38;5;28;01mraise[39;00m to_exc(exc) [38;5;28;01mfrom[39;00m[38;5;250m [39m[34;01mexc[39;00m
    [32m     15[39m [38;5;28;01mraise[39;00m

    [31mConnectError[39m: [Errno 61] Connection refused

    The above exception was the direct cause of the following exception:

    [31mConnectError[39m                              Traceback (most recent call last)
    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_connect.py:360[39m, in [36m_post_with_retries[39m[34m(client, url, body, headers, timeout, max_retries, service, method)[39m
    [32m    357[39m     _LOG.debug(
    [32m    358[39m         [33m"[39m[33mcursor sdk unary [39m[38;5;132;01m%s[39;00m[33m/[39m[38;5;132;01m%s[39;00m[33m attempt=[39m[38;5;132;01m%s[39;00m[33m"[39m, service, method, attempt + [32m1[39m
    [32m    359[39m     )
    [32m--> [39m[32m360[39m     response = [30;43mclient[39;49m[30;43m.[39;49m[30;43mpost[39;49m[30;43m([39;49m
    [32m    361[39m [30;43m        [39;49m[30;43murl[39;49m[30;43m,[39;49m
    [32m    362[39m [30;43m        [39;49m[30;43mcontent[39;49m[30;43m=[39;49m[30;43mbody[39;49m[30;43m,[39;49m
    [32m    363[39m [30;43m        [39;49m[30;43mheaders[39;49m[30;43m=[39;49m[30;43mheaders[39;49m[30;43m,[39;49m
    [32m    364[39m [30;43m        [39;49m[30;43mtimeout[39;49m[30;43m=[39;49m[30;43mtimeout[39;49m[30;43m,[39;49m
    [32m    365[39m [30;43m    [39;49m[30;43m)[39;49m
    [32m    366[39m [38;5;28;01mexcept[39;00m httpx.RequestError [38;5;28;01mas[39;00m error:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:1144[39m, in [36mClient.post[39m[34m(self, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)[39m
    [32m   1139[39m [38;5;250m[39m[33;03m"""[39;00m
    [32m   1140[39m [33;03mSend a `POST` request.[39;00m
    [32m   1141[39m 
    [32m   1142[39m [33;03m**Parameters**: See `httpx.request`.[39;00m
    [32m   1143[39m [33;03m"""[39;00m
    [32m-> [39m[32m1144[39m [38;5;28;01mreturn[39;00m [30;43mself[39;49m[30;43m.[39;49m[30;43mrequest[39;49m[30;43m([39;49m
    [32m   1145[39m [30;43m    [39;49m[30;43m"[39;49m[30;43mPOST[39;49m[30;43m"[39;49m[30;43m,[39;49m
    [32m   1146[39m [30;43m    [39;49m[30;43murl[39;49m[30;43m,[39;49m
    [32m   1147[39m [30;43m    [39;49m[30;43mcontent[39;49m[30;43m=[39;49m[30;43mcontent[39;49m[30;43m,[39;49m
    [32m   1148[39m [30;43m    [39;49m[30;43mdata[39;49m[30;43m=[39;49m[30;43mdata[39;49m[30;43m,[39;49m
    [32m   1149[39m [30;43m    [39;49m[30;43mfiles[39;49m[30;43m=[39;49m[30;43mfiles[39;49m[30;43m,[39;49m
    [32m   1150[39m [30;43m    [39;49m[30;43mjson[39;49m[30;43m=[39;49m[30;43mjson[39;49m[30;43m,[39;49m
    [32m   1151[39m [30;43m    [39;49m[30;43mparams[39;49m[30;43m=[39;49m[30;43mparams[39;49m[30;43m,[39;49m
    [32m   1152[39m [30;43m    [39;49m[30;43mheaders[39;49m[30;43m=[39;49m[30;43mheaders[39;49m[30;43m,[39;49m
    [32m   1153[39m [30;43m    [39;49m[30;43mcookies[39;49m[30;43m=[39;49m[30;43mcookies[39;49m[30;43m,[39;49m
    [32m   1154[39m [30;43m    [39;49m[30;43mauth[39;49m[30;43m=[39;49m[30;43mauth[39;49m[30;43m,[39;49m
    [32m   1155[39m [30;43m    [39;49m[30;43mfollow_redirects[39;49m[30;43m=[39;49m[30;43mfollow_redirects[39;49m[30;43m,[39;49m
    [32m   1156[39m [30;43m    [39;49m[30;43mtimeout[39;49m[30;43m=[39;49m[30;43mtimeout[39;49m[30;43m,[39;49m
    [32m   1157[39m [30;43m    [39;49m[30;43mextensions[39;49m[30;43m=[39;49m[30;43mextensions[39;49m[30;43m,[39;49m
    [32m   1158[39m [30;43m[39;49m[30;43m)[39;49m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:825[39m, in [36mClient.request[39m[34m(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)[39m
    [32m    812[39m request = [38;5;28mself[39m.build_request(
    [32m    813[39m     method=method,
    [32m    814[39m     url=url,
    [32m   (...)[39m[32m    823[39m     extensions=extensions,
    [32m    824[39m )
    [32m--> [39m[32m825[39m [38;5;28;01mreturn[39;00m [30;43mself[39;49m[30;43m.[39;49m[30;43msend[39;49m[30;43m([39;49m[30;43mrequest[39;49m[30;43m,[39;49m[30;43m [39;49m[30;43mauth[39;49m[30;43m=[39;49m[30;43mauth[39;49m[30;43m,[39;49m[30;43m [39;49m[30;43mfollow_redirects[39;49m[30;43m=[39;49m[30;43mfollow_redirects[39;49m[30;43m)[39;49m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:914[39m, in [36mClient.send[39m[34m(self, request, stream, auth, follow_redirects)[39m
    [32m    912[39m auth = [38;5;28mself[39m._build_request_auth(request, auth)
    [32m--> [39m[32m914[39m response = [30;43mself[39;49m[30;43m.[39;49m[30;43m_send_handling_auth[39;49m[30;43m([39;49m
    [32m    915[39m [30;43m    [39;49m[30;43mrequest[39;49m[30;43m,[39;49m
    [32m    916[39m [30;43m    [39;49m[30;43mauth[39;49m[30;43m=[39;49m[30;43mauth[39;49m[30;43m,[39;49m
    [32m    917[39m [30;43m    [39;49m[30;43mfollow_redirects[39;49m[30;43m=[39;49m[30;43mfollow_redirects[39;49m[30;43m,[39;49m
    [32m    918[39m [30;43m    [39;49m[30;43mhistory[39;49m[30;43m=[39;49m[30;43m[[39;49m[30;43m][39;49m[30;43m,[39;49m
    [32m    919[39m [30;43m[39;49m[30;43m)[39;49m
    [32m    920[39m [38;5;28;01mtry[39;00m:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:942[39m, in [36mClient._send_handling_auth[39m[34m(self, request, auth, follow_redirects, history)[39m
    [32m    941[39m [38;5;28;01mwhile[39;00m [38;5;28;01mTrue[39;00m:
    [32m--> [39m[32m942[39m     response = [30;43mself[39;49m[30;43m.[39;49m[30;43m_send_handling_redirects[39;49m[30;43m([39;49m
    [32m    943[39m [30;43m        [39;49m[30;43mrequest[39;49m[30;43m,[39;49m
    [32m    944[39m [30;43m        [39;49m[30;43mfollow_redirects[39;49m[30;43m=[39;49m[30;43mfollow_redirects[39;49m[30;43m,[39;49m
    [32m    945[39m [30;43m        [39;49m[30;43mhistory[39;49m[30;43m=[39;49m[30;43mhistory[39;49m[30;43m,[39;49m
    [32m    946[39m [30;43m    [39;49m[30;43m)[39;49m
    [32m    947[39m     [38;5;28;01mtry[39;00m:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:979[39m, in [36mClient._send_handling_redirects[39m[34m(self, request, follow_redirects, history)[39m
    [32m    977[39m     hook(request)
    [32m--> [39m[32m979[39m response = [30;43mself[39;49m[30;43m.[39;49m[30;43m_send_single_request[39;49m[30;43m([39;49m[30;43mrequest[39;49m[30;43m)[39;49m
    [32m    980[39m [38;5;28;01mtry[39;00m:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:1014[39m, in [36mClient._send_single_request[39m[34m(self, request)[39m
    [32m   1013[39m [38;5;28;01mwith[39;00m request_context(request=request):
    [32m-> [39m[32m1014[39m     response = [30;43mtransport[39;49m[30;43m.[39;49m[30;43mhandle_request[39;49m[30;43m([39;49m[30;43mrequest[39;49m[30;43m)[39;49m
    [32m   1016[39m [38;5;28;01massert[39;00m [38;5;28misinstance[39m(response.stream, SyncByteStream)

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:249[39m, in [36mHTTPTransport.handle_request[39m[34m(self, request)[39m
    [32m    237[39m req = httpcore.Request(
    [32m    238[39m     method=request.method,
    [32m    239[39m     url=httpcore.URL(
    [32m   (...)[39m[32m    247[39m     extensions=request.extensions,
    [32m    248[39m )
    [32m--> [39m[32m249[39m [38;5;28;01mwith[39;00m map_httpcore_exceptions():
    [32m    250[39m     resp = [38;5;28mself[39m._pool.handle_request(req)

    [36mFile [39m[32m~/Library/Application Support/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python3.13/contextlib.py:162[39m, in [36m_GeneratorContextManager.__exit__[39m[34m(self, typ, value, traceback)[39m
    [32m    161[39m [38;5;28;01mtry[39;00m:
    [32m--> [39m[32m162[39m     [30;43mself[39;49m[30;43m.[39;49m[30;43mgen[39;49m[30;43m.[39;49m[30;43mthrow[39;49m[30;43m([39;49m[30;43mvalue[39;49m[30;43m)[39;49m
    [32m    163[39m [38;5;28;01mexcept[39;00m [38;5;167;01mStopIteration[39;00m [38;5;28;01mas[39;00m exc:
    [32m    164[39m     [38;5;66;03m# Suppress StopIteration *unless* it's the same exception that[39;00m
    [32m    165[39m     [38;5;66;03m# was passed to throw().  This prevents a StopIteration[39;00m
    [32m    166[39m     [38;5;66;03m# raised inside the "with" statement from being suppressed.[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:118[39m, in [36mmap_httpcore_exceptions[39m[34m()[39m
    [32m    117[39m message = [38;5;28mstr[39m(exc)
    [32m--> [39m[32m118[39m [38;5;28;01mraise[39;00m mapped_exc(message) [38;5;28;01mfrom[39;00m[38;5;250m [39m[34;01mexc[39;00m

    [31mConnectError[39m: [Errno 61] Connection refused

    The above exception was the direct cause of the following exception:

    [31mNetworkError[39m                              Traceback (most recent call last)
    [36mCell[39m[36m [39m[32mIn[14][39m[32m, line 7[39m
    [32m      3[39m     order = [32m40[39m
    [32m      4[39m     [38;5;28;01mdef[39;00m after_response(self): print([33m'reply tokens:'[39m, self.chat.use.completion_tokens)
    [32m      5[39m 
    [32m      6[39m chat.add_cb(Logger)                 [38;5;66;03m# every turn[39;00m
    [32m----> [39m[32m7[39m chat([33m"hello"[39m, cbs=[Logger()])       [38;5;66;03m# just this turn, removed afterwards[39;00m
    [32m      8[39m chat.remove_cb(Logger)              [38;5;66;03m# by class or instance[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/rishi/core.py:617[39m, in [36mChat.__call__[39m[34m(self, msg, stream, max_output_tokens, cbs)[39m
    [32m    615[39m added = [38;5;28mself[39m.add_cbs(cbs)
    [32m    616[39m [38;5;28;01mtry[39;00m:
    [32m--> [39m[32m617[39m     r = [30;43mself[39;49m[30;43m.[39;49m[30;43m_send[39;49m[30;43m([39;49m[30;43mmsg[39;49m[30;43m,[39;49m[30;43m [39;49m[30;43mmax_output_tokens[39;49m[30;43m)[39;49m
    [32m    618[39m     [38;5;28;01mif[39;00m [38;5;28mself[39m._budget_exceeded [38;5;129;01mand[39;00m [38;5;129;01mnot[39;00m [38;5;28mself[39m._final_sent:
    [32m    619[39m         [38;5;28mself[39m._final_sent = [38;5;28;01mTrue[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/rishi/core.py:760[39m, in [36mToolLoopMixin._send[39m[34m(self, msg, max_output_tokens)[39m
    [32m    758[39m us = []
    [32m    759[39m [38;5;28;01mwhile[39;00m [38;5;28;01mTrue[39;00m:
    [32m--> [39m[32m760[39m     [38;5;28;01mtry[39;00m: res = [30;43mself[39;49m[30;43m.[39;49m[30;43m_model_step[39;49m[30;43m([39;49m[30;43mmax_output_tokens[39;49m[30;43m)[39;49m
    [32m    761[39m     [38;5;28;01mexcept[39;00m [38;5;167;01mException[39;00m [38;5;28;01mas[39;00m e:
    [32m    762[39m         [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m is_ctx_error([38;5;28mself[39m, e): [38;5;28;01mraise[39;00m

    [36mFile [39m[32m~/code/personal/orgs/rishi/rishi/cursor.py:243[39m, in [36mCursorChat._model_step[39m[34m(self, max_output_tokens)[39m
    [32m    241[39m [38;5;28;01mdef[39;00m[38;5;250m [39m[34m_model_step[39m([38;5;28mself[39m, max_output_tokens=[38;5;28;01mNone[39;00m):
    [32m    242[39m     [33m"[39m[33mOne wire call: through the live agent when there is one, else a whole conversation through the CLI.[39m[33m"[39m
    [32m--> [39m[32m243[39m     [38;5;28;01mif[39;00m [38;5;28mself[39m.use_sdk: [38;5;28;01mreturn[39;00m [30;43mself[39;49m[30;43m.[39;49m[30;43m_sdk_step[39;49m[30;43m([39;49m[30;43mmax_output_tokens[39;49m[30;43m)[39;49m
    [32m    244[39m     [38;5;28;01mreturn[39;00m [38;5;28mself[39m._note_usage(norm_cursor(json.loads([38;5;28mself[39m._run([33m'[39m[33mjson[39m[33m'[39m).stdout), [38;5;28mself[39m.model_id))

    [36mFile [39m[32m~/code/personal/orgs/rishi/rishi/cursor.py:334[39m, in [36m_sdk_step[39m[34m(self, max_output_tokens)[39m
    [32m    332[39m [33m"[39m[33mOne turn through the live agent: only what it has not already been told goes out.[39m[33m"[39m
    [32m    333[39m msg, n = [38;5;28mself[39m._tail()
    [32m--> [39m[32m334[39m run = [30;43mself[39;49m[30;43m.[39;49m[30;43magent[39;49m.send(msg)
    [32m    335[39m [38;5;28mself[39m._sent = n
    [32m    336[39m [38;5;28;01mreturn[39;00m [38;5;28mself[39m._note_usage([38;5;28mself[39m._sdk_resp(run))

    [36mFile [39m[32m~/code/personal/orgs/rishi/rishi/cursor.py:301[39m, in [36magent[39m[34m(self)[39m
    [32m    298[39m [38;5;129m@patch[39m(as_prop=[38;5;28;01mTrue[39;00m)
    [32m    299[39m [38;5;28;01mdef[39;00m[38;5;250m [39m[34magent[39m([38;5;28mself[39m:CursorChat):
    [32m    300[39m     [33m"[39m[33mThe live agent, built on first use and kept until something invalidates the conversation.[39m[33m"[39m
    [32m--> [39m[32m301[39m     [38;5;28;01mif[39;00m [38;5;28mself[39m._agent [38;5;129;01mis[39;00m [38;5;28;01mNone[39;00m: [38;5;28mself[39m._agent, [38;5;28mself[39m._sent = [30;43mself[39;49m[30;43m.[39;49m[30;43m_mk_agent[39;49m[30;43m([39;49m[30;43m)[39;49m, [32m0[39m
    [32m    302[39m     [38;5;28;01mreturn[39;00m [38;5;28mself[39m._agent

    [36mFile [39m[32m~/code/personal/orgs/rishi/rishi/cursor.py:296[39m, in [36m_mk_agent[39m[34m(self)[39m
    [32m    290[39m local = LocalAgentOptions(cwd=[38;5;28mstr[39m([38;5;28mself[39m.workspace [38;5;129;01mor[39;00m Path.cwd()),
    [32m    291[39m                           sandbox_options=[38;5;28;01mNone[39;00m [38;5;28;01mif[39;00m [38;5;28mself[39m.sandbox [38;5;129;01mis[39;00m [38;5;28;01mNone[39;00m [38;5;28;01melse[39;00m
    [32m    292[39m                           SandboxOptions(enabled=[38;5;28mself[39m.sandbox [38;5;129;01mnot[39;00m [38;5;129;01min[39;00m ([38;5;28;01mFalse[39;00m, [33m'[39m[33mdisabled[39m[33m'[39m)))
    [32m    293[39m opts = AgentOptions(model=cursor_model([38;5;28mself[39m.model_id, [38;5;28mself[39m.effort, [38;5;28mself[39m.fast, via=[33m'[39m[33msdk[39m[33m'[39m),
    [32m    294[39m                     api_key=[38;5;28mself[39m.api_key, mode=sdk_mode([38;5;28mself[39m.mode),
    [32m    295[39m                     tools=[38;5;28mself[39m.cursor_tools, disallowed_tools=[38;5;28mself[39m.cursor_disallowed, local=local)
    [32m--> [39m[32m296[39m [38;5;28;01mreturn[39;00m [30;43mAgent[39;49m[30;43m.[39;49m[30;43mcreate[39;49m[30;43m([39;49m[30;43mopts[39;49m[30;43m)[39;49m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_agent.py:119[39m, in [36mAgent.create[39m[34m(cls, options, client, model, api_key, name, local, cloud, idempotency_key)[39m
    [32m    105[39m [38;5;129m@classmethod[39m
    [32m    106[39m [38;5;28;01mdef[39;00m[38;5;250m [39m[34mcreate[39m(
    [32m    107[39m     [38;5;28mcls[39m,
    [32m   (...)[39m[32m    116[39m     idempotency_key: [38;5;28mstr[39m | [38;5;28;01mNone[39;00m = [38;5;28;01mNone[39;00m,
    [32m    117[39m ) -> [33m"[39m[33mAgent[39m[33m"[39m:
    [32m    118[39m     client = client [38;5;129;01mor[39;00m _default_client()
    [32m--> [39m[32m119[39m     [38;5;28;01mreturn[39;00m [30;43mclient[39;49m[30;43m.[39;49m[30;43mcreate_agent[39;49m[30;43m([39;49m
    [32m    120[39m [30;43m        [39;49m[30;43moptions[39;49m[30;43m,[39;49m
    [32m    121[39m [30;43m        [39;49m[30;43mmodel[39;49m[30;43m=[39;49m[30;43mmodel[39;49m[30;43m,[39;49m
    [32m    122[39m [30;43m        [39;49m[30;43mapi_key[39;49m[30;43m=[39;49m[30;43mapi_key[39;49m[30;43m,[39;49m
    [32m    123[39m [30;43m        [39;49m[30;43mname[39;49m[30;43m=[39;49m[30;43mname[39;49m[30;43m,[39;49m
    [32m    124[39m [30;43m        [39;49m[30;43mlocal[39;49m[30;43m=[39;49m[30;43mlocal[39;49m[30;43m,[39;49m
    [32m    125[39m [30;43m        [39;49m[30;43mcloud[39;49m[30;43m=[39;49m[30;43mcloud[39;49m[30;43m,[39;49m
    [32m    126[39m [30;43m        [39;49m[30;43midempotency_key[39;49m[30;43m=[39;49m[30;43midempotency_key[39;49m[30;43m,[39;49m
    [32m    127[39m [30;43m    [39;49m[30;43m)[39;49m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_client.py:428[39m, in [36mClient.create_agent[39m[34m(self, options, model, api_key, name, local, cloud, idempotency_key)[39m
    [32m    426[39m     request[[33m"[39m[33midempotencyKey[39m[33m"[39m] = idempotency_key
    [32m    427[39m [38;5;28;01mtry[39;00m:
    [32m--> [39m[32m428[39m     response = [30;43mself[39;49m[30;43m.[39;49m[30;43m_agent_unary[39;49m[30;43m([39;49m[30;43m"[39;49m[30;43mCreateAgent[39;49m[30;43m"[39;49m[30;43m,[39;49m[30;43m [39;49m[30;43mrequest[39;49m[30;43m)[39;49m
    [32m    429[39m [38;5;28;01mexcept[39;00m [38;5;167;01mException[39;00m:
    [32m    430[39m     [38;5;28;01mif[39;00m registered_agent_id [38;5;129;01mand[39;00m unregister_custom_tools [38;5;129;01mis[39;00m [38;5;129;01mnot[39;00m [38;5;28;01mNone[39;00m:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_client.py:675[39m, in [36mClient._agent_unary[39m[34m(self, method, message, skip_remote_cloud_guard)[39m
    [32m    673[39m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m skip_remote_cloud_guard:
    [32m    674[39m     [38;5;28mself[39m._require_explicit_api_key_for_remote_cloud_agent_rpc(method, message)
    [32m--> [39m[32m675[39m [38;5;28;01mreturn[39;00m [30;43mself[39;49m[30;43m.[39;49m[30;43m_transport[39;49m[30;43m.[39;49m[30;43munary[39;49m[30;43m([39;49m[30;43mAGENT_SERVICE[39;49m[30;43m,[39;49m[30;43m [39;49m[30;43mmethod[39;49m[30;43m,[39;49m[30;43m [39;49m[30;43mstrip_empty[39;49m[30;43m([39;49m[30;43mmessage[39;49m[30;43m)[39;49m[30;43m)[39;49m

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_connect.py:307[39m, in [36mConnectTransport.unary[39m[34m(self, service, method, message)[39m
    [32m    303[39m headers = [38;5;28mself[39m._headers(
    [32m    304[39m     accept=[33m"[39m[33mapplication/json[39m[33m"[39m, content_type=[33m"[39m[33mapplication/json[39m[33m"[39m
    [32m    305[39m )
    [32m    306[39m url = urljoin([38;5;28mself[39m.base_url, [33mf[39m[33m"[39m[38;5;132;01m{[39;00mservice[38;5;132;01m}[39;00m[33m/[39m[38;5;132;01m{[39;00mmethod[38;5;132;01m}[39;00m[33m"[39m)
    [32m--> [39m[32m307[39m response = [30;43m_post_with_retries[39;49m[30;43m([39;49m
    [32m    308[39m [30;43m    [39;49m[30;43mself[39;49m[30;43m.[39;49m[30;43m_client[39;49m[30;43m,[39;49m
    [32m    309[39m [30;43m    [39;49m[30;43murl[39;49m[30;43m,[39;49m
    [32m    310[39m [30;43m    [39;49m[30;43mbody[39;49m[30;43m,[39;49m
    [32m    311[39m [30;43m    [39;49m[30;43mheaders[39;49m[30;43m,[39;49m
    [32m    312[39m [30;43m    [39;49m[30;43mself[39;49m[30;43m.[39;49m[30;43m_httpx_timeout[39;49m[30;43m([39;49m[30;43mself[39;49m[30;43m.[39;49m[30;43munary_timeout[39;49m[30;43m)[39;49m[30;43m,[39;49m
    [32m    313[39m [30;43m    [39;49m[30;43mmax_retries[39;49m[30;43m=[39;49m[30;43mself[39;49m[30;43m.[39;49m[30;43mmax_retries[39;49m[30;43m,[39;49m
    [32m    314[39m [30;43m    [39;49m[30;43mservice[39;49m[30;43m=[39;49m[30;43mservice[39;49m[30;43m,[39;49m
    [32m    315[39m [30;43m    [39;49m[30;43mmethod[39;49m[30;43m=[39;49m[30;43mmethod[39;49m[30;43m,[39;49m
    [32m    316[39m [30;43m[39;49m[30;43m)[39;49m
    [32m    317[39m body_bytes = response.content
    [32m    318[39m [38;5;28;01mif[39;00m [38;5;129;01mnot[39;00m body_bytes:

    [36mFile [39m[32m~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_connect.py:370[39m, in [36m_post_with_retries[39m[34m(client, url, body, headers, timeout, max_retries, service, method)[39m
    [32m    368[39m         _sleep_before_retry(attempt)
    [32m    369[39m         [38;5;28;01mcontinue[39;00m
    [32m--> [39m[32m370[39m     [38;5;28;01mraise[39;00m _network_error(error) [38;5;28;01mfrom[39;00m[38;5;250m [39m[34;01merror[39;00m
    [32m    371[39m [38;5;28;01mif[39;00m response.status_code < [32m400[39m:
    [32m    372[39m     [38;5;28;01mreturn[39;00m response

    [31mNetworkError[39m: Bridge request failed: ConnectError: [Errno 61] Connection refused

## Knowing when to compress

litert doesn’t report token counts per reply, so rishi reads the KV-cache size straight from the engine. After each turn `chat.use` holds that turn’s input and output tokens, `chat.token_count` is the live context size, and `chat.pct_full` is that size over `ctx_limit`.

If the window does fill up mid-turn, rishi doesn’t let the turn die with a backend traceback: it shrinks the oldest tool results still in the history, rebuilds whatever state the backend holds, and asks the model to summarize with what’s left. [`ContextWindowExceededError`](https://vedicreader.github.io/rishi/core.html#contextwindowexceedederror) is raised only if that retry fails too. `recover_context` is the method a backend overrides to do this its own way, and litert does: it evicts the middle of the conversation and replays the turn.

For a long conversation, the cheaper move is to not reach the limit at all. [`SlidingWindowCallback`](https://vedicreader.github.io/rishi/core.html#slidingwindowcallback)
checks `pct_full` before each turn and, past a threshold, drops whole message groups from the middle
of the history - keeping the earliest turns and the recent thread - then has the backend rebuild from
what is left. Your system prompt is never evicted, and a tool call is never separated from its result.
Every backend registers it by default now, so a long conversation degrades instead of dying. It only
acts once `ctx_limit` is set, and it is deliberately conservative. Dropping turns is still lossy, so
`summarize=True` spends one model call to keep the gist of what went. To retune it, drop the default
instance and add your own; `default_cbs=False` clears it along with the rest of the defaults.

This matters most on litert, whose KV cache has no automatic recycling upstream: filling it OOMs on
GPU/NPU and makes the CPU path repeat itself indefinitely, rather than failing cleanly.

``` python
chat = Chat(gemma4_e2b, ctx_limit=4096)
chat.remove_cb(SlidingWindowCallback)                  # the default one, at threshold=0.9
chat.add_cb(SlidingWindowCallback(threshold=0.8, keep_first=2, keep_last=4, summarize=True))
for i in range(50): chat(f"Tell me fact number {i} about the sea.")
print(chat.pct_full, 'full;', getattr(chat, 'evicted', 0), 'messages evicted')
```

    0.5927734375 full; 55 messages evicted

``` python
chat('give me a summary of the conversation so far')
```

Here is a summary of our conversation so far:

The conversation has been a structured exchange where the user repeatedly requested a specific “Fact Number” about the sea. The assistant responded by providing a distinct, fundamental scientific or factual statement for each requested number, sequentially numbering them from 1 up to 49.

**Key Themes Covered:**

The facts covered a wide range of aspects of the sea, including:

- **Physical Characteristics:** Surface coverage, composition (saltwater), and physical forces (waves, tides).
- **Ecology & Biology:** Biodiversity, marine life webs, nutrient cycling, and habitat diversity.
- **Climate & Chemistry:** Role in the global carbon cycle, heat distribution, and water regulation.
- **Geology & History:** Influence on geological processes and recording ancient history.
- **Human Impact:** Role in food security, resource provision, and economic activity.
- **Extreme Environments:** Deep trenches and the limits of exploration.

The conversation successfully followed the pattern of providing a unique, foundational fact for each sequential number requested.

## Installing the skill

rishi bundles `skill.md`, an agent skill describing the API. A harness can install it into the standard skill directories (a dry run by default prints where it would write):

``` python
from rishi.core import mv_skill_md
mv_skill_md(dry_run=False)   # writes SKILL.md under .claude/skills/rishi/ and .agents/skills/rishi/
```

    Installed -> ['/Users/71293/code/personal/orgs/rishi/.agents/skills/rishi/SKILL.md', '/Users/71293/code/personal/orgs/rishi/.claude/skills/rishi/SKILL.md']

## Sharing a model and benchmarks

Loading a model costs a few seconds and a couple of gigabytes of RAM. To run several conversations off
one load, build the engine once and hand it to each chat. A [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) you build owns its engine and frees
it on `close()`; a [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) you hand an engine to leaves it alone, so the others keep working.

Each chat keeps its own history and its own conversation state, so they never see each other’s turns.
The engine underneath is the one thing they do share, so drive them one at a time. A single [`Chat`](https://vedicreader.github.io/rishi/core.html#chat) is a
single conversation - `hist`, the turn counters and the backend’s KV cache are all live state - and
neither one chat nor a shared engine is built to be called from two threads at once.

``` python
eng = LitertChat.create_engine(cache_dir='.cache/litertlm')
a, b = Chat(engine=eng), Chat(engine=eng)   # two chats over one loaded model
```

The default backend is CPU; for GPU pass `backend=Backend.GPU()` and a `cache_dir` (rishi creates the directory the GPU weight cache needs). [`bench()`](https://vedicreader.github.io/rishi/litert.html#bench) reports init time, time to first token, and prefill and decode tokens per second. Browse models at [huggingface.co/litert-community](https://huggingface.co/litert-community).

``` python
bench(cache_dir='.cache/litertlm')   # init time, time to first token, prefill and decode tok/s
```

    BenchmarkInfo(init_time_in_second=0.413105, time_to_first_token_in_second=0.6271803592499999, last_prefill_token_count=64, last_prefill_tokens_per_second=108.18723830098705, last_decode_token_count=64, last_decode_tokens_per_second=28.07935048952399)
