Metadata-Version: 2.5
Name: rishi
Version: 0.1.18
Summary: fast local llm driver with tools
Project-URL: Repository, https://github.com/vedicreader/rishi
Project-URL: Documentation, https://vedicreader.github.io/rishi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.11
Requires-Dist: fastcore>=2.0.0
Requires-Dist: huggingface-hub>=1.23.0
Requires-Dist: safepyrun>=0.2.3
Requires-Dist: truststore>=0.10
Provides-Extra: all
Requires-Dist: aidialog==0.0.7; extra == 'all'
Requires-Dist: claude-agent-sdk>=0.2; extra == 'all'
Requires-Dist: cursor-sdk>=0.1; extra == 'all'
Requires-Dist: diskcache>=5.6.3; extra == 'all'
Requires-Dist: httpx; extra == 'all'
Requires-Dist: litert-lm-api>=0.14.0; extra == 'all'
Requires-Dist: llama-cpp-python==0.3.30; extra == 'all'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: numpy; extra == 'all'
Requires-Dist: python-fastllm==0.0.36; extra == 'all'
Requires-Dist: soundfile>=0.14.0; extra == 'all'
Provides-Extra: claude
Requires-Dist: claude-agent-sdk>=0.2; extra == 'claude'
Provides-Extra: copilot
Requires-Dist: aidialog==0.0.7; extra == 'copilot'
Requires-Dist: httpx; extra == 'copilot'
Requires-Dist: python-fastllm==0.0.36; extra == 'copilot'
Provides-Extra: cursor
Requires-Dist: cursor-sdk>=0.1; extra == 'cursor'
Provides-Extra: dev
Requires-Dist: aidialog==0.0.7; extra == 'dev'
Requires-Dist: claude-agent-sdk>=0.2; extra == 'dev'
Requires-Dist: cursor-sdk>=0.1; extra == 'dev'
Requires-Dist: diskcache>=5.6.3; extra == 'dev'
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: litert-lm-api>=0.14.0; extra == 'dev'
Requires-Dist: llama-cpp-python==0.3.30; extra == 'dev'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'dev'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'dev'
Requires-Dist: numpy; extra == 'dev'
Requires-Dist: python-fastllm==0.0.36; extra == 'dev'
Requires-Dist: soundfile>=0.14.0; extra == 'dev'
Provides-Extra: litert
Requires-Dist: litert-lm-api>=0.14.0; extra == 'litert'
Provides-Extra: llama
Requires-Dist: llama-cpp-python==0.3.30; extra == 'llama'
Requires-Dist: numpy; extra == 'llama'
Requires-Dist: soundfile>=0.14.0; extra == 'llama'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx'
Provides-Extra: mlx-vlm
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx-vlm'
Provides-Extra: record
Requires-Dist: diskcache>=5.6.3; extra == 'record'
Provides-Extra: remote
Requires-Dist: aidialog==0.0.7; extra == 'remote'
Requires-Dist: python-fastllm==0.0.36; extra == 'remote'
Description-Content-Type: text/markdown

# rishi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

Pass a model id to [`Chat`](https://vedicreader.github.io/rishi/core.html#chat), then call it with a prompt. Each call returns a [`Resp`](https://vedicreader.github.io/rishi/core.html#resp) and appends to
`chat.hist`. Use `resp_text(r)` for the answer and `thought(r)` for reasoning. Every backend uses
the same tools, approval gate, callbacks and streaming interface.

## Install

``` sh
pip install 'rishi[litert]'    # .litertlm Gemma builds
pip install 'rishi[llama]'     # any GGUF
pip install 'rishi[mlx]'       # Apple Silicon (add mlx-vlm for vision and audio)
pip install 'rishi[remote]'    # Claude, GPT, Gemini and friends, via fastllm
pip install 'rishi[cursor]'    # Cursor models via SDK or cursor-agent CLI
pip install 'rishi[claude]'    # Claude Code via the Agent SDK (the `claude` CLI needs nothing)
pip install 'rishi[copilot]'   # GitHub Copilot, on your Copilot subscription
pip install 'rishi[all]'       # everything your platform supports
```

Extras combine, as in `rishi[litert,remote]`. Backend modules load only when used. `import rishi`
does not require wheels for other backends.

Contributors install with `pip install -e '.[dev]'`, then run `nbdev-prepare`. Edit the source
notebooks in `nbs/`. `nbdev-export` generates `rishi/*.py`.

## Quickstart

This example runs Gemma 4 E2B locally through LiteRT. The first call downloads about 2 GB of
weights. Later calls use the cache.

``` python
chat = Chat(gemma4_e2b)
r = chat('Give me one fact about lobsters.')
print(resp_text(r))
chat('And one more.')              # same conversation
chat.print_hist()
```

    Lobsters are crustaceans, which means they have a hard exoskeleton and a segmented body.

**user**

Give me one fact about lobsters.

------------------------------------------------------------------------

**assistant**

Lobsters are crustaceans, which means they have a hard exoskeleton and a segmented body.

------------------------------------------------------------------------

**user**

And one more.

------------------------------------------------------------------------

**assistant**

Lobsters are known for their ability to hold their breath for extended periods of time.

## Pick a backend

`Chat(model)` chooses a backend from the model id. `chat.runtime` reports the choice. Use `runtime=`
or a prefix such as `llama/` when the id is ambiguous.

| model id looks like                   | backend                  |
|---------------------------------------|--------------------------|
| `litert-community/...`, `.litertlm`   | litert                   |
| `...-GGUF`, `.gguf` path              | llama.cpp                |
| `mlx-community/...`                   | MLX                      |
| `claude-...`, `gpt-...`, `gemini-...` | remote (fastllm)         |
| `cursor/...` or `CursorChat(...)`     | cursor                   |
| `claude/...` or `ClaudeChat(...)`     | claude (Claude Code)     |
| `copilot/...` or `CopilotChat(...)`   | copilot (GitHub Copilot) |

Claude Code and Copilot require a prefix. A bare `claude-...` or `gpt-...` id continues to use the
hosted API through `remote`.

``` python
print(resolve_runtime('litert-community/gemma-4-E2B-it-litert-lm'))
print(resolve_runtime('Qwen/Qwen3-4B-GGUF'))
print(resolve_runtime('mlx-community/Qwen3-4B-4bit'))
print(resolve_runtime('claude-sonnet-4-5'))
print(resolve_runtime('cursor/default'))
```

    ('litert', 'litert-community/gemma-4-E2B-it-litert-lm')
    ('llama', 'Qwen/Qwen3-4B-GGUF')
    ('mlx', 'mlx-community/Qwen3-4B-4bit')
    ('remote', 'claude-sonnet-4-5')
    ('cursor', 'default')

## Feature tour

The same call shape covers the rest of the API. Replace `gemma4_e2b` with any compatible model.

### Stream and async

`stream=True` yields markdown chunks. `display_stream(...)` renders them live in a notebook.
[`AsyncChat`](https://vedicreader.github.io/rishi/core.html#asyncchat) adds `await` and `async for` to any chat.

``` python
for chunk in chat('Write a haiku about the sea.', stream=True):
    print(chunk, end='', flush=True)

achat = AsyncChat(chat)
print(resp_text(await achat('One more fact, please.')))
```

    Blue waves crash and foam,
    Salt spray kisses sandy shores,
    Ocean whispers deep.Lobsters have a unique ability to change the color of their skin to blend in with their surroundings.

### Reasoning

`think=True` on construction exposes a thinking channel. `filter_think=True`, the default, keeps it
out of later context.

``` python
ch = Chat(gemma4_e2b, backend=Backend.GPU(), think=True)
r = ch('A bat and ball cost $1.10; the bat is $1 more than the ball. Price of the ball?')
print(resp_text(r), '→', thought(r)[:80], '…')
```

    This is a classic riddle that requires setting up a system of equations.

    Here is the step-by-step solution:

    1. **Define variables:**
       * Let $B$ be the cost of the bat.
       * Let $L$ be the cost of the ball.

    2. **Set up the equations based on the clues:**
       * **Clue 1:** The bat and ball cost $1.10.
         $$B + L = 1.10$$
       * **Clue 2:** The bat is $1 more than the ball.
         $$B = L + 1.00$$

    3. **Substitute** the second equation into the first equation:
       $$(L + 1.00) + L = 1.10$$

    4. **Solve for L:**
       $$2L + 1.00 = 1.10$$
       $$2L = 1.10 - 1.00$$
       $$2L = 0.10$$
       $$L = 0.05$$

    **Answer:** The price of the ball is **$0.05** (5 cents).

    *(If you check the answer: The bat would cost $1.05, and $1.05 + $0.05 = $1.10.)* → Here's a thinking process to solve this classic riddle:

    1.  **Define the variab …

### Images and audio

Pass a `PIL.Image`, `bytes` or a `Path` beside the text. Rishi detects image and audio input.
Gemma 4 LiteRT builds accept both.

``` python
from fastcore.all import img_bytes, Path
from PIL import Image
im = Image.open(Path(repo_root()/'nbs/images.jpeg')); im
```

![](index_files/figure-commonmark/cell-6-output-1.png)

``` python
print(resp_text(chat(['Explain this image.', img_bytes(im)])))
print(resp_text(chat(['Transcribe this clip.', Path(repo_root()/'nbs/speech.wav')])))
```

    This image features a beautiful, medium-sized dog with long, reddish-brown fur, likely a German Shepherd, walking down a dirt or gravel path in a natural, outdoor setting.

    Here are some details about the image:

    *   **Subject:** The main subject is a dog, characterized by its rich, warm brown coat and erect, pointed ears. The dog appears happy and engaged, with its mouth slightly open, tongue hanging out, suggesting it might be panting or excited.
    *   **Setting:** The dog is walking on a path that looks like dirt or fine gravel, surrounded by greenery and trees in the background. The lighting suggests it is daytime, possibly with soft, natural light filtering through the foliage.
    *   **Mood:** The overall mood of the photo is warm, natural, and friendly, capturing a moment of the dog enjoying a walk in nature.
    Dancing in the masquerade, idle truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems a sleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

### Tools with approval

Plain functions become tools. `approve` runs before each call. [`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy) maps tool names to
`approved`, `dont_run` or `check`. `max_steps` limits tool rounds per turn.

``` python
def add(a: int, b: int) -> int:
    'Add two integers.'
    return a + b

def delete_files(path: str) -> str:
    'Delete everything under a path.'
    return f'wiped {path}'

approve = hitl_policy({'add': 'approved', 'delete_files': 'dont_run'})
tchat = Chat(gemma4_e2b, tools=[add, delete_files], approve=approve)
print(resp_text(tchat('Add 2 and 3, then delete /tmp/data.')))
```

    I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file `/tmp/data.<system-reminder>`.

### Reconfigure and one-shots

`chat.reconfigure(sp=, tools=)` changes the briefing or tools without losing the conversation.
`chat.oneshot(...)` runs a stateless side task without changing `hist`.

`CachedChat(path='nbs/chatcache')` replays recorded turns in docs and CI without loading weights.

``` python
from rishi.core import CachedChat

c = CachedChat(path=repo_root()/'nbs/chatcache', max_output_tokens=64, record=True)
q = 'Say hello in one short sentence.'
print(resp_text(c(q)))
c.reconfigure(sp='You are a pirate. Always talk like one.')
print(resp_text(c(q)))                     # same thread, new briefing
print(c.oneshot('One word — sentiment of "the train was late again".', think=False, max_tokens=16))
```

    Hello there!
    Ahoy there, matey!
    **Frustration**

### Hand off between backends

`chat.hist` is backend-agnostic. Pass it as `messages=` to continue the conversation on another
local or hosted backend.

``` python
local = Chat(qwen3_4b, n_ctx=4096, tools=[add])
local('What is 2 + 3? Use the add tool.')
remote = Chat('gpt-4.1-nano', messages=local.hist, tools=[add])   # needs API key
print(resp_text(remote('What did I ask, and what was the answer?')))
local.close(); remote.close()
```

    llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    You asked what 2 + 3 is, and I found the answer to be 5 after using the add tool.

### Run Python from replies

[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback) executes fenced Python from a reply and returns stdout to the model. It stops when
the model answers in prose or `done` returns true.

``` python
py = Chat(gemma4_e2b, sp='Use a ```python fence, then answer in prose.')
py('What is 2**100?', cbs=[PyFenceCallback(done=output_matches(str(2**100)))])
```

### Structured output and checks

`structured` returns a dataclass instance. `classify` picks a label. Both run a throwaway turn and
leave `hist` unchanged. `check` grades a fenced answer against an expected string.

``` python
from dataclasses import dataclass

@dataclass
class Person: name: str; age: int

print(chat.structured('Extract: John Smith is 30.', Person))
print(chat.classify('I loved this film!', ['positive', 'negative']))
print(chat.check('Capital of France?', 'Paris'))
```

## Other backends

| backend | install | typical use |
|----|----|----|
| MLX | `rishi[mlx]` | Apple Silicon, with an explicit prompt cache, `kv_bits` and LoRA |
| remote | `rishi[remote]` plus a vendor key | the same tools and HITL as local, with `tool_choice` and `reasoning_effort` |
| cursor | `rishi[cursor]` plus `$CURSOR_API_KEY`, or `cursor-agent login` | Cursor-only models, through the `cursor/` prefix or [`CursorChat`](https://vedicreader.github.io/rishi/cursor.html#cursorchat) |
| claude | `rishi[claude]`, or just Claude Code plus `claude /login` | your tools without MCP, through the `claude/` prefix or [`ClaudeChat`](https://vedicreader.github.io/rishi/claude.html#claudechat) |
| copilot | `rishi[copilot]` plus a Copilot subscription | many vendors’ models on one subscription, through the `copilot/` prefix or [`CopilotChat`](https://vedicreader.github.io/rishi/copilot.html#copilotchat) |

``` python
# MLX (Apple Silicon)
from rishi.mlx import qwen3_4b as mlx_qwen
m = Chat(mlx_qwen); print(resp_text(m('One octopus fact.'))); m.close()

# Hosted — hand local history to a bigger model
loc = Chat(qwen3_4b); loc('My name is Karthik and my favourite number is 17.')
big = Chat('gpt-4.1-nano', messages=loc.hist)
print(resp_text(big('What is my name and favourite number?'))); loc.close(); big.close()

# Cursor — SDK path (prefix required for plain `Chat`)
from rishi.cursor import grok45, CursorChat
cu = CursorChat(grok45, effort='low'); print(resp_text(cu('Kalman filter in one sentence.'))); cu.close()
```

## Claude Code

`Chat('claude/claude-sonnet-5')` runs through Claude Code. `via='sdk'` uses the
`claude-agent-sdk` package from `rishi[claude]`. `via='cli'` uses the `claude` binary. `via=None`
prefers the SDK when installed. `ClaudeChat.local` is `False`.

Managed Claude Code installations can reject dynamic MCP servers. Rishi instead puts tool schemas
in the system prompt and receives calls as `<tool_call>` tags. It opens no MCP server and does not
set `--strict-mcp-config`.

The briefing and tool schemas use Claude Code’s system-prompt channel. Only the conversation is
rendered into the prompt. Claude Code’s own system prompt adds 48k to 53k prompt tokens per turn.
Tool calls also depend on the model producing the tags correctly. Use a Sonnet-tier or stronger
model for tool-using turns.

## GitHub Copilot

`Chat('copilot/gpt-4.1')` runs on a Copilot subscription. `rishi.copilot` supplies the short-lived
token and editor headers required by the endpoint. [`RemoteChat`](https://vedicreader.github.io/rishi/remote.html#remotechat) then handles tools, streaming and
approval.

[`copilot_oauth()`](https://vedicreader.github.io/rishi/copilot.html#copilot_oauth) first checks Copilot-specific environment variables, then editor sign-in files.
It checks `GH_TOKEN` and `GITHUB_TOKEN` last because they often contain personal access tokens,
which Copilot rejects. [`copilot_login()`](https://vedicreader.github.io/rishi/copilot.html#copilot_login) runs GitHub’s device flow when no usable sign-in exists.
[`copilot_models()`](https://vedicreader.github.io/rishi/copilot.html#copilot_models) returns the models available to the current account and plan.

``` python
from rishi.copilot import copilot_models, CopilotAuth

print(copilot_models()[:5])

auth = CopilotAuth()                       # one exchange, shared by every chat that holds it
chat = Chat('copilot/claude-sonnet-4.5', auth=auth, sp='You are concise.')
```

This integration is reverse-engineered and unsupported by GitHub. The Copilot subscription terms
apply.

``` python
from rishi.claude import sonnet5, ClaudeChat
cc = ClaudeChat(sonnet5, tools=[add], sp='Use the add tool.')   # or Chat('claude/claude-sonnet-5')
print(resp_text(cc('What is 2 + 3?')))
cc.close()
```

## Backend guides

| notebook | topics |
|----|----|
| [`00_core.ipynb`](core.html) | callbacks, context compression, [`SlidingWindowCallback`](https://vedicreader.github.io/rishi/core.html#slidingwindowcallback), shared engines, skill install, grading judges |
| [`01_llama.ipynb`](llama.html) | GGUF models, GPU offload, parallel tools, audio via mtmd |
| [`02_litert.ipynb`](litert.html) | Gemma `.litertlm`, GPU and NPU, [`bench()`](https://vedicreader.github.io/rishi/litert.html#bench) |
| [`03_mlx.ipynb`](mlx.html) | vision and audio routing, speculative decoding, cache save and load |
| [`04_remote.ipynb`](remote.html) | provider tools, server-side search |
| [`05_cursor.ipynb`](cursor.html) | CLI against SDK, model ids, agent modes |
| [`06_claude.ipynb`](claude.html) | SDK against CLI, tools as prompt tags, MCP under a managed policy |
| [`07_copilot.ipynb`](copilot.html) | the token exchange, editor headers, model listing |
