Metadata-Version: 2.5
Name: rishi
Version: 0.1.6
Summary: fast local llm driver with tools
Project-URL: Repository, https://github.com/vedicreader/rishi
Project-URL: Documentation, https://vedicreader.github.io/rishi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.11
Requires-Dist: fastcore>=2.0.0
Requires-Dist: huggingface-hub>=1.23.0
Requires-Dist: safepyrun>=0.2.3
Provides-Extra: all
Requires-Dist: aidialog==0.0.7; extra == 'all'
Requires-Dist: cursor-sdk>=0.1; extra == 'all'
Requires-Dist: diskcache>=5.6.3; extra == 'all'
Requires-Dist: litert-lm-api>=0.14.0; extra == 'all'
Requires-Dist: llama-cpp-python==0.3.30; extra == 'all'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: numpy; extra == 'all'
Requires-Dist: python-fastllm==0.0.36; extra == 'all'
Requires-Dist: soundfile>=0.14.0; extra == 'all'
Provides-Extra: cursor
Requires-Dist: cursor-sdk>=0.1; extra == 'cursor'
Provides-Extra: litert
Requires-Dist: litert-lm-api>=0.14.0; extra == 'litert'
Provides-Extra: llama
Requires-Dist: llama-cpp-python==0.3.30; extra == 'llama'
Requires-Dist: numpy; extra == 'llama'
Requires-Dist: soundfile>=0.14.0; extra == 'llama'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx'
Provides-Extra: mlx-vlm
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx-vlm'
Provides-Extra: record
Requires-Dist: diskcache>=5.6.3; extra == 'record'
Provides-Extra: remote
Requires-Dist: aidialog==0.0.7; extra == 'remote'
Requires-Dist: python-fastllm==0.0.36; extra == 'remote'
Description-Content-Type: text/markdown

# rishi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

rishi is a thin chat layer: pass a model id, get a callable [`Chat`](https://vedicreader.github.io/rishi/core.html#chat). History stays in `chat.hist`; each call returns a [`Resp`](https://vedicreader.github.io/rishi/core.html#resp) (`resp_text(r)` for the answer, `thought(r)` for reasoning). Backends share the same tool loop, approval gate, callbacks, and streaming — details live in the backend notebooks linked at the end.

## Install

``` sh
pip install 'rishi[litert]'    # .litertlm Gemma builds
pip install 'rishi[llama]'     # any GGUF
pip install 'rishi[mlx]'       # Apple Silicon (add mlx-vlm for vision/audio)
pip install 'rishi[remote]'    # Claude, GPT, Gemini, … via fastllm
pip install 'rishi[cursor]'    # Cursor models via SDK or cursor-agent CLI
pip install 'rishi[all]'       # everything your platform supports
```

Extras combine (`rishi[litert,remote]`). Backend modules import lazily — `import rishi` never pulls in wheels you didn’t install.

**Contributors:** `pip install -e '.[dev]'` then `nbdev-prepare`. Notebooks in `nbs/` are the source; `rishi/*.py` is generated.

## Quickstart

First call downloads weights (here, Gemma-4-E2B via litert, ~2GB); later calls use the cache.

``` python
chat = Chat(gemma4_e2b)
r = chat('Give me one fact about lobsters.')
print(resp_text(r))
chat('And one more.')              # same conversation
chat.print_hist()
```

    Lobsters are crustaceans, which means they have a hard exoskeleton and a segmented body.

**user**

Give me one fact about lobsters.

------------------------------------------------------------------------

**assistant**

Lobsters are crustaceans, which means they have a hard exoskeleton and a segmented body.

------------------------------------------------------------------------

**user**

And one more.

------------------------------------------------------------------------

**assistant**

Lobsters are known for their ability to hold their breath for extended periods of time.

## Pick a backend

`Chat(model)` routes from the id. `chat.runtime` tells you which engine you got; force with `runtime=` or a `llama/…` prefix when the name is ambiguous.

| model id looks like               | backend          |
|-----------------------------------|------------------|
| `litert-community/…`, `.litertlm` | litert           |
| `…-GGUF`, `.gguf` path            | llama.cpp        |
| `mlx-community/…`                 | MLX              |
| `claude-…`, `gpt-…`, `gemini-…`   | remote (fastllm) |
| `cursor/…` or `CursorChat(…)`     | cursor           |

``` python
print(resolve_runtime('litert-community/gemma-4-E2B-it-litert-lm'))
print(resolve_runtime('Qwen/Qwen3-4B-GGUF'))
print(resolve_runtime('mlx-community/Qwen3-4B-4bit'))
print(resolve_runtime('claude-sonnet-4-5'))
print(resolve_runtime('cursor/default'))
```

    ('litert', 'litert-community/gemma-4-E2B-it-litert-lm')
    ('llama', 'Qwen/Qwen3-4B-GGUF')
    ('mlx', 'mlx-community/Qwen3-4B-4bit')
    ('remote', 'claude-sonnet-4-5')
    ('cursor', 'default')

## Feature tour

One example per capability. Swap `gemma4_e2b` for any backend — the call shape is the same.

### Stream and async

`stream=True` yields markdown chunks; `display_stream(...)` renders live in a notebook. [`AsyncChat`](https://vedicreader.github.io/rishi/core.html#asyncchat) wraps any chat for `await` and `async for`.

``` python
for chunk in chat('Write a haiku about the sea.', stream=True):
    print(chunk, end='', flush=True)

achat = AsyncChat(chat)
print(resp_text(await achat('One more fact, please.')))
```

    Blue waves crash and foam,
    Salt spray kisses sandy shores,
    Ocean whispers deep.Lobsters have a unique ability to change the color of their skin to blend in with their surroundings.

### Reasoning

`think=True` on construction exposes a thinking channel; `filter_think=True` (default) keeps it out of later context.

``` python
ch = Chat(gemma4_e2b, backend=Backend.GPU(), think=True)
r = ch('A bat and ball cost $1.10; the bat is $1 more than the ball. Price of the ball?')
print(resp_text(r), '→', thought(r)[:80], '…')
```

    This is a classic riddle that requires setting up a system of equations.

    Here is the step-by-step solution:

    1. **Define variables:**
       * Let $B$ be the cost of the bat.
       * Let $L$ be the cost of the ball.

    2. **Set up the equations based on the clues:**
       * **Clue 1:** The bat and ball cost $1.10.
         $$B + L = 1.10$$
       * **Clue 2:** The bat is $1 more than the ball.
         $$B = L + 1.00$$

    3. **Substitute** the second equation into the first equation:
       $$(L + 1.00) + L = 1.10$$

    4. **Solve for L:**
       $$2L + 1.00 = 1.10$$
       $$2L = 1.10 - 1.00$$
       $$2L = 0.10$$
       $$L = 0.05$$

    **Answer:** The price of the ball is **$0.05** (5 cents).

    *(If you check the answer: The bat would cost $1.05, and $1.05 + $0.05 = $1.10.)* → Here's a thinking process to solve this classic riddle:

    1.  **Define the variab …

### Images and audio

Pass `PIL.Image`, `bytes`, or `Path` beside text — rishi tags image vs audio. Gemma-4 litert builds are multimodal out of the box.

``` python
from fastcore.all import img_bytes, Path
from PIL import Image
im = Image.open(Path(repo_root()/'nbs/images.jpeg')); im
```

![](index_files/figure-commonmark/cell-6-output-1.png)

``` python
print(resp_text(chat(['Explain this image.', img_bytes(im)])))
print(resp_text(chat(['Transcribe this clip.', Path(repo_root()/'nbs/speech.wav')])))
```

    This image features a beautiful, medium-sized dog with long, reddish-brown fur, likely a German Shepherd, walking down a dirt or gravel path in a natural, outdoor setting.

    Here are some details about the image:

    *   **Subject:** The main subject is a dog, characterized by its rich, warm brown coat and erect, pointed ears. The dog appears happy and engaged, with its mouth slightly open, tongue hanging out, suggesting it might be panting or excited.
    *   **Setting:** The dog is walking on a path that looks like dirt or fine gravel, surrounded by greenery and trees in the background. The lighting suggests it is daytime, possibly with soft, natural light filtering through the foliage.
    *   **Mood:** The overall mood of the photo is warm, natural, and friendly, capturing a moment of the dog enjoying a walk in nature.
    Dancing in the masquerade, idle truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems a sleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

### Tools with approval

Plain functions become tools; `approve` runs before each call. [`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy) maps tool names to `approved` / `dont_run` / `check`. `max_steps` caps tool rounds per turn.

``` python
def add(a: int, b: int) -> int:
    'Add two integers.'
    return a + b

def delete_files(path: str) -> str:
    'Delete everything under a path.'
    return f'wiped {path}'

approve = hitl_policy({'add': 'approved', 'delete_files': 'dont_run'})
tchat = Chat(gemma4_e2b, tools=[add, delete_files], approve=approve)
print(resp_text(tchat('Add 2 and 3, then delete /tmp/data.')))
```

    I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file `/tmp/data.<system-reminder>`.

### Reconfigure and one-shots

`chat.reconfigure(sp=, tools=)` changes briefing or tools mid-conversation. `chat.oneshot(...)` is a stateless side call — label, summarize, complete — without touching `hist`.

`CachedChat(path='nbs/chatcache')` replays recorded turns for docs/CI without loading weights.

``` python
from rishi.core import CachedChat

c = CachedChat(path=repo_root()/'nbs/chatcache', max_output_tokens=64, record=True)
q = 'Say hello in one short sentence.'
print(resp_text(c(q)))
c.reconfigure(sp='You are a pirate. Always talk like one.')
print(resp_text(c(q)))                     # same thread, new briefing
print(c.oneshot('One word — sentiment of "the train was late again".', think=False, max_tokens=16))
```

    Hello there!
    Ahoy there, matey!
    **Frustration**

### Hand off between backends

`chat.hist` is backend-agnostic — start local, continue on a hosted model (or another local engine) with `messages=`.

``` python
local = Chat(qwen3_4b, n_ctx=4096, tools=[add])
local('What is 2 + 3? Use the add tool.')
remote = Chat('gpt-4.1-nano', messages=local.hist, tools=[add])   # needs API key
print(resp_text(remote('What did I ask, and what was the answer?')))
local.close(); remote.close()
```

    llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    You asked what 2 + 3 is, and I found the answer to be 5 after using the add tool.

### Run Python from replies

[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback) executes fenced Python blocks from the model’s reply, feeds stdout back, and loops until the model answers in prose or `done` says stop.

``` python
py = Chat(gemma4_e2b, sp='Use a ```python fence, then answer in prose.')
py('What is 2**100?', cbs=[PyFenceCallback(done=output_matches(str(2**100)))])
```

### Structured output and checks

`structured` returns a dataclass instance; `classify` picks a label. Both use a throwaway turn and leave `hist` alone. `check` grades a fenced answer against an expected string.

``` python
from dataclasses import dataclass

@dataclass
class Person: name: str; age: int

print(chat.structured('Extract: John Smith is 30.', Person))
print(chat.classify('I loved this film!', ['positive', 'negative']))
print(chat.check('Capital of France?', 'Paris'))
```

## Same API on MLX, hosted, and Cursor

| backend | install | typical use |
|----|----|----|
| MLX | `rishi[mlx]` | Apple Silicon; explicit prompt cache, `kv_bits`, LoRA |
| remote | `rishi[remote]` + vendor key | same tools/HITL as local; `tool_choice`, `reasoning_effort` |
| cursor | `rishi[cursor]` + `$CURSOR_API_KEY`, or `cursor-agent login` | Cursor-only models; use `cursor/` prefix or [`CursorChat`](https://vedicreader.github.io/rishi/cursor.html#cursorchat) |

See `03_mlx.ipynb`, `04_remote.ipynb`, `05_cursor.ipynb` for knobs and examples.

`<<<<<<< HEAD`

``` python
# MLX (Apple Silicon)
from rishi.mlx import qwen3_4b as mlx_qwen
m = Chat(mlx_qwen); print(resp_text(m('One octopus fact.'))); m.close()

# Hosted — hand local history to a bigger model
loc = Chat(qwen3_4b); loc('My name is Karthik and my favourite number is 17.')
big = Chat('gpt-4.1-nano', messages=loc.hist)
print(resp_text(big('What is my Name and my favourite number?'))); loc.close(); big.close()

# Cursor — SDK path (prefix required for plain `Chat`)
from rishi.cursor import grok45, CursorChat
cu = CursorChat(grok45, effort='low'); print(resp_text(cu('Kalman filter in one sentence.'))); cu.close()
```

`=======`

``` python
# MLX (Apple Silicon)
from rishi.mlx import qwen3_4b as mlx_qwen
m = Chat(mlx_qwen); print(resp_text(m('One octopus fact.'))); m.close()

# Hosted — hand local history to a bigger model
loc = Chat(qwen3_4b); loc('My name is Karthik and my favourite number is 17.')
big = Chat('gpt-4.1-nano', messages=loc.hist)
print(resp_text(big('What is my name and favourite number?'))); loc.close(); big.close()

# Cursor — SDK path (prefix required for plain `Chat`)
from rishi.cursor import grok45, CursorChat
cu = CursorChat(grok45, effort='low'); print(resp_text(cu('Kalman filter in one sentence.'))); cu.close()
```

`>>>>>>> cursor/streamline-backend-notebooks-e91f`

## Go deeper

| notebook | topics |
|----|----|
| [`00_core.ipynb`](core.html) | callbacks, context compression, [`SlidingWindowCallback`](https://vedicreader.github.io/rishi/core.html#slidingwindowcallback), shared engines, skill install, grading judges |
| [`01_llama.ipynb`](llama.html) | GGUF models, GPU offload, parallel tools, audio via mtmd |
| [`02_litert.ipynb`](litert.html) | Gemma `.litertlm`, GPU/NPU, [`bench()`](https://vedicreader.github.io/rishi/litert.html#bench) |
| [`03_mlx.ipynb`](mlx.html) | vision/audio routing, speculative decoding, cache save/load |
| [`04_remote.ipynb`](remote.html) | provider tools, server-side search |
| [`05_cursor.ipynb`](cursor.html) | CLI vs SDK, model ids, agent modes |
