Metadata-Version: 2.4
Name: pylolm
Version: 0.1.4
Summary: LoLm: run GGUF language models on-device on Android from Python (llama.cpp bindings for Kivy, Flet and Termux).
Author-email: AuraWay <auraway.dev@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/aura-way/pylolm
Project-URL: Issues, https://github.com/aura-way/pylolm/issues
Project-URL: Changelog, https://github.com/aura-way/pylolm/blob/main/CHANGELOG.md
Keywords: llm,llama.cpp,gguf,android,termux,kivy,flet,on-device,local-llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: Android
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: C
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: jinja
Requires-Dist: jinja2>=3.0; extra == "jinja"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: jinja2>=3.0; extra == "dev"
Dynamic: license-file

# pylolm (LoLm)

**Local LLMs on Android, from Python.** An Android-friendly take on `llama-cpp-python`, built on
[llama.cpp](https://github.com/ggml-org/llama.cpp) and designed for **Kivy**, **Flet** and **Termux**.

- Zero Python dependencies: a thin `ctypes` layer over a small C shim.
- `llama-cpp-python`-style API: `Llama(...)`, `create_chat_completion(...)`, `create_completion(...)`.
- Streaming generators, stop strings, a `stop()` button for your UI.
- Multi-turn chat that reuses the KV cache, so long conversations stay fast.
- Aimed at small GGUF models (0.5B-2B) on phones.

> **Status: alpha (0.1.0).** Text generation only. TTS / STT / OCR are on the roadmap.

## Install

```bash
pip install pylolm
pylolm get-libs      # downloads the prebuilt native libraries (seconds)
pylolm doctor        # checks that everything loads
```

`get-libs` fetches the Android arm64 libraries attached to the matching GitHub Release, checks their
SHA-256 checksum, and installs them to `~/.pylolm/lib`. These are built by CI to run on a wide range
of phones.

Prefer to compile for your exact device, or `get-libs` is not available for your setup?

```bash
pkg install cmake clang git
pylolm build-native  # clones llama.cpp and builds on the device (slow: ~30 min)
```

Libraries are searched in `$PYLOLM_LIB_DIR`, then `pylolm/lib/`, then `~/.pylolm/lib/`.

## Quick start

```python
from pylolm import Llama

llm = Llama("/sdcard/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf", n_ctx=2048)

messages = [{"role": "user", "content": "Explain GGUF in two sentences."}]

# whole answer
out = llm.create_chat_completion(messages, max_tokens=200)
print(out["choices"][0]["message"]["content"])

# streaming
for chunk in llm.create_chat_completion(messages, max_tokens=200, stream=True):
    piece = chunk["choices"][0]["delta"].get("content")
    if piece:
        print(piece, end="", flush=True)
```

Try it from the terminal: `pylolm chat /path/to/model.gguf`. Use `pylolm info /path/to/model.gguf`
to see a model's details and chat template, and `--chat-template NAME` if its template is not
recognised.

## API

| Call | Notes |
|---|---|
| `Llama(model_path, n_ctx=2048, n_threads=None, verbose=False)` | `n_threads` defaults to `min(4, cpu_count)` |
| `create_chat_completion(messages, max_tokens=256, temperature=0.7, top_p=0.95, top_k=40, min_p=0.05, repeat_penalty=1.0, seed=None, stop=None, stream=False, chat_template=None)` | OpenAI-style dict, or chunk iterator with `stream=True` |
| `create_completion(prompt, ...)` / `llm(prompt, ...)` | raw text completion |
| `stop()` | stop the running generation from another thread |
| `reset()` | clear the KV cache (forget the conversation) |
| `count_tokens(text)` | tokens used by a string |
| `format_chat(messages, chat_template=None)` | the prompt text for this model (debugging) |
| `chat_template` (property) | the model's own Jinja chat template, or `None` |
| `close()` / `with Llama(...) as llm:` | free the model |

Errors: `LibraryNotFoundError`, `ModelLoadError`, `ContextOverflowError`, `GenerationError`
(all subclass `PyLoLmError`).

`usage` in a non-streaming result includes `cached_prompt_tokens`: how many prompt tokens were
reused from the previous turn.

**Chat templates.** Every chat model expects its own prompt format. pylolm handles this for you:
it first tries llama.cpp's built-in templates (ChatML/Qwen, Llama 3, Gemma 2/3, ...), and if the
model's template is not one of those (typical for brand-new models) it renders the template stored
inside the GGUF file itself, in Python. That second path needs the small optional `jinja2`
package: `pip install "pylolm[jinja]"`.

Use the `chat_template=` argument to override: `"jinja"` (always use the model's own template),
a built-in name such as `"chatml"` or `"gemma"`, or your own Jinja string.
`llm.format_chat(messages)` shows the exact prompt text that will be sent.

Debugging a model: `pylolm info model.gguf` (no native libraries needed) prints its metadata and chat
template, and `pylolm template model.gguf` loads it and shows how prompts get formatted.

## Using it in apps

**UI threads.** Generation blocks, so run it off the UI thread. A streaming generator is easiest:

```python
import threading

def worker():
    for chunk in llm.create_chat_completion(messages, stream=True):
        text = chunk["choices"][0]["delta"].get("content")
        if text:
            Clock.schedule_once(lambda dt, t=text: append_to_label(t))  # Kivy
            # page.run_thread / page.update(...) for Flet

threading.Thread(target=worker, daemon=True).start()
```

Call `llm.stop()` from your Stop button; breaking out of the loop also stops generation.

**Flet.** `flet build apk` installs packages from PyPI using the Android platform tag, so the
`android_24_arm64_v8a` wheel is what ships the native libraries. This path needs on-device
verification for your Flet/Python version.

**Kivy / python-for-android.** p4a does not consume Android wheels yet. For now, bundle the
`.so` files with your app and point `PYLOLM_LIB_DIR` at them before importing pylolm:

```python
import os
os.environ["PYLOLM_LIB_DIR"] = os.path.join(os.path.dirname(__file__), "pylolm_libs")
import pylolm
```

A proper p4a recipe is on the roadmap.

## Building from source

```bash
git clone https://github.com/aura-way/pylolm && cd pylolm
pip install -e ".[dev]"
python -m pytest                                   # pure-Python tests, no model needed
PYLOLM_MODEL=/path/model.gguf python -m pytest     # + integration tests
```

Native libraries: `src/pylolm/native/build_termux.sh` (on a phone) or `build_android_ndk.sh`
(cross-compile). The llama.cpp version is pinned in `src/pylolm/native/LLAMA_CPP_REF`.

## Releasing to PyPI

1. Set `LLAMA_CPP_REF` to a llama.cpp tag/commit you have tested.
2. Bump `version` in `pyproject.toml` and add a `CHANGELOG.md` entry.
3. On PyPI, add a [trusted publisher](https://docs.pypi.org/trusted-publishers/) for this repo
   (workflow `release.yml`, environment `pypi`).
4. `git tag v0.1.0 && git push --tags`. CI builds the sdist, a pure wheel, and the Android arm64
   wheel, publishes them to PyPI, and attaches `pylolm-libs-android-arm64-v8a.tar.gz` (+ `.sha256`)
   to the GitHub Release for `pylolm get-libs`. The tag must match the version in `pyproject.toml`.

## Roadmap

- p4a recipe for Kivy; verified Flet build
- whisper.cpp (speech-to-text) module
- TTS and OCR modules
- GPU backends (Vulkan/OpenCL) as an opt-in

## License

MIT. llama.cpp is MIT-licensed too; model licenses are your responsibility.
