Metadata-Version: 2.4
Name: boole
Version: 0.0.1
Summary: Local-first LLM inference SDK with optional remote burst scaling
Project-URL: Homepage, https://github.com/boole-ai/boole-py
Project-URL: Repository, https://github.com/boole-ai/boole-py
Project-URL: Documentation, https://github.com/boole-ai/boole-py#readme
Project-URL: Issues, https://github.com/boole-ai/boole-py/issues
Author-email: Boole AI <hello@boole.ai>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: huggingface-hub>=0.20.0
Requires-Dist: llama-cpp-python>=0.2.0
Requires-Dist: platformdirs>=4.0.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.3.0; extra == 'dev'
Requires-Dist: twine>=5.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# Boole

**Local-first LLM inference for Python.** Run GGUF models on your own hardware via llama.cpp — get cloud-SDK ergonomics without the cloud bill.

```bash
pip install boole
```

---

## Why Boole

Most inference SDKs assume every call leaves your machine. You pay per token, per second of
GPU time, per cold start — even for workloads your own laptop or workstation could handle in
milliseconds. Boole flips the default: **inference runs locally unless you tell it not to.**

- **~10x cheaper by default** — no metered API calls for work your hardware can already do.
- **No cold starts** — models load once into a long-lived local process, not a fresh
  container on every request.
- **No data leaves your machine** — prompts, context, and outputs stay local unless you
  explicitly opt into remote burst.
- **Familiar shape** — `App`, `Function`, and `Sandbox` primitives will feel immediately
  natural if you've used a serverless inference SDK before.
- **Burst when you need to** — for models too large for local hardware, or workloads that
  need to scale past one machine, the same function can transparently hand off to remote
  compute (opt-in, planned).

## Quickstart

```python
from boole import App

app = App(name="my-app")

generate = app.function(
    {"model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF", "quant": "Q4_K_M"},
    lambda ctx, prompt: ctx.llm.generate(prompt),
)

result = generate.call("Write a haiku about GPUs")
print(result)
```

The first call downloads and caches the GGUF weights to `~/.boole/models`; every call
after that loads from disk and runs entirely on your machine.

You can also use `Function` as a decorator:

```python
@app.function({"model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF", "quant": "Q4_K_M"})
def generate(ctx, prompt: str) -> str:
    return ctx.llm.generate(prompt)
```

## Core concepts

| Primitive                | What it does                                                                                               |
| ------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `App`                    | Top-level container that groups functions and shared config.                                               |
| `Function`               | A typed, callable unit of inference work, bound to a specific model.                                       |
| `Sandbox`                | An isolated local execution context for running arbitrary code with resource limits (timeout, memory cap). |
| `Client`                 | SDK entry point — model cache directory, default backend, auth for future remote mode.                     |
| `RemoteBurst` *(planned)* | Routes a `Function` call to remote compute when local hardware can't handle it.                            |

### Streaming generation

```python
def handler(ctx, prompt: str):
    return ctx.llm.stream(prompt)


stream_generate = app.function({"model": "...", "quant": "Q4_K_M"}, handler)

for token in stream_generate.stream("Write a haiku about GPUs"):
    print(token, end="", flush=True)
```

### Running untrusted code in a Sandbox

```python
sandbox = app.sandbox(timeout_ms=5000, memory_limit_mb=512)
result = sandbox.exec("python3", ["-c", "print(1 + 1)"])
print(result.stdout)
```

### Async usage

`Function.call()` is synchronous by default. If you're already inside an event loop
(e.g. a FastAPI handler), use `acall()` instead:

```python
result = await generate.acall("Write a haiku about GPUs")
```

## Platform support

Boole uses [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python) bindings to talk
to llama.cpp directly, with GPU offload where available.

| Platform              | CPU | GPU acceleration |
| ---------------------- | --- | ----------------- |
| macOS (Apple Silicon)  | ✅   | ✅ Metal           |
| macOS (Intel)          | ✅   | —                  |
| Linux (x64/arm64)      | ✅   | ✅ CUDA / Vulkan   |
| Windows (x64)          | ✅   | ✅ CUDA / Vulkan   |

`llama-cpp-python` publishes prebuilt wheels for common platform/CUDA combinations; unsupported
combinations fall back to compiling llama.cpp from source on install (a C++ compiler is required
in that case).

## Configuration

```python
from boole import Client

client = Client(
    model_cache_dir="~/.boole/models",  # where GGUF files are stored
    default_backend="llama-cpp",  # inference backend
)

app = App(name="my-app", client=client)
```

## Roadmap

- [x] Local inference via llama.cpp / GGUF (`llama-cpp-python`)
- [x] `App` / `Function` / `Sandbox` primitives
- [ ] `RemoteBurst` — opt-in remote fallback for oversized models / scaled workloads
- [ ] Structured output / grammar-constrained generation helpers
- [ ] Async-native backend (non-blocking generation without a thread pool)

## Contributing

Issues and PRs welcome. See [CONTRIBUTING.md](./CONTRIBUTING.md) for local dev setup.

```bash
pip install -e ".[dev]"
pytest
ruff check .
```