Metadata-Version: 2.4
Name: oragon
Version: 0.1.0
Summary: Local-first LLM serving for software that has to work: a resident engine, typed errors, validated extraction and an OpenAI-compatible API — on hardware you don't control, with data that can't leave.
License: Proprietary
License-File: LICENSE
Keywords: llm,local,inference,offline,openai-compatible,rag,on-premise
Author: Oragon Labs
Author-email: hello@oragonlabs.com
Requires-Python: >=3.9,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Provides-Extra: gpu
Provides-Extra: langchain
Requires-Dist: httpx (>=0.27,<1.0)
Requires-Dist: langchain-core (>=0.2,<0.3) ; extra == "langchain"
Requires-Dist: pydantic-settings (>=2.0,<3.0)
Requires-Dist: pynacl (>=1.5) ; extra == "gpu"
Requires-Dist: requests (>=2.31,<3.0) ; extra == "langchain"
Requires-Dist: vllm (>=0.8) ; (sys_platform == "linux" and python_version >= "3.10") and (extra == "gpu")
Project-URL: Homepage, https://www.oragonlabs.com
Description-Content-Type: text/markdown

# Oragon

**Run AI models inside your own software** — on a laptop, your own server or
your own datacenter, CPU or GPU. Your data never leaves the machine.

```sh
pip install oragon
```

That is the whole install. The engine is inside the package: no compiler, no
separate download, no signup, no API key.

```python
from oragon import Oragon, Engine, LocalModel, user

client = Oragon(engine=Engine.EMBEDDED, model=LocalModel.QWEN2_5_0_5B)

reply = client.chat.generate(
    model=LocalModel.QWEN2_5_0_5B,
    messages=[user("Name one primary color.")],
)
print(reply.choices[0].message.content)
```

The model loads once and stays ready. Requests run side by side — four at
once on the free tier, and as many as your machine holds on a paid plan.

## Why Oragon

- **It tells you when it doesn't know.** You get valid data or a clear error,
  the exact label you asked for or "no match", a cited answer or an honest
  refusal. Never a guess dressed up as an answer.
- **It fails like production software.** Every error is a typed exception that
  says whether it is safe to retry. Memory is checked before a model loads.
- **It behaves like a server.** Requests are batched, shared prompts are
  cached, and when it is full it says "busy, try again" instead of hanging.
- **It runs anywhere.** The same code works on a laptop or a server.
- **Your data stays yours.** No telemetry, ever.

## Quick start

```sh
oragon try            # download a small model and chat with it
oragon try extract    # turn text into validated JSON
oragon try rag        # ask questions about your documents
oragon serve          # run an OpenAI-compatible API on this machine
oragon doctor         # check what this machine can run
```

The first run downloads the model and checks it. Set
`ORAGON_AUTO_DOWNLOAD=0` to stop the SDK from ever going online.

## What you can do

Everything in this section is free.

### Get structured data

Pass a schema. You get data that matches it, or an `ExtractionError` —
nothing in between.

```python
rows = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user(invoice_text)],
    response_schema={"type": "array", "items": {
        "type": "object",
        "required": ["date", "amount"],
        "properties": {"date": {"type": "string"}, "amount": {"type": "number"}},
    }},
)
```

### Make a decision

You get back one of the labels you offered, or `matched=False`.

```python
decision = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user("Is 'J. Smith' the same person as 'John Smith'?")],
    response_format={"type": "choice", "choices": ["SAME", "DIFFERENT"]},
)
print(decision.choice, decision.matched)
```

For harder calls, `Settings(thinking=True)` has the model check its own
answer first.

### Use tools

Give it plain Python functions and `act()` runs the whole loop for you.

```python
def get_weather(city: str) -> dict:
    """Current weather for a city."""
    return {"city": city, "temp_c": 22}

reply = client.chat.act(
    model=LocalModel.QWEN2_5_1_5B,
    messages="What's the weather in Lagos?",
    tools=[get_weather],
)
print(reply.choices[0].message.content)
```

- If a tool fails, the model sees the error and tries again.
- Pass `messages` as a list and the conversation is kept, so you can answer a
  follow-up question and carry on.
- Several tool calls in one round run in parallel.
- `on_tool` and `on_round` let you watch what it does.

Prefer to run the loop yourself? `generate(tools=...)` uses the same format as
the OpenAI client.

### Ask questions about your documents

```python
from oragon.rag import DocumentQA, EngineRetriever, ingest

retriever = EngineRetriever()
ingest(["handbook.md", "policy.pdf"], retriever)

qa = DocumentQA(client, model=LocalModel.QWEN2_5_1_5B, retriever=retriever)
result = qa.ask("How many vacation days do new hires get?")

if result.refused:
    print("The documents don't say.")
else:
    print(result.text)               # with citations like [1]
    for hit in result.sources:       # the exact place in each file
        print(hit.chunk.source, hit.chunk.start, hit.chunk.end)
```

It searches by words and by meaning, entirely on your machine. If nothing
relevant is found, it refuses instead of guessing.

### Stream and run async

```python
for chunk in client.chat.generate(model=..., messages=[...], stream=True):
    if isinstance(chunk, str):
        print(chunk, end="", flush=True)
```

`AsyncOragon` has the same API with `await`, and parallel requests are
batched together.

### Use your own model

```sh
oragon import ./my-finetune/       # a training checkpoint folder
oragon import my-model.gguf        # or a model file
```

It is converted and optimized for you. If it cannot be converted correctly,
it is refused with the reason.

### Serve it over HTTP

```sh
oragon serve --auth-token "$TOKEN"
```

Any OpenAI client, in any language, can now use your model:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=TOKEN)
```

### Handle errors

All errors share a base class, `ApiError`, and say whether a retry is safe.

```python
from oragon import (
    ApiError, ContextOverflowError, EngineBusyError, EngineDiedError,
    EngineTimeoutError, ExtractionError, InsufficientMemoryError,
)
```

- `EngineBusyError` — nothing was started; retry.
- `InsufficientMemoryError` — the model will not fit; caught before loading.
- `ContextOverflowError` — the conversation is too long. Pass
  `overflow="slide"` to drop the oldest turns instead.

### More

- **Speculative decoding** — `draft_model=` lets a small model speed up a
  large one, up to 1.56× on a 3B model. It switches itself off when it would
  not help.
- **Conversations that survive restarts** — pass `session="..."`.
- **Reproducible answers** — `Oragon(reproducible=True)`.
- **Audit export** — `oragon audit export --format jsonl`.
- **Local usage stats** — `oragon stats`. Never uploaded.
- **LangChain** — `pip install 'oragon[langchain]'`.

## Servers and GPUs

The same engine runs on a server or in a datacenter.

```sh
pip install "oragon[gpu]"          # Linux, Python 3.10+, an NVIDIA driver
oragon serve Qwen/Qwen2.5-1.5B-Instruct --gpu auto --expose
```

- It tunes itself to the GPU it finds.
- If the GPU fills up, it admits fewer requests instead of slowing down.
- If the model server crashes, it restarts it.
- Running several machines? Each response carries an `X-Oragon-Affinity`
  header — route on it, and similar requests land on the machine that has
  them cached.

## Benchmarks

Measured, not estimated. CPU figures come from a 16 GB MacBook (Apple M4)
running `qwen2.5-1.5b` — the smallest machine Oragon is built for. Servers do
better.

### Oragon on its own

| Test | Result |
|---|---|
| 16 requests at once, CPU only | each gets 12+ tokens/s, first word in 1 s |
| 16 requests at once, one Tesla T4 GPU | 224 tokens/s total, slowest request still 59.9 tokens/s |
| 6,000 requests at the same instant | every one answered or told "busy" — none left hanging |
| 600,000 requests in three floods, never restarted | 0 lost, 0 unexplained errors, memory flat, health check in 0.02 ms |
| First request with a long system prompt | 10 s → 0.5 s when warmed at startup |
| 8 agents sharing one prompt | 11 s → 2.2 s with caching |
| Document search over 10,000 passages | 7 ms per search; 150 of 150 citations pointed at the exact text |
| Same model, with Oragon's checks around it | 7/18 → 11/18 correct on graded tasks |
| Invoice extraction on documents that aren't invoices | 5/7 handled correctly, against 2/7 for the model alone |
| Startup, small model | first answer 0.8 s after launch |

### Oragon and Ollama, side by side

Same model file, same machine, one running at a time (Ollama 0.33.3).

| Test | Oragon | Ollama |
|---|---|---|
| 1,000 different requests, CPU, both at 16 parallel | **done in 810 s** | 1,244 s — **Oragon 1.54× faster** |
| 6,000 requests at the same instant, one CPU | **none left hanging**, 17 answers/s | 4,757 got no answer, 8 answers/s |
| 16 requests at once, one Tesla T4 GPU | **224 tokens/s**, slowest 60 tokens/s | 140 tokens/s, slowest 25 tokens/s |
| First read of a long instruction, CPU | **10.9 s** | 23.8 s — **2.2× faster** |
| 16 requests at once, CPU, default settings | **9.3 s** | 20.1 s |
| 16 requests at once, Apple M4, default settings | **10.9 s** | 15.7 s |
| Startup to first word | **0.96 s** | 1.61 s |
| Invoice extraction, correct outcome | **5/7** | 2/7 |
| One request, generation speed | 92 tokens/s | 94 tokens/s — a tie |
| Repeated long prompt (caching) | 0.17 s | 0.18 s — a tie |
| Accuracy on the same model | 14/36 | 14/36 — a tie |
| One request on the T4, time to first word | 228 ms | **24 ms** — Ollama ahead |

Ollama answers one request at a time by default; with its parallel setting
raised it matches us on the Apple M4. The 1,000-request, 6,000-request, T4 and
long-instruction results hold with Ollama tuned. Ollama spent about 14% more
work per request on the 1,000-request run, so not all of that gap is engine.

Speed per request is set by your hardware, and answer quality by the model —
what Oragon adds is everything around it.

## Platforms

Python 3.9 to 3.13.

| Platform | Supported | Speeds up with |
|---|---|---|
| macOS 11+, Apple silicon | ✅ | Apple GPU |
| Linux x86-64 (Ubuntu 24.04+, Debian 13+, Fedora 40+) | ✅ | NVIDIA, AMD/Intel (Vulkan), or CPU |
| Linux arm64 (same versions) | ✅ | CPU |
| RHEL 9, Ubuntu 22.04, Debian 12 | coming soon | |
| Windows, Intel Macs | coming soon | |

No GPU needed. If one is there, it is used automatically. On an unsupported
platform, pip installs the older `0.0.1` placeholder instead.

## Pricing

Everything above is free — no signup, no limits on tokens.

| | Free | Paid |
|---|---|---|
| One process, 4 requests at once | ✅ | ✅ |
| Every feature on this page | ✅ | ✅ |
| Many processes sharing one model (`Engine.SHARED`) | — | ✅ |
| More than 4 requests at once | — | ✅ |
| Oragon's own licensed models | — | ✅ |

If four at a time in one process is enough, you never need to pay.

## Your data

- Your prompts and answers never leave the machine.
- No telemetry, no crash reports, no update checks.
- The only network calls: downloading a model you asked for, and — only on a
  machine with an activated licence — a check-in that sends usage counts,
  never content. No licence, no check-in.

## Not yet supported

- Images, audio and video — text only for now.
- The `hosted` engine.

## Learn more

- Documentation: <https://www.oragonlabs.com>
- Contact: [hello@oragonlabs.com](mailto:hello@oragonlabs.com)

Already have an Oragon account? The same package is on your private index, so
you can install it from your dashboard.

