Metadata-Version: 2.4
Name: oragon
Version: 0.1.1
Summary: Local-first LLM serving for software that has to work: a resident engine, typed errors, validated extraction and an OpenAI-compatible API — on hardware you don't control, with data that can't leave.
License: Proprietary
License-File: LICENSE
Keywords: llm,local,inference,offline,openai-compatible,rag,on-premise
Author: Oragon Labs
Author-email: hello@oragonlabs.com
Requires-Python: >=3.9,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Provides-Extra: gpu
Provides-Extra: langchain
Requires-Dist: httpx (>=0.27,<1.0)
Requires-Dist: langchain-core (>=0.2,<0.3) ; extra == "langchain"
Requires-Dist: pydantic-settings (>=2.0,<3.0)
Requires-Dist: pynacl (>=1.5) ; extra == "gpu"
Requires-Dist: requests (>=2.31,<3.0) ; extra == "langchain"
Requires-Dist: vllm (>=0.8) ; (sys_platform == "linux" and python_version >= "3.10") and (extra == "gpu")
Project-URL: Homepage, https://www.oragonlabs.com
Description-Content-Type: text/markdown

# Oragon

**Run AI models inside your own software** — on a laptop, your own server or
your own datacenter, CPU or GPU. Your data never leaves the machine.

```sh
pip install oragon
```

That is the whole install. The engine is inside the package: no compiler, no
separate download, no signup, no API key.

```python
from oragon import Oragon, Engine, LocalModel, Settings, user

client = Oragon(engine=Engine.EMBEDDED, model=LocalModel.QWEN2_5_0_5B)

reply = client.chat.generate(
    model=LocalModel.QWEN2_5_0_5B,
    messages=[user("Name one primary color.")],
)
print(reply.choices[0].message.content)
```

The model loads once and stays ready, so the second request does not pay for
the first one's startup. Requests run side by side: up to four at once for free,
and more than four on a paid plan. When more arrive than can run, they wait
in a queue instead of failing.
[How that works](#many-requests-at-once).

## Why Oragon

- **It tells you when it doesn't know.** You get valid data or a clear error,
  the exact label you asked for or "no match", a cited answer or an honest
  refusal. Never a guess dressed up as an answer.
- **It fails like production software.** Every error is a typed exception that
  says whether it is safe to retry. Memory is checked before a model loads,
  and context length before a word is generated.
- **It handles many requests at once.** Requests are batched together, shared
  prompts are cached, extra requests wait in a bounded queue, and a request
  turned away is retried for you.
- **It runs anywhere.** The same code works on a laptop, a server or a GPU.
  Moving between them is a setting, not a rewrite.
- **Your data stays yours.** No telemetry, ever. The full list of what touches
  the network is [below](#your-data).

## Quick start

```sh
oragon try                # download a small model and chat with it
oragon try --tunnel       # the same chat at a public HTTPS link — open it on your phone
oragon try extract        # turn text into validated JSON
oragon try rag            # ask questions about your documents
oragon serve              # run an OpenAI-compatible API, with a browser playground
oragon doctor             # check what this machine can run
```

The chat runs against a live OpenAI-compatible API the whole time, so any
OpenAI client in any language can join in while it is up.

The first run downloads the model. Disk space is checked first, progress is
shown, and the file is verified. Every later run finds it on disk. To fetch
ahead of time (CI images, air-gapped machines), run
`oragon download qwen2.5-0.5b`. Set `ORAGON_AUTO_DOWNLOAD=0` to stop the SDK
from ever going online; a missing model then becomes a clear error that names
the download command.

If anything goes wrong, `oragon doctor` reports the platform, memory, engine,
models and settings in one paste. `ORAGON_DEBUG=1` turns on detailed logging.

## The browser playground

`oragon serve` also serves a playground page from the engine itself. Open
`http://127.0.0.1:8080/` in a browser and you get:

- **a chat pane** to talk to the model,
- **a structured-output demo** that turns text into validated JSON,
- **a live console** showing what the engine is doing right now: tokens per
  second, lanes in use, requests waiting, requests answered, time to first
  word, how much of each prompt came from cache, and how full the cache and
  GPU memory are.

The page uses nothing from the internet, so it works the same on an
air-gapped machine. When the server needs a token, the link `oragon serve`
prints already carries it. With `--tunnel`, it also prints a public playground
link you can send to someone.

## Many requests at once

Oragon is built to serve traffic, not one chat window. This is what happens
when requests pile up, in order:

1. **Several run at the same time.** The engine has *lanes*, and each lane
   generates one request at a time. **Up to 4 lanes are free**, in every
   install. **More than 4 needs a paid plan.** Requests in different lanes are
   batched into the same passes over the model, so four requests cost far
   less than four times one. A long prompt is read in chunks, so it cannot
   freeze the others.
2. **Shared prompts are computed once.** Requests that start with the same
   text, such as a shared system prompt, share its cache across lanes. In one
   test, eight agents sharing an instruction went from 11 s to 2.2 s.
3. **The rest wait in a queue.** When every lane is busy, a new request waits
   and starts as soon as a lane frees up. Nothing is dropped, and nothing is
   cut off partway through an answer.
4. **The queue has limits, so it cannot sink the machine.** By default it
   holds 256 requests per lane, capped by a memory budget sized from free RAM.
   It also has a time limit: a request that would wait more than 60 seconds is
   turned away when it arrives, instead of being answered a minute late.
5. **A request that is turned away is never half-done.** It gets an immediate
   `overloaded` answer and nothing was generated, so sending it again is safe.
6. **The SDK retries it for you.** A turned-away request is retried
   automatically, up to three times, with short randomized pauses (about two
   seconds in total) so a thousand callers do not all retry at once. Streams
   are retried too: a refusal always comes before the first word, so a retry
   never repeats output. You only see `EngineBusyError` when the overload
   lasts. Before that, an `EngineBusyWarning` on requests that *succeeded*
   tells you pressure is building while you can still act on it.
7. **Over HTTP, it is a `429`.** The API returns the status OpenAI clients
   already understand, so the official OpenAI libraries retry it on their
   own. A server that is shutting down returns `503`, finishes what is
   running, and then exits.

On a GPU, the same rules cover memory. New work is turned away before the card
fills, while running requests finish, so there is no out-of-memory crash to
recover from. A request whose prompt and answer would not fit the cache waits
for room instead of being started and pushed out halfway.

You can change all of it. On the free tier, `slots` goes up to 4: asking
for more runs with 4 and says so. It is not an error, and nothing else
changes. A paid plan raises the limit.

```python
client = Oragon(
    engine=Engine.EMBEDDED,
    model=LocalModel.QWEN2_5_1_5B,
    slots=4,             # lanes: 1-4 free; more than 4 needs a paid plan
    max_queue=512,       # how many may wait for a lane
    max_queue_wait=10,   # seconds a caller will wait; 0 = always wait, never turn away
)

client.stats()   # lanes busy, requests waiting, queue limits, cache reuse
client.ping()    # health check; answers between words, even when busy
```

`AsyncOragon` has the same behaviour: `asyncio.gather` over many calls
really does overlap, and the engine batches them. Over HTTP, the same
figures are at `/metrics` (Prometheus) and `/v1/status` (JSON).

Measured on a 16 GB laptop with no GPU: 16 requests at once, each streaming
at 12+ tokens/s with its first word in about a second. In a flood of 6,000
requests arriving at the same instant, every one was either answered or told
`overloaded` right away. None were left hanging.

## What you can do

Everything in this section is free.

### Three engines, one API

```python
Oragon(engine=Engine.EMBEDDED, model=...)   # the model runs in a process this one owns
Oragon(engine=Engine.SHARED,   model=...)   # one copy of the model per machine (paid)
Oragon(engine=Engine.HOSTED,   model=...)   # the Oragon API (not available yet)
```

All three have the same `generate()`, the same response type and the same
errors, so where the model runs is a deployment choice, not a rewrite.

### Get structured data

Pass a schema. You get data that matches it, or an `ExtractionError` —
nothing in between.

```python
rows = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user(invoice_text)],
    response_schema={"type": "array", "items": {
        "type": "object",
        "required": ["date", "amount"],
        "properties": {"date": {"type": "string"}, "amount": {"type": "number"}},
    }},
)
```

The schema is enforced while the model writes, so output that would break it
cannot be produced.

### Make a decision

You get back one of the labels you offered, or `matched=False`. It is never
stretched to fit the closest-looking label.

```python
decision = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user("Is 'J. Smith' the same person as 'John Smith'?")],
    response_format={"type": "choice", "choices": ["SAME", "DIFFERENT"]},
)
print(decision.choice, decision.matched)
```

For harder calls, `Settings(thinking=True)` shows the model its own answer and
asks it to check. Its first answer is the floor, so this cannot do worse than
leaving it off:

```python
decision = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user("Meeting started 9:45, ran 100 min — over before noon?")],
    response_format={"type": "choice", "choices": ["YES", "NO"]},
    settings=Settings(temperature=0.0, thinking=True),
)
print(decision.reason)   # the reasoning, for your audit log
print(decision.choice)   # the verdict, for your code
```

### Use tools

Give it plain Python functions and `act()` runs the whole loop for you. The
tool's description is read from its signature, type hints and docstring.

```python
def get_weather(city: str) -> dict:
    """Current weather for a city."""
    return {"city": city, "temp_c": 22}

reply = client.chat.act(
    model=LocalModel.QWEN2_5_1_5B,
    messages="What's the weather in Lagos?",
    tools=[get_weather],
)
print(reply.choices[0].message.content)
```

- If a tool fails, the model sees the error and tries again.
- Pass `messages` as a list and the conversation is kept, so you can answer a
  follow-up question and carry on.
- Several tool calls in one round run in parallel.
- `on_tool` and `on_round` let you watch what it does.

Prefer to run the loop yourself? `generate(tools=...)` uses the same format as
the OpenAI client.

### Ask questions about your documents

```python
from oragon.rag import DocumentQA, EngineRetriever, ingest

retriever = EngineRetriever()
ingest(["handbook.md", "policy.pdf"], retriever)

qa = DocumentQA(client, model=LocalModel.QWEN2_5_1_5B, retriever=retriever)
result = qa.ask("How many vacation days do new hires get?")

if result.refused:
    print("The documents don't say.")
else:
    print(result.text)               # with citations like [1]
    for hit in result.sources:       # the exact place in each file
        print(hit.chunk.source, hit.chunk.start, hit.chunk.end)
```

It searches by words and by meaning, entirely on your machine. A small
embedding model runs inside the same engine, so there is no second service to
run. If nothing relevant is found, it refuses instead of guessing, and a
citation the model made up is flagged rather than quietly fixed.

### Stream and run async

```python
for chunk in client.chat.generate(model=..., messages=[...], stream=True):
    if isinstance(chunk, str):
        print(chunk, end="", flush=True)
```

```python
from oragon import AsyncOragon

client = AsyncOragon(engine=Engine.EMBEDDED, model=LocalModel.QWEN2_5_1_5B)
reply = await client.chat.generate(model=..., messages=[...])
```

### Long conversations

A conversation too long for the model raises `ContextOverflowError` before
anything is generated. It is never silently cut. Pass `overflow="slide"` to
drop the oldest turns instead (the system prompt is kept):

```python
client.chat.generate(model=..., messages=history, overflow="slide")
```

### Fast without tuning

- **Tuned to your machine.** On first run the engine measures this machine
  and sizes its own work to match. The measurement is saved, so it happens
  once.
- **Prompt reuse.** A shared system prompt or a growing conversation is read
  from cache rather than recomputed. Watch `reply.usage.cached_tokens` climb.
- **Warm start.** `Oragon(warm_prompt=SYSTEM_PROMPT)` reads a long standing
  instruction at startup, so even the first request is fast: 10 s down to
  0.5 s in our test.
- **Conversations that survive a restart.** Pass `session="..."` and a
  conversation's computed state is kept on disk. The next request only
  processes what is new. The answer is the same with or without it.
- **Repeated questions answered instantly.** `oragon serve --cache auto` (or
  `Oragon(answer_cache="auto")`) answers a request it has already answered,
  word for word, without generating it again: under a millisecond instead of
  seconds. Only exact matches are served. `--cache measure` reports how much
  it would save on your traffic before you turn it on. Off by default.
- **Speculative decoding.** `draft_model=` lets a small model speed up a
  large one, up to 1.56× on a 3B model. It checks whether this helps on your
  machine and switches itself off when it would not.

`oragon doctor` tells you what this machine will do before you choose a model:
its measured memory speed, the generation speed to expect from each model, and
how much context each one has room for.

### Use your own model

```sh
oragon import ./my-finetune/       # a training checkpoint folder
oragon import my-model.gguf        # or a model file
```

It is converted and optimized for you, and your original file is never
changed. If it cannot be converted correctly, it is refused with the reason.

### Serve it over HTTP

```sh
oragon serve                               # this machine only: 127.0.0.1:8080
oragon serve --expose                      # other machines on your network
oragon serve --tunnel                      # anyone: a public HTTPS link
```

`--expose` and `--tunnel` create an access token for the run unless you pass
`--auth-token`. Any OpenAI client, in any language, can now use your model:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=TOKEN)
client.chat.completions.create(model="qwen2.5-1.5b", messages=[...])
```

| Route | What it is |
|---|---|
| `/` | the [browser playground](#the-browser-playground) |
| `/v1/chat/completions` | chat, with streaming, tools and `response_format` |
| `/v1/embeddings` | vectors, for a vector database you already run |
| `/v1/models` | what is loaded |
| `/health` | liveness |
| `/v1/status`, `/metrics` | load, queue and cache figures (behind the token, when one is set) |

When the engine cannot do something, it says so instead of improvising.
Asking for a model it has not loaded is a `404`, not a quiet swap. Binding to
a network address without a token is refused at startup.

### Handle errors

All errors share a base class, `ApiError`, and say whether a retry is safe.

```python
from oragon import (
    ApiError, ContextOverflowError, EngineBusyError, EngineDiedError,
    EngineTimeoutError, ExtractionError, InsufficientMemoryError,
)
```

| Error | What it means |
|---|---|
| `EngineBusyError` | Still overloaded after the automatic retries. Nothing was started; retry later |
| `InsufficientMemoryError` | The model will not fit. Caught before loading, not a crash |
| `ContextOverflowError` | The conversation is too long; see `overflow="slide"` |
| `ExtractionError` | The output still did not match your schema after one corrected retry |
| `EngineDiedError` | The engine stopped mid-request; the next call starts a new one |
| `EngineTimeoutError` | The request took longer than your timeout |

### See what your app did — locally

Every request is summarized into a store on this machine: timings, token
counts, models, error codes, tool names. Prompts and answers are never
stored, and nothing is ever uploaded.

```sh
oragon stats             # last 7 days: latency, error rates, tool calls
oragon stats --json
oragon obs clear         # delete the store; always safe
```

`ORAGON_OBSERVABILITY=0` turns recording off entirely.

### More

- **Reproducible answers** — `Oragon(reproducible=True)`: an answer never
  depends on what was asked before it.
- **Audit export** — `oragon audit export --format jsonl` (or `cef`) for your
  audit system.
- **See a model's chat template** — `oragon template qwen2.5-1.5b`.
- **LangChain** — `pip install 'oragon[langchain]'`, then
  `from oragon.llm import OragonChat`.

## Servers and GPUs

The same engine runs on a server or in a datacenter.

```sh
pip install "oragon[gpu]"          # Linux, Python 3.10+, an NVIDIA driver
oragon serve Qwen/Qwen2.5-1.5B-Instruct --gpu auto --expose
```

The engine starts a GPU model server, waits for it to load, and serves the
API in front of it. Typed errors, the queue, licensing and document search all
stay the same.

- **Tuned to the card.** `--gpu-profile auto` (the default) reads the card's
  memory and sets itself up. `small`, `balanced` and `throughput` choose
  explicitly.
- **It does not thrash.** If the GPU starts running out of room, the engine
  admits fewer requests until it recovers, instead of slowing everyone down.
- **It reserves before it starts.** A request is only started when its prompt
  and answer fit. Otherwise it waits for a running one to finish.
- **It restarts itself.** If the model server crashes, it is replaced, and
  queued requests wait for it.

**On CPU servers**, `--kv-pool auto` sizes one shared cache from the free
memory. Requests that share a prompt store it once, so more lanes fit in the
same memory, and requests that do not fit wait instead of failing. None of
this changes an answer.

**Running several machines?** Each response carries an `X-Oragon-Affinity`
header. Route on it, and requests with the same system prompt land on the
machine that already has it cached. Several machines then behave like one
deployment instead of several cold starts.

No GPU is required. CPU is a first-class path, not a fallback. If a GPU is
there it is used automatically; pass `cpu=True` to force CPU and compare.

## Benchmarks

Measured, not estimated. CPU figures come from a 16 GB MacBook (Apple M4)
running `qwen2.5-1.5b` — the smallest machine Oragon is built for. Servers do
better.

### Oragon on its own

| Test | Result |
|---|---|
| 16 requests at once, CPU only | each gets 12+ tokens/s, first word in 1 s |
| 16 requests at once, one Tesla T4 GPU | 224 tokens/s total, slowest request still 59.9 tokens/s |
| 6,000 requests at the same instant | every one answered or told "busy" — none left hanging |
| 600,000 requests in three floods, never restarted | 0 lost, 0 unexplained errors, memory flat, health check in 0.02 ms |
| First request with a long system prompt | 10 s → 0.5 s when warmed at startup |
| 8 agents sharing one prompt | 11 s → 2.2 s with caching |
| The same question asked again, answer cache on (0.5B model) | 0.5 ms instead of 1.3 s |
| Document search over 10,000 passages | 7 ms per search; 150 of 150 citations pointed at the exact text |
| Same model, with Oragon's checks around it | 7/18 → 11/18 correct on graded tasks |
| Invoice extraction on documents that aren't invoices | 5/7 handled correctly, against 2/7 for the model alone |
| Startup, small model | first answer 0.8 s after launch |

### Oragon and Ollama, side by side

Same model file, same machine, one running at a time (Ollama 0.33.3).

| Test | Oragon | Ollama |
|---|---|---|
| 1,000 different requests, CPU, both at 16 parallel | **done in 810 s** | 1,244 s — **Oragon 1.54× faster** |
| 6,000 requests at the same instant, one CPU | **none left hanging**, 17 answers/s | 4,757 got no answer, 8 answers/s |
| 16 requests at once, one Tesla T4 GPU | **224 tokens/s**, slowest 60 tokens/s | 140 tokens/s, slowest 25 tokens/s |
| First read of a long instruction, CPU | **10.9 s** | 23.8 s — **2.2× faster** |
| 16 requests at once, CPU, default settings | **9.3 s** | 20.1 s |
| 16 requests at once, Apple M4, default settings | **10.9 s** | 15.7 s |
| Startup to first word | **0.96 s** | 1.61 s |
| Invoice extraction, correct outcome | **5/7** | 2/7 |
| One request, generation speed | 92 tokens/s | 94 tokens/s — a tie |
| Repeated long prompt (caching) | 0.17 s | 0.18 s — a tie |
| Accuracy on the same model | 14/36 | 14/36 — a tie |
| One request on the T4, time to first word | 228 ms | **24 ms** — Ollama ahead |

Ollama answers one request at a time by default; with its parallel setting
raised it matches us on the Apple M4. The 1,000-request, 6,000-request, T4 and
long-instruction results hold with Ollama tuned. Ollama spent about 14% more
work per request on the 1,000-request run, so not all of that gap is engine.

Speed per request is set by your hardware, and answer quality by the model —
what Oragon adds is everything around it.

## How it compares

| | Oragon | Ollama |
|---|---|---|
| Install | `pip install oragon` | a separate installer, then a client library |
| Runs on | laptop or server, CPU or GPU | laptop or server, CPU or GPU |
| Many requests at once | up to 4 lanes free (more on a paid plan), a bounded queue, automatic retry | one at a time by default |
| When it is full | answers `429` at once; the SDK retries for you | requests can wait with no answer |
| Asking for JSON | valid data, or a clear error | text you check yourself |
| Asking for a category | the exact label, or an honest "no match" | text you have to interpret |
| Questions over your documents | included, with citations, and it says when it doesn't know | build it yourself |
| When something breaks | typed errors that say whether to retry | a status code and a message |
| Model too big for the machine | you are told before it loads | it fails partway through loading |
| Your data | stays on the machine | stays on the machine |

Ollama is a good way to try models. Oragon is built for the next step:
software you hand to someone else, running on a machine you do not control,
where an odd answer from the model has to be something your code can catch.

## Platforms

Python 3.9 to 3.13.

| Platform | Supported | Speeds up with |
|---|---|---|
| macOS 11+, Apple silicon | ✅ | Apple GPU |
| Linux x86-64 (Ubuntu 24.04+, Debian 13+, Fedora 40+) | ✅ | NVIDIA, AMD or Intel GPU (via Vulkan), or CPU |
| Linux arm64 (same versions) | ✅ | CPU |
| RHEL 9, Ubuntu 22.04, Debian 12 | coming soon | |
| Windows, Intel Macs | coming soon | |

Every feature works the same on every platform; only the speed-up differs.
No GPU needed. If one is there, it is used automatically. One install covers
every machine: there is no separate GPU package to choose. On an unsupported
platform, pip installs the older `0.0.1` placeholder instead — if you see
version 0.0.1, that is why.

## Pricing

Everything above is free — no signup, no limits on tokens, and commercial use
is allowed within the free tier.

| | Free | Paid |
|---|---|---|
| Up to 4 lanes (requests generating at once), plus the queue | ✅ | ✅ |
| Every feature on this page, including the HTTP API and GPU serving | ✅ | ✅ |
| Many processes sharing one model (`Engine.SHARED`) | — | ✅ |
| More than 4 lanes | — | ✅ |
| Oragon's own licensed models, encrypted on disk | — | ✅ |

If four at a time in one process is enough, you never need to pay. That is
intended, not a loophole: you can ship a desktop app to someone who never
signs up, never connects, and never tells you their name.

## Your data

The complete list of what crosses the network, so you can check it rather
than trust it:

- **Your prompts and answers** never leave the machine. There is no code path
  that could send them, and they are never used to train anyone's model.
- **No telemetry**, no crash reports, no update checks.
- **Model downloads** — only the model you asked for, only incoming.
  `ORAGON_AUTO_DOWNLOAD=0` turns even that off.
- **Licence check-in** — only on a machine with an activated licence. It sends
  the licence key, usage counts (token totals, model names, timings — never
  content) and a device ID. The device ID is a short hash, not the machine's
  own identifier, and `ORAGON_DEVICE_ID` lets you set it yourself. No licence,
  no check-in.

Licences work offline for their whole validity period, so air-gapped
deployments are fully supported.

## Not yet supported

- Images, audio and video — text only for now.
- The `hosted` engine. Asking for it raises an error rather than pretending.
  Code written today will move to it by changing one word.
- Some platforms — see [Platforms](#platforms).

## Who builds this

Oragon Labs — an AI research lab building AI that runs where people and their
data already are: on their own machines, in their own country, at a price
local economies can sustain. No GPU required, no data leaving your control,
no single company or server as a point of failure.

## Learn more

- Documentation: <https://www.oragonlabs.com>
- Contact: [hello@oragonlabs.com](mailto:hello@oragonlabs.com)

Already have an Oragon account? The same package is on your private index, so
you can install it from your dashboard.

