Metadata-Version: 2.5
Name: bastion-broker
Version: 0.6.0
Summary: A single-GPU safety broker for Ollama — serializes model loads and budgets VRAM to prevent crashes on a shared GPU. Not a serving engine.
Project-URL: Homepage, https://github.com/cypwin/bastion
Project-URL: Repository, https://github.com/cypwin/bastion
Project-URL: Documentation, https://github.com/cypwin/bastion/tree/main/docs
Project-URL: Changelog, https://github.com/cypwin/bastion/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/cypwin/bastion/issues
Author: Cyprian Winogradow
License-Expression: MIT
License-File: LICENSE
Keywords: broker,gpu,inference,ollama,proxy,scheduler,vram
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Hardware
Requires-Python: >=3.11
Requires-Dist: fastapi<1.0,>=0.115
Requires-Dist: httpx<1.0,>=0.27
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: uvicorn[standard]<1.0,>=0.30
Provides-Extra: a2a
Requires-Dist: a2a-sdk[http-server]<1.0,>=0.3; extra == 'a2a'
Provides-Extra: dashboard
Requires-Dist: httpx<1.0,>=0.27; extra == 'dashboard'
Requires-Dist: psutil<8.0,>=5.9; extra == 'dashboard'
Requires-Dist: textual<8.0,>=1.0; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: httpx<1.0,>=0.27; extra == 'dev'
Requires-Dist: hypothesis<7.0,>=6.0; extra == 'dev'
Requires-Dist: mypy<2.0,>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio<1.0,>=0.24; extra == 'dev'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'dev'
Requires-Dist: pytest<9.0,>=8.0; extra == 'dev'
Requires-Dist: respx<1.0,>=0.21; extra == 'dev'
Requires-Dist: ruff<1.0,>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml<7.0,>=6.0; extra == 'dev'
Provides-Extra: metrics
Requires-Dist: prometheus-client<1.0,>=0.20; extra == 'metrics'
Provides-Extra: persistence
Requires-Dist: aiosqlite<1.0,>=0.20; extra == 'persistence'
Description-Content-Type: text/markdown

# BASTION

![CI](https://github.com/cypwin/bastion/actions/workflows/ci.yml/badge.svg)
![PyPI](https://img.shields.io/pypi/v/bastion-broker)
![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)
![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-blue.svg)
![repo status: Active](https://www.repostatus.org/badges/latest/active.svg)

**A single-GPU safety broker for [Ollama](https://ollama.com) -- not a serving engine.**

BASTION sits as a transparent HTTP proxy in front of Ollama on the standard
port. It serializes model loads, enforces a VRAM budget, and queues requests by
model affinity so a single GPU shared by several clients does not thrash. On my
single RTX 5090 (32 GB) workstation this **eliminated the hard reboots I was
hitting under rapid model cycling** -- a single-machine (n=1) result whose
mechanism is *hypothesized*, not proven. See
[Crash Mitigation](docs/crash-prevention.md) for the full caveat.

> **Name disambiguation.** This is not Supergiant Games' *Bastion* (the 2011
> action RPG), not `bastion-rs` (the Rust actor framework), and not an SSH
> bastion host. It is a GPU inference broker. Installed from PyPI as
> **`bastion-broker`**; the command and module are both `bastion`.

Only **BASTION** is published. It is a personal project by a single developer,
MIT-licensed, run in the open. No SLA, no third-party audit -- see
[Support & status](#support--status).

---

## Scope and non-goals

**BASTION is** a scheduling and safety layer for **one Ollama instance on one
NVIDIA GPU**: request queueing, model-swap serialization, VRAM budgeting, a
crash-mitigation default, an admin/observability surface, and an optional A2A
agent interface.

**BASTION is not a serving engine.** It deliberately does not implement the
throughput primitives a production inference server has. If you need those,
BASTION is the wrong tool -- use one below and stop reading:

| You need | Use instead |
|----------|-------------|
| Continuous/dynamic **batching**, **PagedAttention** KV cache | [vLLM](https://github.com/vllm-project/vllm) |
| **Tensor / pipeline parallelism** across GPUs | vLLM, TGI |
| **Multi-node** GPU clusters, **autoscaling** | [GPUStack](https://github.com/gpustack/gpustack), Ray Serve |
| A hosted, multi-tenant inference API | a managed provider |

BASTION optimizes for *stability on a single shared GPU*, not tokens/sec.

## Where BASTION fits

| | raw Ollama | **BASTION** | vLLM | GPUStack |
|---|:---:|:---:|:---:|:---:|
| Drop-in on the Ollama port | -- | ✅ | -- | -- |
| Model-swap serialization + cooldown | -- | ✅ | n/a | -- |
| VRAM budget / overcommit guard | -- | ✅ | ✅ | ✅ |
| Affinity queue across many clients | -- | ✅ | ✅ (batching) | ✅ |
| High-throughput batching / PagedAttention | -- | -- | ✅ | ✅ |
| Multi-GPU / multi-node | -- | -- | ✅ | ✅ |
| Target | one model | **one shared GPU** | one big model, max throughput | a cluster |

The honest one-liner: BASTION is what you put in front of Ollama when several
agents share **one** GPU and you want it to stay up, not to go faster.

## Why I built it

I ran several Claude Code agents and Python pipelines against one Ollama
instance on an RTX 5090. Under rapid model cycling -- many clients requesting
different models in quick succession -- the machine would hard-reboot after a
few dozen swaps. Ollama has no queue, no VRAM budget, and no cooldown between
model transitions, so nothing upstream throttled the thrash.

I could reproduce the reboot reliably and stop it by (a) disabling
memory-mapped loading on scheduled requests, (b) serializing model swaps behind
a budget, and (c) rate-limiting swaps with a cooldown. That is what BASTION
does. **What I cannot do is prove the mechanism**: this is one machine, no
controlled A/B, and the documented Blackwell reboot causes (power delivery,
GSP/driver Xid faults) are exactly the kind of thing I had no instrument to
rule in or out. The [Crash Mitigation](docs/crash-prevention.md) doc states
this plainly, and an instrumented before/after benchmark is the planned way to
actually test it.

## Hardware

| Tier | Hardware | Behavior |
|------|----------|----------|
| **Tested (n=1)** | RTX 5090, 32 GB | Primary development box; every measurement here comes from it |
| **Minimum** | Any CUDA NVIDIA GPU where `nvidia-smi` responds, ~8 GB | Runs; expect heavy queueing and single-model use on small cards |
| **Recommended** | 24 GB+ NVIDIA (RTX 3090/4090, A6000, L40) | Comfortable multi-model co-residency |
| **Small cards** | < ~12 GB | Budget refuses co-loads that would exceed VRAM -- BASTION queues rather than OOMs |
| **CPU / no GPU** | no CUDA device | BASTION **starts and proxies**, but GPU safety features (crash mitigation, thermal gating) are inert -- it is a plain affinity queue |

Not supported: AMD/ROCm, Apple Silicon, Intel Arc (GPU monitoring is
nvidia-smi-specific). Full matrix and per-size config in the
[Hardware Guide](docs/hardware-guide.md).

## Quick Start

### 1. Install

```bash
pip install bastion-broker
```

Or from a source checkout, for development:

```bash
git clone https://github.com/cypwin/bastion.git
cd bastion
pip install -e ".[dev]"
```

> No real data ships or is required -- BASTION runs against your own local
> Ollama models. There is no bundled corpus and nothing phones home.

### 2. Move Ollama to port 11435

```bash
sudo mkdir -p /etc/systemd/system/ollama.service.d/
sudo tee /etc/systemd/system/ollama.service.d/override.conf > /dev/null << 'EOF'
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11435"
EOF
sudo systemctl daemon-reload
sudo systemctl restart ollama
```

Or manually: `OLLAMA_HOST=127.0.0.1:11435 ollama serve`

### 3. Configure and validate

```bash
bastion --init-config       # Generate ~/.config/bastion/broker.yaml
bastion --detect-models     # Discover installed models
bastion --validate          # Check the setup before starting
```

### 4. Start and verify

```bash
bastion                                    # bind loopback (default)
curl http://localhost:11434                # "Ollama is running"
curl http://localhost:11434/broker/status  # broker status
ollama run llama3.1:8b "Hello, world!"     # transparent proxy
```

## Demo

`bastion-dashboard` against a live broker on a single RTX 5090, under synthetic
load from [`scripts/demo/generate_load.py`](scripts/demo/generate_load.py) --
four simulated agents (an interactive CLI session, a research agent, a batch
ingest, a nightly reindex) contending for one card across three models. Every
still below is a real capture; nothing here is mocked up.

![BASTION dashboard, standard two-column layout](https://raw.githubusercontent.com/cypwin/bastion/main/docs/demo/dashboard-standard.png)

Three models are resident at 2.2 + 2.8 + 4.8 GB, and each matches the size
declared for it in config -- so the `Declared` column stays dim. A loud signed
figure there is the thing to look for: it means admission is approving loads
against a number the card will not honour.

The queue is 13 deep and stalled at `at_max_concurrent`; the thrashing detector
has `WARNED` two of the four agents on their swap ratio -- how often consecutive
requests from that caller ask for a different model.

The VRAM ledger keeps its two error terms apart rather than netting them out.
`Δ booking` compares what the ledger booked against what Ollama reports
resident; `Δ outside` compares the hardware total against that same figure. The
+3.6 GB outside is CUDA contexts, cuBLAS workspaces and the desktop compositor
-- bytes Ollama's `size_vram` never counts. Budgeting as if they were zero is
how a logical ledger talks itself past a physical limit, which is what
`gpu.non_ollama_reserve_gb` exists to prevent.

<details>
<summary>Other layouts -- <code>[1]</code> compact, <code>[3]</code> full, <code>[h]</code> help</summary>

<br>

`[3]` full -- adds the scheduler, watchdog, circuit breaker and a live request
trace with per-request source and priority tier:

![BASTION dashboard, full three-column layout](https://raw.githubusercontent.com/cypwin/bastion/main/docs/demo/dashboard-full.png)

`[1]` compact -- one column (temps, GPU, models, queue, ledger) for a narrow
pane:

![BASTION dashboard, compact one-column layout](https://raw.githubusercontent.com/cypwin/bastion/main/docs/demo/dashboard-compact.png)

`[h]` help -- the full key map, including fan control, GPU process kill,
preload/unload and drain:

![BASTION dashboard help overlay](https://raw.githubusercontent.com/cypwin/bastion/main/docs/demo/dashboard-help.png)

</details>

Reproduce the stills on your own broker with
[`scripts/demo/capture_screenshots.py`](scripts/demo/README.md) -- it refuses to
run when nothing answers on the proxy port, by design.

## Key Features

- **Transparent HTTP proxy** -- drop-in on port 11434; existing clients work
  unchanged. Streams NDJSON faithfully so `ollama run` stays responsive.
- **Affinity-based scheduling** -- per-model sub-queues drain all pending
  requests for the loaded model before swapping, cutting GPU model transitions.
- **Priority aging** -- four tiers (interactive, agent, pipeline, background)
  with time-based aging to prevent starvation.
- **VRAM budget enforcement** -- tracks GPU memory via Ollama `/api/ps` fused
  with `nvidia-smi`; blocks loads that would exceed the budget.
- **Crash mitigation** -- injects `use_mmap: false` into scheduled inference
  requests when the client hasn't set it, plus swap cooldown and rate limiting.
  Targets the rapid-swap pattern that can destabilize the GPU driver (single-config testing; mechanism hypothesized --
  see [Crash Mitigation](docs/crash-prevention.md)).
- **TUI dashboard** -- real-time Textual dashboard: GPU thermals, VRAM, queue
  depth, scheduler state, A2A tasks, leases, audit events.
- **A2A protocol support** -- agent card discovery, task lifecycle, batch
  inference, model reservation leases (optional extra).
- **Prometheus metrics + OpenTelemetry tracing** -- optional, with no-op
  fallbacks when the extras are absent.
- **Circuit breaker + per-IP rate limiting + tiered audit logging.**

## Architecture

```
                          Clients
    (ollama run, Claude Code, Python scripts, A2A agents, curl)
                             |
                             | :11434 (standard Ollama port)
                             v
    +------------------------------------------------------------+
    |                        BASTION                              |
    |  +----------------+  +-----------------+  +--------------+  |
    |  | Ollama Proxy   |  | Admin API       |  | A2A Agent    |  |
    |  | /api/*         |  | /broker/*       |  | /a2a/*       |  |
    |  | - use_mmap     |  | - status/queue  |  | - tasks      |  |
    |  |   injection    |  | - health/vram   |  | - streaming  |  |
    |  | - NDJSON       |  | - preload       |  | - leases     |  |
    |  |   streaming    |  | - unload/drain  |  | - agent card |  |
    |  | - priority     |  | - metrics       |  | - batch      |  |
    |  +-------+--------+  +-----------------+  +--------------+  |
    |          |                                                  |
    |  +-------v-------------------------------------------------+ |
    |  |          Affinity Queue + Scheduler                     | |
    |  |  - Per-model sub-queues (minimize GPU model swaps)      | |
    |  |  - Priority tiers: INTERACTIVE > AGENT > PIPELINE > BG  | |
    |  |  - Cooldown: 2s (escalates with swap rate)              | |
    |  |  - VRAM ledger (assume/confirm/forget pattern)          | |
    |  |  - GPU health gating (temp, power, utilization)         | |
    |  +-------+------------------------------------------------+ |
    +----------|--------------------------------------------------+
               | :11435
    +----------v--------------------------------------------------+
    |                     Ollama (backend)                        |
    |              OLLAMA_HOST=127.0.0.1:11435                     |
    +-------------------------------------------------------------+
```

The **VRAM ledger** is the load-bearing safety primitive: it reserves estimated
VRAM atomically before an async load (assume), marks it on success (confirm),
and returns it on failure or TTL expiry (forget), so concurrent loads can never
overcommit the budget.

## Dashboard

```bash
bastion-dashboard
bastion-dashboard --url http://localhost:11434 --interval 2.0
```

For a clickable app entry, `scripts/install-desktop.sh` (or the Desktop-icon
variant `scripts/install-desktop-icon.sh`) -- see
[Deployment → Desktop Launcher](docs/deployment.md#desktop-launcher).

**Keyboard shortcuts:** `p` preload, `u` unload, `d` drain toggle, `r` refresh,
`h` help, `q` quit.

## Security

BASTION is built for a **single machine or a trusted LAN**, not public internet
exposure. Under the default auth model (ADR-006):

- Binds to `127.0.0.1` (loopback) on 11434 by default.
- A non-loopback bind **refuses to start** unless `auth.enabled: true` --
  exposure is always a deliberate, auth-gated choice, never a default.
- An on-disk bearer token (constant-time compared) gates admin surfaces; a
  DNS-rebinding guard rejects foreign `Host` headers and cross-origin browser
  requests. The loopback exemption is a convenience for local CLI tools, **not**
  a security boundary.
- No TLS in-process -- terminate at a reverse proxy for remote access.

**mTLS and OIDC are deferred, not rejected** (ADR-006-B). They are revisited
when any of: (1) a second operator is added to the host, (2) the MCP adapter is
hosted on a non-loopback bind in production, or (3) the token model is shown
compromised in an incident. For the single-operator workstation target, mTLS is
provisioning friction that buys nothing.

Report vulnerabilities via [GitHub Private Security Advisory](https://github.com/cypwin/bastion/security/advisories/new)
-- see [SECURITY.md](SECURITY.md). Full model in the [Security Guide](docs/security.md).

## Known Limitations

- **n=1.** Every stability claim comes from one RTX 5090. No controlled A/B, no
  second machine, mechanism hypothesized. Treat swap-rate numbers as observed,
  not validated.
- **Not a serving engine.** No batching, PagedAttention, tensor/pipeline
  parallel, multi-node, or autoscaling. See [Scope and non-goals](#scope-and-non-goals).
- **NVIDIA/Linux only.** GPU safety features depend on `nvidia-smi`. AMD, Apple
  Silicon, Intel Arc, and CPU-only run degraded or unsupported.
- **Single developer.** No SLA, no third-party security audit, best-effort
  support. First stable release (`0.6.0`) -- breaking changes to the config
  schema and admin API will come with a major-version bump, but that policy is
  new and untested.
- **No published benchmark yet.** Throughput and the before/after stability
  A/B are planned, not shipped -- do not read performance claims into this repo.

## Documentation

| Guide | Description |
|-------|-------------|
| [Getting Started](docs/getting-started.md) | Full installation walkthrough |
| [Configuration](docs/configuration.md) | Every config option explained |
| [Hardware Guide](docs/hardware-guide.md) | GPU compatibility and VRAM requirements |
| [Troubleshooting](docs/troubleshooting.md) | Common issues and fixes |
| [Operations](docs/operations.md) | Monitoring, restart, day-2 ops |
| [Security](docs/security.md) | Auth, TLS, network isolation, threat model |
| [Crash Mitigation](docs/crash-prevention.md) | The n=1 reboot observation and what BASTION does about it |
| [API Reference](docs/api.md) | All endpoints with examples |
| [Deployment](docs/deployment.md) | Systemd, Docker, desktop launcher |
| [Releasing](docs/releasing.md) | One-time PyPI/OIDC setup and release cut procedure |

## Optional Extras

```bash
pip install "bastion-broker[dashboard]"   # TUI dashboard (Textual)
pip install "bastion-broker[metrics]"     # Prometheus metrics export
pip install "bastion-broker[a2a]"         # A2A agent interface
pip install "bastion-broker[persistence]" # SQLite task persistence
```

From a source checkout use the editable form -- e.g. `pip install -e ".[dashboard]"`
(and `pip install -e ".[dev]"` for testing + linting tools).

## Testing

```bash
pip install -e ".[dev,dashboard,a2a,metrics,persistence]"
python -m pytest tests/ -m "not e2e" -v
```

Install all extras for the full suite -- optional-dependency routes (metrics,
A2A, persistence) are only registered when their extra is present.

## Support & status

- **Status:** Active, alpha. Single-developer personal project, MIT-licensed.
- **Support:** best-effort via GitHub Issues; no SLA. Please include your
  version (`pip show bastion-broker`), GPU/driver, and repro steps.
- **Contributing:** small, focused PRs welcome. There is no CLA and no bug
  bounty. Security issues go through the private advisory flow above, never a
  public issue.
- **AI disclosure:** parts of this project were developed with AI assistance
  and human review.

## License

MIT License. See [LICENSE](LICENSE) for details.
