Metadata-Version: 2.4
Name: bastion-broker
Version: 0.5.0a2
Summary: Batch Affinity Scheduler for Throttled Inference on Ollama Networks
Project-URL: Homepage, https://github.com/cypwin/bastion
Project-URL: Repository, https://github.com/cypwin/bastion
Project-URL: Documentation, https://github.com/cypwin/bastion/tree/main/docs
Project-URL: Changelog, https://github.com/cypwin/bastion/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/cypwin/bastion/issues
Author: Cyprian Winogradow
License-Expression: MIT
License-File: LICENSE
Keywords: broker,gpu,inference,ollama,proxy,scheduler,vram
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Hardware
Requires-Python: >=3.11
Requires-Dist: fastapi<1.0,>=0.115
Requires-Dist: httpx<1.0,>=0.27
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: uvicorn[standard]<1.0,>=0.30
Provides-Extra: a2a
Requires-Dist: a2a-sdk[http-server]<1.0,>=0.3; extra == 'a2a'
Provides-Extra: dashboard
Requires-Dist: httpx<1.0,>=0.27; extra == 'dashboard'
Requires-Dist: psutil<8.0,>=5.9; extra == 'dashboard'
Requires-Dist: textual<8.0,>=1.0; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: httpx<1.0,>=0.27; extra == 'dev'
Requires-Dist: hypothesis<7.0,>=6.0; extra == 'dev'
Requires-Dist: mypy<2.0,>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio<1.0,>=0.24; extra == 'dev'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'dev'
Requires-Dist: pytest<9.0,>=8.0; extra == 'dev'
Requires-Dist: respx<1.0,>=0.21; extra == 'dev'
Requires-Dist: ruff<1.0,>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml<7.0,>=6.0; extra == 'dev'
Provides-Extra: metrics
Requires-Dist: prometheus-client<1.0,>=0.20; extra == 'metrics'
Provides-Extra: persistence
Requires-Dist: aiosqlite<1.0,>=0.20; extra == 'persistence'
Description-Content-Type: text/markdown

# BASTION

![CI](https://github.com/cypwin/bastion/actions/workflows/ci.yml/badge.svg)
![PyPI](https://img.shields.io/pypi/v/bastion-broker)
![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)
![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-blue.svg)

**Batch Affinity Scheduler for Throttled Inference on Ollama Networks**

A system-level GPU inference broker that prevents crashes from concurrent
[Ollama](https://ollama.com) access. BASTION sits as a transparent HTTP proxy
between your applications and Ollama, serializing model loads, enforcing VRAM
budgets, and eliminating the memory-mapped I/O conditions that cause GPU driver
crashes under heavy inference workloads.

---

## Why BASTION?

Running multiple LLM clients against a single Ollama instance is a recipe for
hard crashes. Here is why:

1. **Memory-mapped model loading.** Ollama memory-maps model files by default.
   When multiple models are loaded concurrently, the virtual address space
   balloons far beyond physical VRAM, and the OS page cache competes with GPU
   memory for the same pages.

2. **Rapid model cycling.** With `OLLAMA_MAX_LOADED_MODELS=1`, every request
   for a different model triggers a full unload/load cycle. Under concurrent
   access from multiple clients, this creates dozens of VRAM
   allocation/deallocation cycles per minute.

3. **No upstream protection.** Ollama has no built-in queue, no VRAM budget
   enforcement, and no cooldown between model transitions.

The result: after roughly 60 rapid model swaps, the GPU driver hits an
unrecoverable state -- kernel OOM, display freeze, or full system crash
requiring a hard reboot.

**BASTION solves this at the system level**, sitting transparently on the
standard Ollama port so every client benefits with zero configuration changes.

## Prerequisites

- Python 3.11+
- Linux with NVIDIA GPU and working drivers (`nvidia-smi` must respond)
- [Ollama](https://ollama.com) installed
- At least one model pulled (`ollama pull llama3.1:8b`)

## Quick Start

### 1. Install

```bash
pip install bastion-broker
```

Or from source, for development:

```bash
git clone https://github.com/cypwin/bastion.git
cd bastion
pip install -e ".[dev]"
```

> Installed as **`bastion-broker`** on PyPI; the command you run and the module you import are both `bastion`.

### 2. Move Ollama to port 11435

```bash
sudo mkdir -p /etc/systemd/system/ollama.service.d/
sudo tee /etc/systemd/system/ollama.service.d/override.conf > /dev/null << 'EOF'
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11435"
EOF
sudo systemctl daemon-reload
sudo systemctl restart ollama
```

Or manually: `OLLAMA_HOST=127.0.0.1:11435 ollama serve`

### 3. Configure

```bash
bastion --init-config       # Generate ~/.config/bastion/broker.yaml
bastion --detect-models     # Discover installed models
```

### 4. Validate your setup

```bash
bastion --validate
```

### 5. Start BASTION

```bash
bastion                              # Default config
bastion --config config/broker.yaml  # Custom config
bastion --admin-port 9999            # Two-port mode
```

### 6. Verify

```bash
curl http://localhost:11434                # "Ollama is running"
curl http://localhost:11434/broker/status  # Broker status
ollama run llama3.1:8b "Hello, world!"    # Transparent proxy
```

## Key Features

- **Transparent HTTP proxy** -- drop-in replacement on port 11434; existing
  clients work without any changes. Streams NDJSON faithfully so `ollama run`
  remains responsive.

- **Affinity-based scheduling** -- per-model sub-queues drain all pending
  requests for the currently loaded model before swapping, dramatically
  reducing GPU model transitions.

- **Priority aging** -- four priority tiers (interactive, agent, pipeline,
  background) with time-based aging to prevent starvation of lower-priority
  requests.

- **VRAM budget enforcement** -- tracks GPU memory via Ollama `/api/ps` fused
  with `nvidia-smi` data. Enforces a configurable budget and blocks model
  loads that would exceed it.

- **Crash prevention** -- BASTION injects `use_mmap: false` into scheduled
  inference requests (`/api/generate`, `/api/chat`, `/api/embed`) when the
  client hasn't specified `use_mmap` explicitly. This mitigates PCIe power
  transients that crashed RTX 5090 during mmap-backed model cycling.
  Passthrough endpoints (`/api/pull`, `/api/show`, `/api/tags`, etc.) forward
  unchanged. Cooldown periods between model transitions and swap-rate limiting
  provide additional protection.

- **19-panel TUI dashboard** -- real-time Textual dashboard showing GPU
  thermals, VRAM usage, queue depth, scheduler state, A2A tasks, leases, audit
  events, and more.

- **A2A protocol support** -- Agent-to-Agent protocol with agent card
  discovery, task lifecycle, batch inference, and model reservation leases.

- **Prometheus metrics + OpenTelemetry tracing** -- optional observability
  integrations with graceful no-op fallbacks.

- **Circuit breaker** -- three-state (closed/open/half-open) protection
  against cascading failures when the Ollama backend is unresponsive.

- **Per-IP rate limiting** -- configurable rate limiting middleware.

- **Tiered audit logging** -- JSONL audit logs with content hashing and
  automatic rotation.

## Architecture

```
                          Clients
    (ollama run, Claude Code, Python scripts, A2A agents, curl)
                             |
                             | :11434 (standard Ollama port)
                             v
    +------------------------------------------------------------+
    |                        BASTION                              |
    |                                                             |
    |  +----------------+  +-----------------+  +--------------+  |
    |  | Ollama Proxy   |  | Admin API       |  | A2A Agent    |  |
    |  | /api/*         |  | /broker/*       |  | /a2a/*       |  |
    |  |                |  |                 |  |              |  |
    |  | - use_mmap     |  | - status/queue  |  | - tasks      |  |
    |  |   injection    |  | - health/vram   |  | - streaming  |  |
    |  | - NDJSON       |  | - preload       |  | - leases     |  |
    |  |   streaming    |  | - unload/drain  |  | - agent card |  |
    |  | - priority     |  | - metrics       |  | - batch      |  |
    |  |   detection    |  |                 |  |   inference  |  |
    |  +-------+--------+  +-----------------+  +--------------+  |
    |          |                                                   |
    |  +-------v-------------------------------------------------+ |
    |  |          Affinity Queue + Scheduler                      | |
    |  |                                                          | |
    |  |  - Per-model sub-queues (minimize GPU model swaps)       | |
    |  |  - Priority tiers: INTERACTIVE > AGENT > PIPELINE > BG  | |
    |  |  - Aging: effective = base + (age_seconds * 2.0)        | |
    |  |  - Cooldown: 2s between model transitions               | |
    |  |  - VRAM ledger (assume/confirm/forget pattern)           | |
    |  |  - GPU health gating (temp, power, utilization)          | |
    |  |  - Concurrent co-resident dispatch (up to 3 models)     | |
    |  +-------+-------------------------------------------------+ |
    +-----------|---------------------------------------------------+
                | :11435
    +-----------v---------------------------------------------------+
    |                     Ollama (backend)                           |
    |              OLLAMA_HOST=127.0.0.1:11435                      |
    +---------------------------------------------------------------+
```

## Dashboard

```bash
bastion-dashboard
bastion-dashboard --url http://localhost:11434 --interval 2.0
```

(Or: `python -m bastion.dashboard` / `python -m bastion.dashboard --url http://localhost:11434 --interval 2.0`)

Prefer a clickable app? Install a desktop entry (app menu + optional autostart)
with `scripts/install-desktop.sh` — see
[Deployment → Desktop Launcher](docs/deployment.md#desktop-launcher).

**Keyboard shortcuts:** `p` preload model, `u` unload model, `d` toggle drain
mode, `r` manual refresh, `h` help overlay, `q` quit.

## Documentation

| Guide | Description |
|-------|-------------|
| [Getting Started](docs/getting-started.md) | Full installation walkthrough |
| [Configuration](docs/configuration.md) | Every config option explained |
| [Hardware Guide](docs/hardware-guide.md) | GPU compatibility and VRAM requirements |
| [Troubleshooting](docs/troubleshooting.md) | Common issues and fixes |
| [Operations](docs/operations.md) | Monitoring, restart, day-2 ops |
| [Security](docs/security.md) | Auth, TLS, network isolation |
| [Crash Prevention](docs/crash-prevention.md) | How BASTION prevents GPU crashes |
| [API Reference](docs/api.md) | All endpoints with examples |
| [Deployment](docs/deployment.md) | Systemd, Docker, desktop launcher |
| [Releasing](docs/releasing.md) | One-time PyPI/OIDC setup and release cut procedure |

## Optional Extras

```bash
pip install "bastion-broker[dashboard]"   # TUI dashboard (Textual)
pip install "bastion-broker[metrics]"     # Prometheus metrics export
pip install "bastion-broker[a2a]"         # A2A agent interface
pip install "bastion-broker[persistence]" # SQLite task persistence
```

From a source checkout, use the editable form instead — e.g. `pip install -e ".[dashboard]"` (and `pip install -e ".[dev]"` for testing + linting tools).

## Testing

```bash
pip install -e ".[dev]"
python -m pytest tests/ -v
```

## License

MIT License. See [LICENSE](LICENSE) for details.
