# TODO: Furoshiki Ingest Server

> Status: planned — not yet implemented.

## The core idea

A new **ingest server** (`scripts/ingest_server.py`) is a separate FastAPI process on its own port (`FUROSHIKI_INGEST_PORT`, default `7433`), managed by Brain like the repair API and listener. Operators can expose it to the network while the repair API/dashboard stays on loopback. The server accepts:

- `POST /ingest/{source_id}` — one-shot webhook event
- `WS /ingest/stream/{source_id}` — persistent WebSocket for continuous feeds

Plugins declare which source(s) they subscribe to in `PLUGIN.md` using a new `ingest_subscriber` hook type. When an event arrives, the ingest server dispatches to the subscribed plugin subprocess (same sandboxed model as all other hooks).

---

## Architecture flow

```mermaid
flowchart TD
    ext1["Stock ticker (WebSocket)"]
    ext2["STT push (HTTP POST)"]
    ext3["Audio sensor (WebSocket)"]
    ext4["Security alarm (HTTP POST)"]

    ext1 -->|"WS /ingest/stream/stocks"| IS
    ext2 -->|"POST /ingest/stt"| IS
    ext3 -->|"WS /ingest/stream/audio-events"| IS
    ext4 -->|"POST /ingest/security"| IS

    IS["ingest_server.py\n:7433 (separate port)"]

    IS --> IDB[("ingest_events\n(SQLite audit log)")]
    IS --> router["ingest_router.py\nPlugin dispatch"]

    router -->|"subprocess hook"| P1["plugin: stocks-watcher\n(ingest_subscriber)"]
    router -->|"subprocess hook"| P2["plugin: audio-awareness\n(ingest_subscriber)"]
    router -->|"conversation path"| IC["inject/conversation\nSTT → same pipeline as Telegram"]
    router -->|"no subscriber"| SE[("system_events\nfallback sink")]

    P1 -->|"Plugin API"| MQ[("mind_queue")]
    P2 -->|"Plugin API"| MQ
    SE --> SOUL["Soul Engine\nnext tick"]
    MQ --> VD["Voice Dispatcher\n→ Telegram"]
```

---

## Key design decisions

**Separate process, separate port**
- Brain adds `ingest` to `MANAGED_SERVICES` in `brain.py` (alongside `api` and `listener`)
- Opt-out via `FUROSHIKI_BRAIN_MANAGE_INGEST=0`
- Default host `127.0.0.1` (loopback) — operator explicitly sets `FUROSHIKI_INGEST_HOST=0.0.0.0` to expose it
- Port configurable via `FUROSHIKI_INGEST_PORT` (default `7433`), wired through `operator_tier1.py`

**Auth per source**
- Each registered source gets its own token (HMAC secret or Bearer), stored in a new `ingest_sources` table
- Token scoped to the `source_id` — a compromised token for one source can't pollute others
- Re-uses the existing `api_keys` table infrastructure with a new `ingest` scope

**Plugin subscription (new `ingest_subscriber` hook type)**

New hook type added to `PLUGIN.md` schema:
```yaml
hooks:
  - type: ingest_subscriber
    source: "stocks"           # source_id this plugin handles
    mode: stream               # webhook | stream | both
    entry: scripts/handle_stocks.py
    batch_interval_sec: 5      # optional: batch frames before dispatching subprocess
```

`furoshiki plugins reload` merges these into an `ingest_subscriptions.json` registry that `ingest_server.py` loads on startup (hot-reloads on SIGHUP like schedules.json).

**STT / conversation harness**
- Source declared with `source_type: conversation` in `ingest_sources`
- Ingest server routes text payload directly into the `inject/conversation` pipeline (same as `POST /inject/conversation` on the repair API)
- No plugin needed for this path — it's a first-class alternative harness alongside Telegram

**Stream buffering**
- For high-frequency WebSocket streams, per-source `batch_interval_sec` accumulates frames before spawning a plugin subprocess (avoids subprocess spam for 10 Hz stock ticks)
- Each batch is one subprocess invocation with a JSON array payload on stdin

**Fallback (no subscriber)**
- Events with no registered plugin land in `system_events` with `source_type = ingest/{source_id}` — soul engine can still react to them

---

## Task list

- [ ] Add `ingest_sources` and `ingest_events` tables to `scripts/init_db.py`
- [ ] Add `FUROSHIKI_INGEST_PORT` / `FUROSHIKI_INGEST_HOST` and all resilience knobs to `scripts/operator_tier1.py`
- [ ] Create `scripts/ingest_router.py` — normalizes events and dispatches to plugin subprocesses or `system_events` fallback
- [ ] Create `scripts/ingest_server.py` — FastAPI app with `POST /ingest/{source_id}` webhook and `WS /ingest/stream/{source_id}` WebSocket, auth per source, STT conversation path
- [ ] Add `ingest` to `MANAGED_SERVICES` in `scripts/brain.py` with `FUROSHIKI_BRAIN_MANAGE_INGEST` opt-out
- [ ] Add ingest server to `SYSTEMD_SERVICES` in `scripts/dashboard_data.py`
- [ ] Extend `furoshiki plugins reload` to merge `ingest_subscriber` hooks into `ingest_subscriptions.json`
- [ ] Add `furoshiki ingest` subcommand group to `bin/furoshiki`; update `cmd_help()` and `.cursor/skills/furoshiki-cli/SKILL.md`
- [ ] Create `docs/INGEST-SPEC.md` — authoritative spec for ingest server, subscription contract, and plugin author guide
- [ ] Update `docs/PLUGIN-SPEC.md`, `docs/SELF-AWARENESS-DESIGN.md`, `docs/INTEGRATION.md` with `ingest_subscriber` hook type and architecture
- [ ] Update `https://furoshiki.ai/plugins-extensions.html` — add `ingest_subscriber` hook type section, link to `docs/INGEST-SPEC.md`
- [ ] Update `https://furoshiki.ai/architecture.html` — add ingest server node to diagram, add security layer callout

---

## Files to create / modify

**New files:**
- `scripts/ingest_server.py` — FastAPI app with webhook + WebSocket endpoints, plugin dispatch
- `scripts/ingest_router.py` — routes normalized events to plugin subprocesses or `system_events`
- `docs/INGEST-SPEC.md` — authoritative spec for the ingest server and plugin subscription contract

**Modified files:**
- `scripts/init_db.py` — add `ingest_sources` and `ingest_events` tables
- `scripts/brain.py` — add `ingest` to `MANAGED_SERVICES`
- `scripts/dashboard_data.py` — add ingest to `SYSTEMD_SERVICES` map
- `scripts/operator_tier1.py` — add `FUROSHIKI_INGEST_PORT` / `FUROSHIKI_INGEST_HOST` and all resilience knobs
- `docs/PLUGIN-SPEC.md` — document `ingest_subscriber` hook type, `mode`, `batch_interval_sec`, new `ingest` permission
- `docs/SELF-AWARENESS-DESIGN.md` — add ingest server to architecture overview and diagram
- `docs/INTEGRATION.md` — note ingest as a push-based harness
- `https://furoshiki.ai/plugins-extensions.html` — add `ingest_subscriber` hook type section, link to `docs/INGEST-SPEC.md`
- `https://furoshiki.ai/architecture.html` — add ingest server node to diagram, add security layer callout
- `bin/furoshiki` — add `furoshiki ingest` subcommand group
- `.cursor/skills/furoshiki-cli/SKILL.md` — update command list

---

## New SQLite tables

```sql
CREATE TABLE IF NOT EXISTS ingest_sources (
    id                 INTEGER PRIMARY KEY,
    source_id          TEXT UNIQUE NOT NULL,   -- slug, e.g. "stocks"
    source_type        TEXT NOT NULL,          -- webhook | stream | conversation
    plugin_name        TEXT,                   -- subscribing plugin (null = core/fallback)
    token_hash         TEXT NOT NULL,          -- HMAC secret hash for Bearer auth
    active             INTEGER DEFAULT 1,
    circuit_open       INTEGER DEFAULT 0,
    circuit_open_until TEXT,                   -- ISO timestamp, null when closed
    created_at         TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS ingest_events (
    id            INTEGER PRIMARY KEY,
    source_id     TEXT NOT NULL,
    received_at   TEXT NOT NULL,
    payload       TEXT,                        -- JSON blob
    routed_to     TEXT,                        -- plugin name or "core" or "fallback"
    dropped_frames INTEGER DEFAULT 0,          -- frames dropped due to rate/queue limits
    status        TEXT DEFAULT 'received'      -- received | dispatched | failed
);
```

---

## New CLI commands (`furoshiki ingest …`)

- `furoshiki ingest sources list` — show all registered sources and their subscriber plugin
- `furoshiki ingest sources add <id> --type webhook|stream|conversation [--plugin <name>]` — register a source, print the generated token
- `furoshiki ingest sources remove <id>` — deregister
- `furoshiki ingest sources reset <id>` — clear circuit breaker
- `furoshiki ingest token <id>` — regenerate token for a source
- `furoshiki ingest status` — show if the ingest server is running, port, active connections

---

## Security and resilience

**Threat model and mitigations — built in from the start, not retrofitted.**

**Auth is the outermost gate**
- Every request (webhook POST or WebSocket handshake) hits the per-source Bearer token check before touching any queue, subprocess, or DB
- Unauthenticated floods are rejected at the network boundary with zero downstream cost
- Tokens are scoped to `source_id` — a compromised token for one source can't affect another

**Per-source rate limiting (token bucket)**
- Hard cap on events/sec per source (default `20`, operator-tunable)
- Webhook: excess requests return `429 Too Many Requests`
- WebSocket: excess frames are silently dropped with a `dropped_frames` counter in the audit log (not an error — the connection stays open)

**Bounded in-memory queue per source**
- Fixed-depth queue (default `500` frames) accumulates frames awaiting subprocess dispatch
- When full, oldest frames are dropped with a log warning — no memory growth
- Works in tandem with `batch_interval_sec` which is already a natural throttle

**Global concurrent subprocess cap**
- Max N plugin subprocesses active at once across all sources (default `5`, operator-tunable)
- New dispatches wait behind a bounded asyncio semaphore; queue overflow is dropped with a log warning
- Prevents a flood of events from spawning hundreds of parallel processes

**WebSocket connection limits**
- Global cap (default `10`) + per-source cap (default `3`)
- New connections above the cap receive `1008 Policy Violation` close frame
- Prevents connection exhaustion from rogue clients

**Payload size caps (FastAPI middleware)**
- Max bytes per webhook POST body (default `65536`)
- Max bytes per WebSocket frame (default `65536`)
- Hard limits enforced before any queue or processing logic

**Circuit breaker per source**
- If a source's plugin subprocess error rate exceeds threshold (default `10` failures/min), the source is auto-suspended for a configurable backoff (default `60` sec)
- Cleared automatically after backoff, or manually via `furoshiki ingest sources reset <id>`
- Prevents a broken external system from triggering endless error loops and log spam

**`ingest_events` rolling retention**
- Keep only the last N rows per source (default `1000`), pruned after each write
- Prevents DB bloat from high-frequency sources
- Doctor health check can alert if any source approaches its retention ceiling

**Process isolation boundary**
- Ingest server is a separate Brain-managed child process at its own port
- If it OOMs or crashes (e.g. from a malicious flood), Brain restarts it with exponential backoff
- Repair API, Telegram listener, and soul engine are unaffected — ingest is a guest process, not load-bearing

**All knobs in `operator_tier1.py` / `operator_settings.json` under the `ingest` key:**

```
ingest_max_events_per_source_per_sec    (default 20)
ingest_max_ws_connections               (default 10)
ingest_max_ws_connections_per_source    (default 3)
ingest_max_concurrent_subprocesses      (default 5)
ingest_queue_depth_per_source           (default 500)
ingest_event_retention_per_source       (default 1000)
ingest_max_payload_bytes                (default 65536)
ingest_circuit_breaker_errors_per_min   (default 10)
ingest_circuit_breaker_backoff_sec      (default 60)
```

These follow the existing `_int_env` / `_float_env` pattern and are merged from `operator_settings.json` so operators can tune via dashboard without touching `.env`.

---

## What this unlocks for plugins

An `audio-awareness` plugin could subscribe to source `audio-events` (WebSocket from a microphone pipeline), receive batched JSON frames like `{"event": "door_creak", "confidence": 0.91}`, and write to `mind_queue` with urgency `high` — voice dispatcher surfaces it next outreach window. A `stocks-watcher` plugin subscribes to source `stocks`, receives batched OHLCV frames, and enqueues a note when a threshold crosses. All using the existing plugin sandbox and Plugin API — no core changes needed per data source.
