Metadata-Version: 2.5
Name: digital-twins-kb
Version: 0.11.2
Summary: Environment-portable KB ingestion: layered config, fail-fast named sources, deterministic dedup-safe ingest into user-supplied Qdrant/Neo4j.
Author: Terry Zhou
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: neo4j>=5.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: qdrant-client>=1.7
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: local-embedding
Requires-Dist: sentence-transformers>=3.0; extra == 'local-embedding'
Requires-Dist: torch>=2.0; extra == 'local-embedding'
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.2; extra == 'mcp'
Description-Content-Type: text/markdown

# digital-twins

Environment-portable KB ingestion: layered config, fail-fast named sources, and
deterministic dedup-safe ingest into user-supplied Qdrant + Neo4j. Built per
`specs/001-package-foundation/` (see `AGENTS.md` for sources of truth).

## Fast path (new machine to first ingest)

The fast path is two separate steps: **Step 1 — Install** (the
installer) and **Step 2 — Setup** (the `digital-twins setup` wizard).
The installer is install-only by default: it stops after the
`pip install` and prints "install-only … run `digital-twins setup`" —
it does **not** end with the wizard. Step 2 is a follow-up command you
run when you are ready to configure backends, init the store, and
create the admin account.

### Step 1 (Install)

**No clone / no manual steps — the remote one-liner** (the
`curl | bash` entry point; it self-bootstraps Python ≥ 3.11, creates an
isolated venv, and installs from PyPI):

```bash
curl -fsSL https://raw.githubusercontent.com/terrygzhou/digital-twins/main/scripts/install.sh | bash
```

> **Piping into `bash`** runs code from the network. To audit first,
> download it and read it (`bash -x scripts/install.sh` traces every command):
>
> ```bash
> curl -fsSL https://raw.githubusercontent.com/terrygzhou/digital-twins/main/scripts/install.sh -o install.sh && bash install.sh
> ```
>
> The script performs no network access except the `pip install` from PyPI.

**Already have a checkout?** Run the same installer locally instead of
piping (finds Python ≥ 3.11, creates an isolated venv — via `uv` when
the `uv` binary is already on PATH, else the stdlib `venv` module — and
installs the package):

```bash
bash scripts/install-local.sh
```

It stops after the `pip install` and prints an install-only note ("run
`digital-twins setup` when you are ready to configure backends, init the
store, and create the admin account"). Re-running is a safe no-op on a
host that is already installed. Useful options:
`--extras "mcp,local-embedding"` (pick the extras), `--with-setup` (run
the `digital-twins setup` wizard right after the install — the opt-in
that chains install + setup), `--run-ingest` (ingest a demo source right
away). (`--cloud`, `--cloud-env`, and `--skip-services` only take effect
together with `--with-setup`; `--no-setup` is a no-op alias kept for one
release — the installer is install-only by default, so there is nothing
to opt out of.)
When stdin is not a terminal, the installer just installs: under a pipe
without `--with-setup` there is no wizard to run, so it prints
`install-only … run digital-twins setup` and exits 0. In CI or under a
pipe where you *do* want the wizard in one shot, use `--with-setup`
together with `--cloud-env` (endpoints come from the `KB_*` env vars,
never a prompt). See `bash scripts/install-local.sh --help`.

**Prefer to do it by hand?** Two separate steps — install, then setup
(the installer only does the first one):

```bash
uv venv .venv && source .venv/bin/activate       # uv (if installed); see note below
uv pip install "digital-twins-kb[mcp]"            # step 1: base + MCP; drop [mcp] if not needed
# — or, without uv:
#   python3 -m venv .venv && source .venv/bin/activate
#   pip install "digital-twins-kb[mcp]"
digital-twins setup            # step 2: wizard — backend detection + init + admin + health
digital-twins run --source fs  # your first ingest
```

> **`uv` (optional, recommended):** the installer and the steps above
> use `uv` when it is already on PATH (faster, no system-python
> dependency); otherwise they fall back to the stdlib `venv` module.
> Neither installer ever installs `uv` for you.  To get it:
> `curl -LsSf https://astral.sh/uv/install.sh | sh`
> (macOS: `brew install uv`).

> **PEP 668 ("externally managed" error on macOS/Homebrew, Debian, or
> other distros that protect system Python):** the venv line above is the
> clean fix on any OS. Alternative: `pipx`, which manages its own venv and
> symlinks the CLI to `~/.local/bin`:
>
> ```bash
> # macOS:
> brew install pipx && pipx ensurepath
> # Debian/Ubuntu:
> sudo apt install pipx
> # Arch:
> sudo pacman -S pipx
> # then (on any OS):
> pipx install "digital-twins-kb[mcp]"
> ```
>
> On Debian/Ubuntu where `python3 -m venv` is unavailable (missing the
> `venv` module), install it first: `sudo apt install python3-venv`.

> **CPU-only host?** PyPI's default torch wheel is ~5 GB with CUDA bundled.
> If you plan to use in-process embedding (`[local-embedding]` extra) on a
> machine without a GPU, install the CPU wheel first:
>
> ```bash
> pip install torch --index-url https://download.pytorch.org/whl/cpu
> pip install "digital-twins-kb[local-embedding]"
> ```
>
> On a GPU host skip the CPU-wheel line and just
> `pip install "digital-twins-kb[local-embedding]"`.

> **Three layers, one `-kb` suffix** (only `digital-twins-kb` goes into
> `pip install` — the bare `digital-twins` name is PyPI-blocked as too
> similar to the existing `digital-twins-kb`):
> - `digital-twins-kb` — the **PyPI distribution** name (what you pass to
>   `pip install`; PEP 503 normalizes `-`/`_`/`.` in dist names)
> - `digital-twins` — the **CLI** on your PATH after install
> - `digital_twins` — the **Python import** name (`python -m digital_twins`
>   works too)

### Step 2 (Setup)

`digital-twins setup` is the first-run wizard: backend detection (local
Docker stack or cloud endpoints) + init + first admin account + health
checks, all in one command. This is the step the installer does
**not** run by default (install-only default); run it when you are
ready:

```bash
digital-twins setup
```

The `setup` wizard detects the backend: when Docker is available it offers
to start the bundled local stack (qdrant + neo4j + embedding-model, plus
the bundled LLM when a GPU is present — a host with `nvidia-smi`
installed but no usable GPU is treated as no-GPU, same as the
bootstrap script) and writes `kb.local.yml` for you; when it is not, it
prompts for the three cloud endpoints instead. In either case it creates
the state DB, the first admin account (a generated password is shown
once and written to `admin-credentials.txt` in your state dir, created
with mode 600 — delete it after your first login), runs the health
checks, and prints the next step.

> **Where your files live** — config and state are in *separate* dirs by
> default: `kb.local.yml` is in the config dir
> (`~/.config/digital-twins/`), while `state.db` and
> `admin-credentials.txt` live in the state dir (`~/.digital-twins/`).
> `ls ~/.digital-twins/` showing only `state.db` is normal — the config
> file is not there. Both are overridable via `KB_CONFIG_DIR` /
> `KB_STATE_DIR`.

#### Choosing backends: local Docker or per-service cloud endpoints

> **No Docker / want to use your own endpoints?** Don't run the default
> wizard — pick the mode that fits you:
>
> - **`digital-twins setup --cloud`** — skip Docker entirely; you are
>   prompted for your Qdrant / Neo4j / LLM endpoints (+ optional embedding
>   endpoint + API key + Neo4j credentials). It writes `kb.local.yml` and
>   never starts a container.
> - **`digital-twins setup --cloud-env`** — the same, but non-interactive:
>   endpoints come from the `KB_*` env vars (`KB_QDRANT__URL`,
>   `KB_NEO4J__URL`, `KB_NEO4J__USER` / `KB_NEO4J__PASSWORD`,
>   `KB_LLM__ENDPOINT`, optional `KB_EMBEDDING__ENDPOINT` /
>   `KB_EMBEDDING__API_KEY`). Safe in CI / under a pipe; exits 5 naming
>   any missing var.
> - **`digital-twins setup --backends qdrant=…,neo4j=…,llm=…,embedding=…`** —
>   per-service choice; supply a URL for every service and no Docker is
>   used at all.
>
> Any of these writes `kb.local.yml` and then the rest of the wizard
> (state DB, admin account, health checks) runs as usual.

Each of the four services — qdrant, neo4j, llm, embedding — can be
served by the bundled local Docker stack (`local`) or by an external
cloud endpoint (a URL you point at via env var). Services not named in
`--backends` default to `local`; if every service ends up external the
local stack is not started:

| Service   | Local (Docker)                           | External (cloud URL) via env var |
|-----------|------------------------------------------|----------------------------------|
| qdrant    | `local` (qdrant/qdrant:1.9.7)           | `KB_QDRANT__URL`               |
| neo4j     | `local` (neo4j/neo4j:5.18-community)    | `KB_NEO4J__URL` / `KB_NEO4J__USER` / `KB_NEO4J__PASSWORD` |
| llm       | `local` (bundled, GPU required)         | `KB_LLM__ENDPOINT` / `KB_LLM__MODEL` |
| embedding | `local` (BAAI/bge-small-en-v1.5, 384-dim) | `KB_EMBEDDING__ENDPOINT`     |

Pick the whole local stack at once with `--local` (starts all four
services; skips Docker detection and the cloud prompts):

```bash
digital-twins setup --local
```

Pick per-service backends with `--backends KEY=VAL,...`, where each
`VAL` is `local` (the bundled Docker service) or an external URL
(placeholders below — use your own endpoints):

```bash
# qdrant from Docker, llm + embedding from a cloud endpoint,
# neo4j not named → defaults to local.
digital-twins setup --backends \
    qdrant=local,llm=https://example.com/v1,embedding=https://example.com/v1
```

Flag precedence (highest wins): `--skip-services` > `--cloud-env` >
`--cloud` > `--local` / `--backends` > interactive.

#### Flags

| Flag | Effect |
|------|--------|
| `--skip-services` | "I handle backends myself": never probe Docker, never prompt, never write `kb.local.yml`; only init + admin + health checks. **Takes precedence over `--cloud` and over a valid `kb.local.yml`** (a re-run where the backend is already up). |
| `--cloud` | Skip Docker detection, go straight to cloud mode (prompts for the three required endpoints). Set via env (`KB_QDRANT__URL`, `KB_NEO4J__URL`, `KB_LLM__ENDPOINT`) or answer the prompts; optional `KB_EMBEDDING__ENDPOINT` (+ `KB_EMBEDDING__API_KEY` when the endpoint is hosted and key-protected), `KB_NEO4J__USER`, `KB_NEO4J__PASSWORD`. |
| `--cloud-env` | Non-interactive cloud mode: the three required endpoints must come from `KB_QDRANT__URL` / `KB_NEO4J__URL` / `KB_LLM__ENDPOINT` env vars; never a prompt. Exits 5 naming the missing var(s) when any required var is unset or empty. Safe under a pipe or in CI. |
| `--local` | Start the bundled local stack for all four services (qdrant + neo4j + llm + embedding); skip Docker detection and the cloud prompts. Suppressed by `--skip-services` / `--cloud-env` / `--cloud`. |
| `--backends KEY=VAL,...` | Per-service backend choice (see the table above). Each `VAL` is `local` or a URL; an empty `VAL` marks the service required-external (the URL must come from the `KB_*` env var). Unknown service names are rejected. Suppressed by `--skip-services` / `--cloud-env` / `--cloud`. |

Exit codes (the installer's `--with-setup` pass-through mirrors these):
`0` all checks pass · `1` a check failed (remediation names
the endpoint) · `3` local stack's mandatory services did not become
healthy (a half-up stack where the survivors answer the health poll still
writes `kb.local.yml` and continues — you only get 3 when the survivors
also fail) · `5` cloud endpoints could not be resolved (the remediation
names exactly which of `KB_QDRANT__URL` / `KB_NEO4J__URL` /
`KB_LLM__ENDPOINT` are empty; an env var set to an empty string is
honored as "set but empty" — it fails the gate, it does not fall through
to the prompt) · `6` the wizard was interrupted (Ctrl-C, or EOF at a
prompt) before the backend was configured — re-run `digital-twins setup`
in a real terminal, or set the `KB_*` env vars and re-run with
`--cloud-env`.

## Exposing to agents (MCP)

Once installed with the `[mcp]` extra, exposing your KB to an MCP-capable
agent is two commands + one JSON block:

```bash
export DT_USER_PASSWORD=<admin password>          # minting auth
digital-twins token create --as <admin-email>     # mint a personal token (shown once)
digital-twins serve-mcp --http                    # long-running HTTP server on 127.0.0.1:8770
```

> The token commands authenticate first: either `DT_PERSONAL_TOKEN`
> (a token you already have) or the admin password via
> `DT_USER_PASSWORD` + `--as <admin-email>`. The admin password is the
> one generated at `setup`/`init` — it lives in
> `~/.digital-twins/admin-credentials.txt` (written once, mode 600).

Agent MCP config (Claude Desktop / Cursor / DSH / Hermes — same shape):

```json
{
  "mcpServers": {
    "digital-twins": {
      "url": "http://127.0.0.1:8770/mcp",
      "headers": { "Authorization": "Bearer <your-token>" }
    }
  }
}
```

Stdio variant (agent spawns the server as a child process):

```json
{
  "mcpServers": {
    "digital-twins": {
      "command": "digital-twins",
      "args": ["serve-mcp", "--transport", "stdio"],
      "env": { "DT_MCP_TOKEN": "<your-token>" }
    }
  }
}
```

All 10 tools (6 scheduler: `kb_schedule_list/create/update/delete/run`
+ `kb_run_history`; 4 KB: `kb_search`, `kb_chat`, `kb_ingest`, `kb_health`)
are available; role-gating and owner-scoping apply per call. Full quickstart:
[`specs/004-mcp-scheduler-tools/quickstart.md`](specs/004-mcp-scheduler-tools/quickstart.md).

## Manual onboarding (reference)

The steps below are what `digital-twins setup` does for you. Use them
if you want to do it manually, or to re-do one specific step.

### Prerequisites

- **Python ≥ 3.11** (`python3 --version`)
- Either **Docker** (for the bundled local stack) **or** already-running
  Qdrant + Neo4j + an OpenAI-compatible LLM + embedding endpoint you can
  point at.
- A GPU is optional — CPU-only hosts work (see the install note above).

### Install the package

The fast path above shows the minimal case. Details:

```bash
# Create a venv (recommended: system Python on many distros is
# "externally managed" and refuses direct pip installs — PEP 668)
python3 -m venv .venv
. .venv/bin/activate

# Base install (lightweight, no torch): endpoint-based embedding
# (embedding.endpoint in kb.local.yml) and all non-embed commands work.
pip install digital-twins-kb

# Optional extras (combine as needed):
#   [mcp]             — MCP SDK for external agents
#   [local-embedding] — in-process BGE model, no endpoint needed
#   [dev]             — dev dependencies
pip install "digital-twins-kb[mcp]"
pip install "digital-twins-kb[local-embedding]"
```

From a git checkout instead of PyPI:

```bash
pip install .
```

**Check:**

```bash
digital-twins --version
# digital-twins, version 0.11.0
```

### Start the backend services (pick one)

The tool ingests into **your** Qdrant + Neo4j and uses an OpenAI-compatible
LLM + embedding endpoint. Three ways to get them:

**Option A — bundled local stack (Docker), one command:**

```bash
bash scripts/bootstrap-local.sh
```

Brings up qdrant (6333), neo4j (7474/7687), an LLM (8000) and an
embedding-model service (8080), and writes a machine-local `kb.local.yml`
pointing at them. On a no-GPU host the bundled `llm` is skipped — point
`KB_LLM__ENDPOINT` at an external LLM instead (see
[`docs/configuration.md`](docs/configuration.md) for the no-GPU path).
Tear down with `docker compose down`.
Exit codes: `0` healthy · `1` docker missing/down · `2` port conflict
(names the service) · `3` health timeout (prints `docker compose logs` hint).

> **Note:** run this with `bash`, not `sh`. The script uses
> `set -o pipefail` (line 19) which dash does not support; on systems where
> `sh` is dash, `sh scripts/bootstrap-local.sh` will fail immediately.

**Option B — cloud or external services (no Docker):**

```bash
bash scripts/bootstrap-local.sh --cloud
```

Set the required env vars (or answer the interactive prompts on stdin):

- `KB_QDRANT__URL` — Qdrant URL (e.g. `https://host:6333`)
- `KB_NEO4J__URL` — Neo4j URL (`bolt://` or `http(s)://`)
- `KB_LLM__ENDPOINT` — OpenAI-compatible LLM base URL

Optional: `KB_EMBEDDING__ENDPOINT` (+ `KB_EMBEDDING__API_KEY` for hosted endpoints that need a Bearer token), `KB_NEO4J__USER`, `KB_NEO4J__PASSWORD`.

Writes `~/.config/digital-twins/kb.local.yml` pointing at the cloud
endpoints. No Docker required. Exit codes: `0` success · `5` a required
endpoint was not provided (names the missing variable(s)).

**Option C — your own services (manual):** skip the script and make sure
Qdrant, Neo4j, the LLM and the embedding endpoint are reachable; you'll tell
the tool where they live in the init step below.

### Initialise

> **If you ran `digital-twins setup` (Step 2 of the fast path above), `init`
> and `validate` have already
> been done for you** — this section is only needed when you skipped `setup`
> (e.g. you installed the package by hand and want to configure endpoints
> manually). `setup` runs `init` + the first-admin step + the health checks
> (equivalent to `validate`) in one go, and adds backend detection and the
> `kb.local.yml` write.

```bash
digital-twins init
```

The wizard prompts, in order:

1. `qdrant.url` (e.g. `http://localhost:6333`)
2. `neo4j.url` / `neo4j.user` / `neo4j.password` (e.g. `bolt://localhost:7687`)
3. `llm.endpoint` / `llm.model` (OpenAI-compatible base URL + model name)
4. **first admin email + password** — this account is your `admin` role;
   every later sign-up is a `reader` (see [Roles & tokens](#roles--tokens))

It creates the config dir with `kb.local.yml` (every source `enabled: false`),
the state dir + `state.db`, and prints a health report.
**Check:** the report shows every service `ok` and the command exits 0.
If an endpoint is unreachable the command exits 1 and names the failing
service plus how to fix it — fix it and re-run `digital-twins init --yes`.
Re-running is safe (idempotent): existing values are kept, only missing
pieces are added.

### Validate

> **Subsumed by `setup`** — the wizard runs the same health checks and prints
> the same report. Use `validate` on its own only if you ran `init` manually
> and want to re-check after changing an endpoint.

```bash
digital-twins validate
```

**Check:** exit 0, and every configured endpoint reachable with the Qdrant
collection vector size matching the embedding model. A dimension mismatch is
a hard error naming the mismatch and the remediation ("re-embed, or point at
a new collection"), exit 1.

### Enable a source and run your first ingest

Point a source at real content. The `fs` source is the simplest demo:

```bash
# In <config dir>/kb.local.yml (or via env: KB_SOURCES__FS__ENABLED=true etc.):
#   sources.fs: { enabled: true, extra: { dir: /tmp/kb-demo } }
# (put two .md files in /tmp/kb-demo first)
digital-twins run --source fs
```

**Check:** run 1 reports `fs: 2 item(s)` and writes an audit row with a
`run_id`. Run it again:

```bash
digital-twins run --source fs   # second run
```

Run 2 reports `fs: 0 item(s)` — the collection still holds exactly 2 points.
Dedup-safe and deterministic: the same content ingested via schedule,
`run --once`, MCP, or web UI yields **one** point, not four.

> **Fail-fast:** enabling a source whose credential/prerequisite is unset
> makes `run` exit 2, naming the source, the missing prerequisite, and where
> to set it (e.g. `credential env var YMAIL_APP_PASSWORD is not set
> (required by sources.yahoo)`). Nothing is ingested; the failed run is still
> audited.

### Put it on a schedule (and/or expose it)

Pick the surface(s) you need:

```bash
# 6a — define a recurring schedule (presets: hourly, every-N-hours,
#     daily, weekly, monthly)
digital-twins schedule add --source fs --preset daily --fire-time 03:00
digital-twins schedule list

# 6b — long-running scheduler: fires due schedules
digital-twins serve

# — or — one-shot host-cron path (no long-running process):
digital-twins run --once
# cron: 0 3 * * * digital-twins run --once

# 6c — web app: UI + /api/* REST
digital-twins web

# 6d — MCP server for external agents (see "Exposing to agents" above)
digital-twins token create              # mint a personal token
digital-twins serve-mcp --http         # HTTP on 127.0.0.1:8770
digital-twins serve-mcp --http --port 9000
digital-twins serve-mcp                # stdio NDJSON (agent spawns it)
```

> **6b vs `run --once`:** `serve` keeps a long-running process that fires
> due schedules on their exact times; `run --once` fires whatever is due
> **now** and exits, for a host cron. Pick one, not both (double-fire).
> **6c vs `serve`:** the web app is for humans (UI + REST); `serve` is the
> scheduler.

## Architecture

> **Data lifecycle diagram:**
> Mermaid source (rendered vertically):

```mermaid
flowchart TD
    %% Phase 0: configuration (the substrate)
    config["config/ — 4-layer load<br/>env(.env) → local → yml → defaults"]

    %% Phase 1: ingestion (read from sources)
    subgraph sources["sources (8)"]
        fs["fs: directory of files"]
        hermes["hermes: session export"]
        pi["pi: session store .jsonl"]
        dsh["dsh: .jsonl.zstd sessions"]
        paperclip["paperclip: PG DB"]
        mail["yahoo / gmail: IMAP"]
        custom["custom: entrypoint"]
    end

    %% Phase 2: transformation (chunk + embed)
    prereq{"fail-fast prereq check"}
    chunk["chunk_text (max_chars, overlap)"]
    embed["embedder (endpoint or pinned BGE)"]
    ids{"deterministic point_id (uuid5)"}

    %% Phase 3: storage (upsert + state)
    qdrant[("Qdrant (vectors, payload)")]
    neo4j[("Neo4j (graph, optional)")]
    sqlite[("SQLite (WAL) — highwater + audit_runs")]

    %% Phase 4: scheduling (wraps 1–3)
    presets["presets (daily/hourly/weekly/monthly/N)"]
    loop["serve loop + pidfile, tick ~3s"]

    %% Phase 5: retrieval / serving (read side)
    web["web/ /api/* (search, ingest, audit)"]
    mcp["mcp/ stdio + http (kb_search, kb_chat, kb_ingest, kb_health, …)"]
    search{"owner-scoped vector search"}

    %% Cross-cutting
    audit["audit_runs — row per run (ok/partial/failed)"]
    multi["multi-user auth (sessions, tokens, roles)"]

    %% data flow
    config -->|resolved cfg| sources
    sources -->|"IngestItem (key, content, ts)"| prereq
    prereq -->|ok| chunk
    prereq -.->|"PrerequisiteError → exit 2, audited failed"| audit
    chunk -->|chunk text| embed
    embed -->|vectors| ids
    ids -->|"upsert point (dedup: NFR-1)"| qdrant
    ids -->|MERGE nodes| neo4j
    qdrant -->|highwater cursor| sqlite
    qdrant --> search
    sqlite --> audit

    %% scheduling drives the pipeline
    loop -->|"fire due, trigger=schedule"| prereq
    presets -->|next_fire_at| loop

    %% serving surfaces
    web -->|Bearer session| search
    mcp -->|"role-gate + owner-scope"| search
    web -->|"POST /api/ingest/run, trigger=web"| prereq
    mcp -->|"kb_ingest, trigger=mcp"| prereq

    %% multi-user wraps serving
    multi --> web
    multi --> mcp

```

It shows the phases: **config → sources (read) → ingest (chunk + embed +
deterministic `point_id`) → store (Qdrant / Neo4j / SQLite) → serve
(web + MCP, owner-scoped search)**, with **scheduling** and
**audit** cross-cutting the middle three.


```
                ┌──────────────────────────────────────────────┐
                │           digital_twins (package)            │
                │                                              │
user ──────────▶ │  cli.py  ──▶  config/  ──▶  4-layer load   │
                │   (all cmds)    schema     env→local→yml→def│
                │              │                              │
                │              ▼                              │
                │  sources/ ──▶  ingest/  ──▶  state/        │
                │  (8 named)     ids+chunks   SQLite (WAL)    │
                │                    │              │         │
                │                    ▼              ▼         │
                │              Qdrant (vector)  Neo4j (graph) │
                │                                              │
                │  scheduler/  ◀── cron (host) or serve loop │
                │  web/        ◀── UI + /api/* (REST)         │
                │  mcp/        ◀── NDJSON stdio / POST /mcp   │
                └──────────────────────────────────────────────┘
```

- **config** — 4-layer, env wins; `KB_` prefix, `__` = nesting; local
  secrets in `kb.local.yml` (gitignored).
- **sources** — fail-fast prerequisite check; `fs` needs no credential;
  others need an env var (see `docs/configuration.md`).
- **ingest** — content-addressed chunk ids; deterministic; dedup-safe.
- **state** — SQLite in `~/.digital-twins` (WAL, FK on); runs + access
  log + user config overrides.
- **scheduler / web / mcp** — three independent surfaces that all call the
  same `ingest` core (NFR-14: one point, not four).

## Configuration

Knobs live in `kb.local.yml` (or env). The full list with defaults and
types: [`digital_twins/config/knobs.py`](digital_twins/config/knobs.py).

| Knob | Default | Meaning |
|------|---------|---------|
| `qdrant.url` | (unset) | Qdrant base URL — required. |
| `neo4j.url` / `neo4j.user` / `neo4j.password` | (unset) / `neo4j` / (unset) | Neo4j — required. |
| `llm.endpoint` / `llm.model` | (unset) / (unset) | OpenAI-compatible LLM — required. |
| `embedding.endpoint` | (unset) | Optional; skips local model load when set. |
| `state_dir` | `~/.digital-twins` | Where `state.db` lives. |
| `config_dir` | `~/.config/digital-twins` | Where `kb.local.yml` lives. |
| `sources.<name>.enabled` | `false` | Enable a source. |
| `sources.<name>.extra.*` | source-specific | `dir`, `mailbox`, … |
| `sources.<name>.prerequisites` | built-in | Extra prereqs; unset → fail-fast. |
| `scheduler.presets` | 5 built-ins | Recurring schedule presets. |

Full detail + examples: [`docs/configuration.md`](docs/configuration.md).

## Multi-user & tokens

Accounts and credentials live in the state DB (`~/.digital-twins`):

- **Admins** sign in with a password (PBKDF2, `sessions`); the first
  `init` creates one; `digital-twins account` (list / set-role / delete)
  manages more.
- The generated password is shown once and written to
  `admin-credentials.txt` in the state dir; **there is no
  password-reset path** — if the file is missing, removing
  `~/.digital-twins` and re-running `setup` re-creates the admin
  from scratch.
- To mint a personal token (needed for `digital-twins token …`,
  MCP, cron): export the admin password into
  `DT_USER_PASSWORD` and run
  `digital-twins token create --as <admin-email>` — the plaintext
  token is printed once. (For machine-to-machine auth you can also
  set `DT_SERVICE_TOKEN`.)
- **Readers** sign in and get an 8 h session token (`sessions`), or a
  personal token (`personal_tokens`, `pt_` prefix, revocable).
- **Machine-to-machine** (MCP / cron) uses the shared service token
  `DT_SERVICE_TOKEN` (env), or a personal token minted via
  `digital-twins token create` (see "Exposing to agents" above).
  Minting authenticates with the admin password through
  `DT_USER_PASSWORD` + `--as <admin-email>`.

Role model (default): `reader` → read/search/list; `scheduler` →
reader + schedule CRUD/trigger; `admin` → scheduler + user management.

| `digital-twins` sub-command | Does |
|---|---|
| `signup` | create the first account (admin) or a named reader |
| `account` | list / set-role / delete accounts + `whoami` (admin-gated except `whoami`) |
| `token` | create / list / revoke personal tokens |
| `session` | revoke a web session token |
| `channels` | manage ingestion channels: list / status / enable / disable / add |
| `config` | list / set / unset config knobs (write to `kb.local.yml`) |
| `schedule` | add / list / remove schedules |
| `run` / `run-history` | one-shot ingest run; audit-run history |
| `serve` / `serve-mcp` | long-running scheduler; MCP server (stdio/http) |
| `setup` / `init` | first-run wizard; legacy alias (deprecated) |
| `migrate s4` | drop legacy `:KbItem`/`:KbChunk` graph nodes on upgraded installs |
| `validate` | config + service health checks without ingesting |
| `web` | the web app (UI + /api/*); admin-only ops check the role |

Full detail: [`docs/multi-user.md`](docs/multi-user.md).

## Scheduling

Five presets: `hourly`, `every-N-hours` (`extra: {hours: N}`), `daily`
(default 03:00), `weekly` (default Monday 04:00), `monthly` (default 1st
05:00). Three surfaces: the long-running scheduler (`serve`), the host
cron one-shot (`run --once`), and the web UI `/api/schedules`.

> **Note:** `run --once` fires every due schedule and exits — for a host
> cron entry (`0 3 * * * digital-twins run --once`), not a long-running
> process. Use `serve` when you need the scheduler resident (exact-fire
> times, missed-run catch-up, status endpoint).

Full detail: [`docs/scheduling.md`](docs/scheduling.md).

## Community

Versioning rules (what counts as major / minor / patch, the deprecation
mechanism, when version bumps happen):
[`docs/semver-policy.md`](docs/semver-policy.md). The in-repo issue tracker
lives in `.github/ISSUE_TEMPLATE/` — a bug report, a feature request, and a
config-breaking-change form (required `version_impact` + `migration_note`).
File issues there; no live remote required.

## License

MIT. See [`LICENSE`](LICENSE) and
[`pyproject.toml`](pyproject.toml) (`license = { text = "MIT" }`).

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run the hermetic suite (unit + integration, in-memory Qdrant + stubs)
pytest -q

# Opt-in live tests against real services
KB_LIVE_QDRANT=... KB_LIVE_NEO4J=... pytest -m live
```

## Uninstall

Reverses the fast path above (venv + `pip install "digital-twins-kb[mcp]"` +
`digital-twins setup` + `bash scripts/bootstrap-local.sh`). One script,
one invocation; every step is a no-op when its target is absent, so it is
safe to re-run and safe on a host that never had anything installed:

```bash
bash scripts/uninstall-local.sh                # stop docker stack, pip uninstall
bash scripts/uninstall-local.sh --tear-down-volumes   # also remove the named
                                                      # volumes (all ingested content)
bash scripts/uninstall-local.sh --remove-data         # also delete the config
                                                      # + state dirs
bash scripts/uninstall-local.sh --force               # skip every confirmation
bash scripts/uninstall-local.sh --skip-docker         # cloud/external hosts

> **Where does this script live?** `uninstall-local.sh` ships in the repo's
> `scripts/` dir — it is **not** installed by the `curl … | bash install.sh`
> one-liner, so it isn't on a host that only used that. If you don't have a
> checkout, fetch it without one:
>
> ```bash
> curl -fsSL https://raw.githubusercontent.com/terrygzhou/digital-twins/main/scripts/uninstall-local.sh \
>   -o /tmp/uninstall-local.sh
> bash /tmp/uninstall-local.sh --remove-data --force --skip-docker
> ```
>
> Or skip the script entirely — `--remove-data` is just `rm -rf` of the two
> data dirs (back up `~/.digital-twins/state.db` first):
> ```bash
> cp ~/.digital-twins/state.db ~/state-backup.db
> rm -rf ~/.digital-twins ~/.config/digital-twins
> ```
```

> **Note:** run this with `bash`, not `sh` — the script uses
> `set -o pipefail`, which dash does not support (same caveat as the
> bootstrap script).

> **Warning — `--tear-down-volumes` on shared infrastructure:**
> the shipped `docker-compose.yml` uses **named** volumes (`qdrant-data`,
> `neo4j-data`, `digital-twins-state`). Docker volume names are **global**
> across the whole daemon, not per compose project — if you run another
> stack (yours or a teammate's) that references the same volume names,
> `docker compose down -v` will delete **their** data too, not just the
> local stack's. Check `docker volume inspect qdrant-data` before answering
> "y" to the confirmation prompt.
>
> If your qdrant/neo4j are shared, long-lived, or bind-mounted from a host
> path you maintain yourself: **don't run `--tear-down-volumes` at all** —
> use the plain `bash scripts/uninstall-local.sh` (instances stopped, data
> kept) and let whoever owns the data decide when to remove the volumes
> manually. Note that `down -v` also does *not* delete local images
> (`qdrant:1.9.7`, `neo4j:5.18-community`, the built `digital-twins` image);
> that's a separate manual `docker rmi` step this script does not touch.

What each flag does (all off by default):

| Flag | Effect |
|------|--------|
| _(none)_ | `docker compose down` (volumes kept, so a re-bootstrap resumes) + `pip uninstall -y digital-twins-kb`. Config and state dirs are kept. |
| `--tear-down-volumes` | `docker compose down -v`: also removes the named volumes (`qdrant-data`, `neo4j-data`, `digital-twins-state`) — **all ingested content is lost**. |
| `--remove-data` | Also `rm -rf` the machine-local config dir (`KB_CONFIG_DIR`, default `~/.config/digital-twins`) and state dir (`KB_STATE_DIR`, default `~/.digital-twins`) — including `kb.local.yml`, `state.db`, and `admin-credentials.txt`. |
| `--force` | Skip every interactive `y/N` confirmation. |
| `--skip-docker` | Skip the `docker compose down` step (for hosts that use cloud/external backends, i.e. `bootstrap-local.sh --cloud`). |

Each removal is confirmed interactively unless `--force` is given; a
non-interactive invocation (EOF on stdin) is treated as "no", so nothing is
deleted without an explicit yes. A pipx-managed install is detected and
reported (run `pipx uninstall digital-twins` instead) rather than
removed by this script.

Exit codes: `0` success (or everything was already absent) · `1` a step
failed and `--force` was not given (remediation names the step and the
manual command) · `2` `--tear-down-volumes` was requested and `down -v`
failed (a volume could not be identified; nothing was removed) — this exit
code applies only without `--force`; under `--force` a failed `down -v` is
treated as a transient step failure, the script continues, and the failure
is reported in the final `kept:` line.
