Metadata-Version: 2.5
Name: digitalbase
Version: 26.917.2
Summary: DigitalBase API (FastAPI, LLM_BACKEND=ollama|vllm|sglang)
Requires-Python: <3.14,>=3.13
Requires-Dist: adlfs>=2024.7.0
Requires-Dist: apscheduler>=3.10.0
Requires-Dist: asyncua>=1.1.5
Requires-Dist: bcrypt>=4.0.0
Requires-Dist: boto3>=1.34.0
Requires-Dist: cryptography>=3.1
Requires-Dist: fastapi-azure-auth>=5
Requires-Dist: fastapi>=0.100.0
Requires-Dist: fsspec>=2024.10.0
Requires-Dist: gcsfs>=2024.10.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: ldap3>=2.9.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: oracledb>=2.4.0
Requires-Dist: paho-mqtt>=2.1.0
Requires-Dist: pandas>=3.0.3
Requires-Dist: paramiko>=3.4.0
Requires-Dist: pgvector>=0.4.2
Requires-Dist: pillow>=10.0.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: psycopg2-binary>=2.9.11
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyjwt[crypto]>=2.8
Requires-Dist: pymcprotocol>=0.3.0
Requires-Dist: pymodbus>=3.6.0
Requires-Dist: pymysql>=1.1.1
Requires-Dist: pyodbc>=5.1.0
Requires-Dist: pypdfium2>=5.0.0
Requires-Dist: pytesseract>=0.3.13
Requires-Dist: python-docx>=1.1.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: python-multipart>=0.0.20
Requires-Dist: python-pptx>=0.6.23
Requires-Dist: pywhispercpp>=1.4.0
Requires-Dist: s3fs>=2024.10.0
Requires-Dist: scikit-learn>=1.5.0
Requires-Dist: scipy>=1.14.0
Requires-Dist: smbprotocol>=1.12.0
Requires-Dist: sqlalchemy>=2.0.44
Requires-Dist: uvicorn[standard]>=0.23.0
Requires-Dist: watchdog>=4.0
Requires-Dist: webdav4>=0.10.0
Provides-Extra: demo
Requires-Dist: embedded-postgres>=18.6.3; extra == 'demo'
Provides-Extra: gpu
Requires-Dist: faster-whisper>=1.1; extra == 'gpu'
Requires-Dist: nvidia-cublas-cu12; (sys_platform == 'linux') and extra == 'gpu'
Requires-Dist: nvidia-cudnn-cu12==9.*; (sys_platform == 'linux') and extra == 'gpu'
Requires-Dist: vllm; (sys_platform == 'linux') and extra == 'gpu'
Provides-Extra: vllm
Requires-Dist: vllm; (sys_platform == 'linux') and extra == 'vllm'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.1; extra == 'whisper'
Description-Content-Type: text/markdown

# DigitalBase API

FastAPI backend of DigitalBase (internal name: lmlight). Single source tree — the
inference backend is chosen at runtime with `LLM_BACKEND=ollama|vllm|sglang|none`
(`vllm` / `sglang` spawn and supervise their own chat/embed/vision servers via
`common/llm/local_engine.py`; `none` = cloud LLM APIs only). Shipped to customers
as a PyInstaller single binary built by `.github/workflows/release.yml`.

## Run from source

```bash
cd api
uv sync                      # deps + dev group (pytest, ruff, cython, ...)
# write api/.env (python-dotenv): DATABASE_URL, JWT_SECRET, LLM_BACKEND, ... (all keys: common/config/)
uv run python main.py        # listens on API_HOST:API_PORT (default 0.0.0.0:8000)
uv run python manage.py db diff   # show pending schema ALTERs (see ../docs/migrations.md)
```

PostgreSQL with pgvector is required for knowledge (RAG) (`DATABASE_URL`). Startup creates schemas, tables
and the `vector` extension itself; the database must already exist. Without the extension the API still
starts with RAG disabled, and an admin enables it from License > RAG search enhancement once a superuser has
run `CREATE EXTENSION vector` (see `../docs/migrations.md`). The SPA build in `static/` is served by the
same process when present.

## Layout

| Dir | Role |
|---|---|
| `main.py` | app assembly: lifespan (DB migration, local engine start), router registration, SPA static files |
| `auth/`, `user/`, `admin/`, `system/`, `system_settings/` | login/JWT/LDAP/OIDC, user settings, admin console (users/models/license/env/audit), health/license |
| `chat/` | chat orchestration (NDJSON stream, tool loop, approvals, history, compaction, prompt suggest) |
| `rag/` | bots, documents, ingest jobs (`/api/bots`, `/api/documents`, `/api/upload`) |
| `sql/`, `table/`, `files/` | SQL connections + datalake + dashboards, table editing engine, file storage with shared-dir ACL (`files/virtual_path.py` = the user_id-based virtual path resolver operators and tools share) |
| `operators/` | execution units shared by workflows, chat tools and MCP: flow/transform/ai/files/data/saas/fa/actions (`@register`, `ExecutionContext`) |
| `pipeline/` | workflow engine: graph, scheduler, runs, webhooks, AI flow editing (runs `operators/`) |
| `connections/` | API connection (SaaS credential) CRUD; SQL / file / OAuth connections live in `sql/` `files/` `oauth/` |
| `mcp_servers/`, `oauth/`, `approvals/`, `prompts/`, `history_share/`, `transcribe/`, `benchmark/` | managed custom MCP servers, OAuth connectors, tool approvals, prompt library, history sharing, whisper, benchmark |
| `gateway/` | `/v1/*` OpenAI/Anthropic-compatible API and `/mcp` server for `sk-` API keys |
| `common/` | cross-cutting code: `config/`, `db/` (models + startup migrations), `llm/` (all LLM I/O over httpx, `invoke.py` = one-shot call helper), `rag/` (`lifecycle.py` = document / chunk deletion shared by upload and workflows), `sql/`, `tools/` (tool handlers, permissions, MCP), `connectors/` (HTTP retry / templating core), `document/`, `file_connector/` (`snapshot.py` = file_landed diff ledger), `runtime/` |
| `tests/` | `unit/` + `contract/` (no external deps), `e2e/` + `connection/` (need `E2E_BASE`), see `tests/README.md` |

## Conventions (short list)

- Function-first Python; classes only for state / inheritance / library requirements. One-line Japanese comments.
- LLM HTTP goes through `common/llm` (httpx, `auth_headers()`), never an SDK. Chat uses `/v1/chat/completions` only.
- New module → import it explicitly in the package `__init__.py` **and** add a `--hidden-import` in `release.yml`
  (`tests/contract/test_packaging.py` enforces this; Cython-compiled modules are invisible to PyInstaller).
- Tunable constants live in `common/config` via `get_env(..., advanced=True)`; API datetimes use `iso_utc()`.
- Document text comes only from `common.document.extract_document` (chat attachments, tools, RAG, pipelines); callers never import parsers or branch on format (`tests/contract/test_document_extraction_boundary.py`).
- Full principles: `../docs/CODING_PRINCIPLES.md`.

## Tests

Single entry point is the repo-root `./test.sh` (layer definitions: `../docs/test/MATRIX.md`).

```bash
../test.sh unit          # api unit+contract + app vitest (no external deps)
../test.sh api           # api unit+contract only
../test.sh integration   # tests/e2e against E2E_BASE (default http://localhost:8000); add `fast` to skip @llm
../test.sh connection    # tests/connection (mocks: ../test.sh env up demo)
../test.sh smoke         # GET sweep of every route on a running instance
uv run ruff check .      # lint / format (ruff format)
```

Release gate (`release.yml`) runs only `tests/contract/test_packaging.py` plus the app `tsc` / `vite build`;
everything else is run manually before tagging.
