Metadata-Version: 2.4
Name: fused
Version: 2.9.2b0
Summary: MCP server for end-to-end data work with cloud-native datasets
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: aiohttp<4,>=3.8.3
Requires-Dist: anthropic>=0.40.0
Requires-Dist: boto3>=1.34.0
Requires-Dist: click>=8.1.7
Requires-Dist: cryptography>=42
Requires-Dist: duckdb>=1.4.4
Requires-Dist: fastapi>=0.100.0
Requires-Dist: fsspec>=2024.3.1
Requires-Dist: httpx>=0.27
Requires-Dist: keyring>=24
Requires-Dist: loguru<0.8,>=0.7.0
Requires-Dist: mcp[cli]>=1.0.0
Requires-Dist: numpy>=1.20
Requires-Dist: pandas<3,>=2.2
Requires-Dist: pluggy>=1.5
Requires-Dist: pyarrow>=14
Requires-Dist: pydantic<3,>=2.7.0
Requires-Dist: pyjwt>=2.10
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests<3,>=2.31
Requires-Dist: rtoml<1,>=0; sys_platform != 'emscripten'
Requires-Dist: tabulate>=0.10.0
Requires-Dist: tomlkit>=0.13
Requires-Dist: tqdm<5,>=4.67.1
Requires-Dist: ty>=0.0.1a1
Requires-Dist: uvicorn[standard]>=0.23.0
Requires-Dist: yarl<2,>=1.9.4
Provides-Extra: all
Requires-Dist: fastmcp>=3.2.4; extra == 'all'
Requires-Dist: geopandas<2,>=0.14; extra == 'all'
Requires-Dist: mercantile<2,>=1.2.1; extra == 'all'
Requires-Dist: pillow>=10; extra == 'all'
Requires-Dist: rasterio<2,>=1.3.10; extra == 'all'
Requires-Dist: rioxarray<0.16,>=0.15; extra == 'all'
Requires-Dist: selenium>=4.44.0; extra == 'all'
Requires-Dist: shapely<3,>=2; extra == 'all'
Requires-Dist: xarray>=2024.3; extra == 'all'
Provides-Extra: batch
Requires-Dist: affine<3,>=2; extra == 'batch'
Requires-Dist: geopandas<2,>=0.14; extra == 'batch'
Requires-Dist: importlib-resources<6,>=5.12.0; extra == 'batch'
Requires-Dist: jinja2<4,>=3.1.2; extra == 'batch'
Requires-Dist: mercantile<2,>=1.2.1; extra == 'batch'
Requires-Dist: pandas<3,>=2.2.0; extra == 'batch'
Requires-Dist: pillow>=10; extra == 'batch'
Requires-Dist: pyproj<4,>=3.7.1; extra == 'batch'
Requires-Dist: rioxarray<0.16,>=0.15; extra == 'batch'
Requires-Dist: shapely<3,>=2; extra == 'batch'
Requires-Dist: xarray>=2024.3; extra == 'batch'
Provides-Extra: dropbox
Requires-Dist: dropbox; extra == 'dropbox'
Provides-Extra: hubspot
Requires-Dist: hubspot-api-client; extra == 'hubspot'
Provides-Extra: local
Requires-Dist: keyrings-alt>=5; extra == 'local'
Provides-Extra: raster
Requires-Dist: mercantile<2,>=1.2.1; extra == 'raster'
Requires-Dist: pillow>=10; extra == 'raster'
Requires-Dist: rasterio<2,>=1.3.10; extra == 'raster'
Requires-Dist: rioxarray<0.16,>=0.15; extra == 'raster'
Requires-Dist: xarray>=2024.3; extra == 'raster'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.0.0; extra == 'snowflake'
Provides-Extra: vector
Requires-Dist: geopandas<2,>=0.14; extra == 'vector'
Requires-Dist: mercantile<2,>=1.2.1; extra == 'vector'
Requires-Dist: shapely<3,>=2; extra == 'vector'
Description-Content-Type: text/markdown

# fused

[![CI](https://github.com/fusedio/openfused/actions/workflows/ci.yml/badge.svg)](https://github.com/fusedio/openfused/actions/workflows/ci.yml)

Fused agent toolkit — the data SDK is `fused`; agent/MCP commands are top-level, the legacy proprietary SDK CLI is under `fused workbench`.

An MCP server for end-to-end data work with cloud-native datasets.

**Find → Load → Explore → Analyze → Decide → Act**

It gives AI agents and developers a single toolset to explore, run code against, and act on cloud data — with the compute environment managed for you.

Data lives in S3 as Parquet/Arrow files. Code runs in AWS Lambda — or fully locally when you don't have (or don't want) a cloud account, or before one is provisioned. Everything is accessible to AI agents via the Model Context Protocol and CLI. No manual installation needed.

### Two surfaces

fused is the **core daemon that executes data work** — the platform layer that data agents and apps build on:

1. **Bare CLI / MCP server** (`fused …` / `uv run fused`) — run and execute code against cloud data; agents drive it over the Model Context Protocol.
2. **Bare widget UI** (`fused widget open` / `fused dev serve`) — open and interact with widgets in the context of a folder, operating through UDFs.

The full UI-first app experience — the local web control plane for the complete agent-orchestration workflow — is a separate product, **[Flow](https://github.com/fusedio/flow)** (`fusedio/flow`), that consumes fused's CLI/MCP and its `_core` UDF store.

## Capabilities

fused gives an agent (or a person at the CLI) the tools for an end-to-end data workflow:

- **Find & inspect data** — list buckets and objects, count and filter files, read a Parquet/Arrow/CSV schema and row count, and mint presigned download URLs.
- **Run code against that data** — execute Python in an isolated, per-call sandbox with the environment's packages and input files, returning a `result`; run pytest suites with line and branch coverage.
- **Scale out** — fan out across partitions from inside an execution using the same coordinator/worker pattern on every backend.
- **Verify & audit** — optionally gate each execution with static and dependency scans, an LLM spec check, and result expectations, recording an immutable audit event per call.
- **Cache** — content-address results so a repeat call returns a stored value instead of re-executing.
- **Serve** — expose a route's code as an HTTP endpoint, locally or as a managed cloud deployment.
- **Visualize** — render an interactive, data-bound dashboard/widget from a JSON tree of `{type, props, children}` nodes, served by the local widget-host viewer (`fused widget open` / `fused dev serve`); data-bound nodes resolve DuckDB SQL through the same compute and cache. The CLI closes the loop with the human: `widget open` blocks until they press a submit button or close the tab and prints their input values as JSON, and the **parley** (`widget push` / `widget watch`) keeps a standing agent↔human channel on the same server — the agent pushes successive views into one persistent page while the human's events stream back.
- **Manage secrets & infrastructure** — read/write provider secrets and plan/apply/teardown the cloud resources each environment needs.

The same tool surface works across [pluggable backends](#backends), so code written against fused doesn't change when the execution target does. See [Tools](#tools) for the full list, or [`spec/overview.md`](spec/overview.md) for a deeper tour.

## Quickstart

```bash
uv sync
claude mcp add openfused -- uv run fused
```

Or add manually to your MCP client config (e.g. Claude Desktop `claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "openfused": {
      "command": "uv",
      "args": ["run", "openfused"]
    }
  }
}
```

### Verify it works

Create an environment and run a one-line smoke test. **You don't need a cloud account to get started:** the local backend runs code on your machine in a cached virtualenv.

```bash
# Option A — AWS: runs code in Lambda, auto-provisioning a managed IAM role +
# S3 cache bucket; the compute function runs the env's container image
# (build it with `infra build-image`)
uv run fused env create prod --backend aws

# Option B — local, no AWS account required: runs code on this machine in a
# cached virtualenv (created with uv when available, else pip)
uv run fused env create dev --backend local

# Run code — with a single env, it is selected automatically
uv run fused code run -c "result = 1 + 1"   # -> result: 2
```

The paths are independent: the local backend makes no AWS calls, and AWS needs only credentials. For a guided first-time setup (backend choice, AWS credential checks, provisioning, and teardown), see the bundled `openfused-setup` skill.

On first use against AWS, fused automatically creates and manages the IAM role needed to run Lambda functions. You can override this with `OPENFUSED_ROLE_ARN` if you prefer to supply your own role.

By default the MCP server exposes only read and compute tools. Pass flags to unlock additional capability:

```bash
# Also expose tools that create/update environments and apply infra changes
uv run fused --enable-infra

# Also expose tools that permanently delete resources (use with care)
uv run fused --enable-infra --enable-destructive
```

Both flags have env-var equivalents: `OPENFUSED_ENABLE_INFRA=1` and `OPENFUSED_ENABLE_DESTRUCTIVE=1`.

## Backends

Fused runs the same tools against a pluggable backend. Two run on infrastructure you control, plus one Fused-hosted backend:

- **AWS** (`aws`, default) — storage in S3, compute in AWS Lambda, secrets in AWS Secrets Manager. The path for production and scale: it fans out horizontally and needs only AWS credentials (fused creates and manages the IAM role and supporting resources for you). The compute function runs the environment's container image, built and pushed with `infra build-image`.
- **Local** (`local`) — storage on the local filesystem, compute on the host in a cached virtualenv built with uv or pip, secrets in an encrypted local file. No cloud account required. Use it for development, offline/air-gapped work, CI, or before a cloud environment is provisioned; code runs directly on the host, so there is no container isolation.
- **Fused** (`fused`) — Fused's hosted, managed environment, reached over its data-plane endpoint (an MCP client of the remote `fused` tool surface). Code runs in a per-tenant sandbox Fused operates; the local side provisions nothing. See [`spec/fused-onboarding.md`](spec/fused-onboarding.md) for login → key → env setup.

All implement the same contract, so code written against one runs unchanged on the others — switching is just a config change. Select the backend with `OPENFUSED_BACKEND=aws|local` (see [docs/configuration.md](docs/configuration.md)) or per named environment with `fused env create --backend …`.

## Tools

| Tool | What it does |
|---|---|
| `list_files` | List S3 buckets, or list files in a bucket with pagination |
| `count_files` | Count files in a bucket, optionally filtered by extension |
| `get_file` | Get a presigned download URL for a file |
| `get_file_schema` | Get column schema and row count for a Parquet, Arrow IPC, or CSV file |
| `upload_file` | Upload text or binary content to S3 |
| `execute_code` | Run Python code in Lambda; assign `result` to return a value. Ship files via `input_files` (filename→host path, read from disk and inspectable in the call) or `input_file_contents` (filename→base64, for inline content). Large results spill to S3 (`result_ref`; `large_result_delivery` picks inline vs. presigned). `cache_max_age` memoizes the result by content hash and skips re-execution on a hit; it defaults to `"1h"` (pass `"0s"` to disable for live data, or any duration like `"15m"`) and `cache_refresh=True` busts a fresh entry. Pass `project=<name>` (local backend only) to use the project's `.venv` interpreter and project-aware cache identity; a stale venv is auto-reconciled (`uv sync`) before running unless `OPENFUSED_NO_VENV_SYNC` is set, else it runs stale, skips caching, and surfaces `warnings` |
| `test_code` | Run pytest tests against user code in Lambda; returns per-test outcomes, line coverage, and branch coverage — also available as `fused code test`. Supports `cache_max_age`/`cache_refresh`. `project` is **required on the local backend** (pytest/coverage must be dev deps in the project's `pyproject.toml`) |
| `verify_code` | Scan code, dependencies, and input files for security issues without executing — also available as `fused code verify` |
| `cache_clear` | Delete cached/spilled result objects for the resolved environment (`route` / `object_key` / default-env); `all=True` (every env + spill) requires `--enable-destructive`. Always audited |
| `get_audit_log` | Return recent security audit events from the local SQLite audit store (`~/.openfused/audit.db`, optionally merged with S3); filter by event type, status, or workspace project — CLI: `fused audit log` |
| `get_secret` | Retrieve a secret from AWS Secrets Manager |
| `put_secret` | Create or update a secret in AWS Secrets Manager |
| `list_secrets` | List secrets, optionally filtered by name prefix |
| `delete_secret` | *(`--enable-destructive`)* Delete a secret (AWS: scheduled with the default 30-day recovery window; local backends: removed from the encrypted store immediately). Also available as `fused secrets delete` |
| `project_new` | Scaffold a new project in the default workspace; auto-creates the workspace (git init + pre-commit hook) on first use |
| `list_projects` | Return a sorted list of project names in the default workspace |
| `project_show` | Re-sync the project manifest from disk, then return name, path, and UDF list |
| `project_status` | Show the live deploy status for all UDFs in a project from the resolved env's cloud snapshot; flags orphaned UDFs |
| `list_deployed` | List all projects with a cloud deploy snapshot in the resolved environment |
| `list_envs` | List all named environments with their backend |
| `env_show` | Show full config for a named or the resolved environment |
| `get_project_context` | Context packet for the workspace project in effect: identity, SKILL.md contract, dataset notes, widgets, UDF scripts, knowledge, resolved environment — CLI: `fused project show` |
| `infra_plan` | Dry-run: show AWS resource drift without applying changes |
| `env_create` | *(`--enable-infra`)* Create a named environment and optionally provision AWS resources |
| `env_update` | *(`--enable-infra`)* Update config fields for a named environment |
| `infra_apply` | *(`--enable-infra`)* Reconcile AWS resources (IAM role, Lambda, ECR, S3) to desired state |
| `infra_build_image` | *(`--enable-infra`)* Build the environment's container image and push to ECR |
| `infra_lambda_reset` | *(`--enable-infra`)* Delete all Lambda functions with the active prefix, clear the cache, and recreate; suppressed by `--disable-reset` |
| `udf_deploy` | *(`--enable-infra`)* Deploy a UDF to preview or release channel; requires AWS env + `cache_bucket` + provisioned serving plane |
| `udf_promote` | *(`--enable-infra`)* Promote a UDF from preview to release (repoints release to preview's current commit) |
| `udf_rollback` | *(`--enable-infra`)* Roll back the release channel of a UDF to a prior commit |
| `env_delete` | *(`--enable-destructive`)* Remove environment config (cloud resources untouched) |
| `infra_teardown` | *(`--enable-destructive`)* Delete Lambda functions, IAM role, ECR repo, and S3 cache bucket |
| `udf_retire` | *(`--enable-destructive`)* Revoke a UDF's channel mounts and drop it from the deploy snapshot (workspace-id conflict gated; `force` to take over) |

> **Consuming-app teamwork tool (Tier B, not on this MCP).** A consuming app such as
> [Flow](https://github.com/fusedio/flow) injects its own loopback teamwork MCP into the agent runs it
> spawns — e.g. an approval gate requested through `ask_user(… effect: "approval_gate", effectArgs: { verb, detail })`
> (`verb`: `storage_write` \| `secret_write` \| `external_side_effect`) that an agent calls **before** a gated
> Act to request human approval (the approval-gate contract lives in `spec/autonomy.md` §3;
> cooperative/advisory). That surface is the consuming app's, not fused's: it is **never** exposed on the
> external read-only project MCP listed above.

## Configuration

Configure fused with [named environments](#named-environments) (`fused env`) or environment variables. The full list of variables and their defaults lives in [docs/configuration.md](docs/configuration.md).

## Named environments

`fused env` manages named configurations stored in `~/.openfused/envs.json`, so you can switch between backends without managing environment variables manually:

```bash
# Create and provision a named AWS environment
fused env create prod --backend aws --prefix myapp-

# Run a command against a specific environment
fused --env prod secrets list
fused --env staging secrets list

# Pin a project's default environment (validated against stored envs)
fused project set my-project --env prod
```

Environment resolution order for every command:
1. `--env` flag or `OPENFUSED_ENV` env var → explicit override
2. Inside a project with `[project].default_env` set in `openfused.toml` → manifest pin
3. Exactly one environment exists → selected automatically (sole-env auto)
4. Multiple environments, no pin → error: set `default_env` or pass `--env`

See `fused env --help` for the full subcommand list.

## Observability

Logs, CloudWatch metrics (Lambda + cache), the audit log, and the system signals
an operator should watch (concurrency, cache hit rate, bucket/DB growth) are
documented in [docs/observability.md](docs/observability.md).

## Examples

Runnable examples live in [`examples/`](examples/) — from [`hello_world_python.py`](examples/hello_world_python.py) and single-file DuckDB queries to multi-step pipelines (`examples/github_archive/`, `examples/ais_sf_ferries/`) and Lambda fan-out across partitions. Run any of them with:

```bash
uv run fused code run examples/hello_world_python.py --file
```

For the conventions these examples follow when you build your own project, see [docs/project-structure.md](docs/project-structure.md).

## Requirements

- Python 3.11+ and [uv](https://docs.astral.sh/uv/)
- For the AWS backend: AWS credentials with permissions to create Lambda functions, manage IAM roles, and read/write S3. The environment's container image is built in **AWS CodeBuild by default** (no local Docker); pass `--builder local` to build with a host Docker daemon instead
- For the local backend: nothing beyond Python — uv is used for fast venv creation when present, else pip
- For the `fused widget ...` viewer (the widget-host): **Node 20+** on PATH. Its UI + server are bundled into the published package, so a single `pip install fused` ships them — no extra install or source checkout. (Building the package from source additionally needs Node 22.13+ and pnpm + npm; see [docs/building.md](docs/building.md).)

## Documentation

- [CONTRIBUTING.md](CONTRIBUTING.md) — development setup and architecture overview.
- [docs/building.md](docs/building.md) — how fused is built and distributed as a single self-contained Python package (the `widget-host` standalone viewer bundled into the wheel), builder requirements, and the publish steps.
- [docs/project-structure.md](docs/project-structure.md) — how to lay out a project built on fused (scripts, specs, workers/coordinators, tests).
- [`spec/`](spec/) — the design specification. Start with [`spec/overview.md`](spec/overview.md) for capabilities, entrypoints, and system parts; see [`spec/assumptions.md`](spec/assumptions.md) for runtime prerequisites. Each topic has a provider-neutral hub (`backends.md`, `environments.md`, `infra.md`) plus per-provider files (e.g. `-aws.md`, `-local.md`).
