# cube-mcp

MCP server for EdgescaleAI Cube management and Apollo deployments.

> **New owner / inheriting this repo?** Jump to [Operations & Ownership](#operations--ownership). The two runbooks you need are [docs/admin.md](docs/admin.md) (full RBAC + infra reference) and [docs/adding-new-user.md](docs/adding-new-user.md) (onboard a teammate end-to-end). Both are written so you can point Claude Code at them and have it execute the steps for you.

## Architecture

**cube-agent** runs locally as an MCP server (via `npx @edgescaleai/cube-mcp`). It handles authentication, Docker builds, Helm chart packaging, and app proxy tunnels. Server-side operations (Kubernetes, Teleport, Apollo, Groundcover observability) are proxied to **cube-cloud**, a FastAPI backend hosted on AWS ECS.

```
┌──────────────────────────────────────────────────────────────────────────────┐
│  LOCAL (developer machine)                                                   │
│                                                                              │
│  ┌─────────────┐    MCP JSON-RPC     ┌──────────────────────────────────┐   │
│  │ Claude Code  │◄──────────────────►│  cube-agent (MCP server)         │   │
│  └─────────────┘                     │                                  │   │
│                                      │  Local tools:                    │   │
│                                      │   • agent_login_browser          │   │
│                                      │   • agent_logout                 │   │
│                                      │   • agent_status                 │   │
│                                      │   • build_and_publish_to_apollo  │   │
│                                      │   • app_proxy / stop / status    │   │
│                                      │                                  │   │
│                                      │  Remote tools:                   │   │
│                                      │   (proxied to cube-cloud ──────) │   │
│                                      └────────┬──────────┬──────────────┘   │
│                                               │          │                   │
│   ~/.cube-agent/  ◄── API key storage         │          │ WebSocket         │
│   localhost:PORT  ◄── app proxy listener      │          │ /tunnel           │
│                                               │          │                   │
└───────────────────────────────────────────────┼──────────┼───────────────────┘
                                                │          │
                               HTTPS + Bearer   │          │  TCP-over-WS
                               POST /mcp/        │          │  relay
                                                │          │
┌───────────────────────────────────────────────┼──────────┼───────────────────┐
│  CLOUD (AWS ECS)                              │          │                   │
│                                               ▼          ▼                   │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │  cube-cloud (FastAPI)                                                  │  │
│  │                                                                        │  │
│  │  ┌─────────────────────┐     ┌──────────────────────────────────────┐ │  │
│  │  │  Auth Middleware     │     │  RBAC (Cognito + Profiles)          │ │  │
│  │  │  • Validate API key  │────►│  • Cognito group → profile          │ │  │
│  │  │    (DynamoDB lookup)│     │  • Profile → Apollo credentials     │ │  │
│  │  │  • Attach profile   │     │    (AWS Secrets Manager)            │ │  │
│  │  └─────────────────────┘     └──────────────────────────────────────┘ │  │
│  │                                                                        │  │
│  │  Cloud tools (RBAC-gated):                                            │  │
│  │  Kubernetes        Apollo Environments      Apollo Products           │  │
│  │  • cube_list       • list_environments      • list_products           │  │
│  │  • cube_status     • install_entity         • compare_product_        │  │
│  │  • kubectl_exec    • uninstall_entity         versions                │  │
│  │  • app_list        • entity_health          • list_release_channels   │  │
│  │  • list_services   • plan_details           • get_product_releases    │  │
│  │                    • cancel_active_plan                               │  │
│  │  Expose (Teleport) • update_entity_config   Groundcover (read-only)   │  │
│  │  • expose          • enforce_entity_config  • groundcover_list_*      │  │
│  │  • unexpose                                 • groundcover_query_*     │  │
│  │  • list_exposed    Modules                  • groundcover_search_*    │  │
│  │                    • list_modules           • groundcover_k8s_events  │  │
│  │  Secrets           • install_module                                   │  │
│  │  • create_secret   • uninstall_module       Registry                  │  │
│  │  • update_secret   • update_module_         • acr_get_token           │  │
│  │                      variables                                        │  │
│  └────────────────────────┬───────────────────────────┬──────────────────┘  │
│                           │                           │                      │
│  ┌────────────────────────┴────────────────┐          │                      │
│  │  tbot sidecar (Teleport credentials)    │          │                      │
│  │  • IAM join method (Fargate identity)   │          │                      │
│  │  • identity/  → tsh commands            │          │                      │
│  │  • kube/{cluster}/ → kubeconfigs        │          │                      │
│  │  • app-{name}/ → app TLS certs          │          │                      │
│  └──────────┬─────────────────────────────┘          │                      │
│             │                                         │                      │
└─────────────┼─────────────────────────────────────────┼──────────────────────┘
              │                                         │
              │ tsh/tctl (short-lived certs)            │ GraphQL (OAuth2)
              ▼                                         ▼
┌──────────────────────────┐            ┌──────────────────────────────────┐
│  Teleport                │            │  Apollo                          │
│  edgescaleai.teleport.sh │            │  edgescaleai.palantirapollo.com  │
│                          │            │                                  │
│  • Kube clusters         │            │  • Environments & Modules        │
│  • App proxies           │            │  • Entities (Helm charts)        │
│  • SSH access            │            │  • Products & Release Channels   │
│  • Identity & TLS certs  │            │  • Change Requests               │
└──────────────────────────┘            │  • ACR (Docker + Helm registry)  │
                                        └──────────────────────────────────┘

                        │ HTTPS (per-cube, read-only)
                        ▼
            ┌──────────────────────────────────┐
            │  Groundcover                     │
            │  api.groundcover.com             │
            │  • Metrics / logs / traces       │
            │  • K8s events & workload health  │
            └──────────────────────────────────┘

A separate browser front door (not shown above) serves `expose`d in-cube web
apps to account-less external users: browser → `<app>-<env>` wildcard subdomain
→ ALB → cube-cloud browser proxy → tbot mTLS dial into the cube. See
[docs/browser-app-proxy-spec.md](docs/browser-app-proxy-spec.md).
```

### How it works

1. **Local tools** run directly on the developer's machine inside `cube-agent`. These handle authentication (Cognito browser login, API key storage), Docker builds, Helm chart packaging, and app proxy tunnels.

2. **Cloud tools** are proxied from `cube-agent` to `cube-cloud` (FastAPI on AWS ECS) via MCP Streamable HTTP. The MCP endpoint (`/mcp`) and the tunnel relay (`/tunnel`) are authenticated with a Bearer API key validated against DynamoDB. (The browser surfaces additionally accept the same key from a parent-domain session cookie — on app subdomains, and on the apex under the `/console` prefix only; `/mcp` and `/tunnel` stay Bearer-only as a CSRF gate.)

3. **RBAC** maps Cognito user groups to profiles. Each profile resolves to Apollo OAuth2 credentials stored in AWS Secrets Manager, scoping what the user can access.

4. **Teleport** access is provided by a `tbot` sidecar running alongside `cube-cloud` in the same ECS task. It uses IAM join to obtain short-lived certificates for Kubernetes clusters and app proxies — no `tsh` is needed on the developer's machine.

5. **Apollo** operations (environments, modules, entities, products, releases, change requests) go through a GraphQL API authenticated with per-profile OAuth2 client credentials.

6. **Groundcover** provides read-only, per-cube observability (metrics, logs, traces, K8s events, workload health) via the `groundcover_*` tools. Access is scoped to the cubes your Apollo credentials can see, and every query is pinned server-side to a single cube — see [docs/groundcover-integration-spec.md](docs/groundcover-integration-spec.md).

7. **Browser app proxy** gives account-less external users a browser-only path to in-cube web apps you've `expose`d — no Teleport account and no local agent. Requests hit an `<app>-<env>` wildcard subdomain, authenticate via a Cognito session cookie, and are dialed into the cube over the same tbot mTLS path. See [docs/browser-app-proxy-spec.md](docs/browser-app-proxy-spec.md).

8. **Console** is the browser UI at `https://<apex>/console` — sign in at the apex and you land on it. A React SPA (built from `console/web`, compiled by a node stage in the cube-cloud Dockerfile) over JSON endpoints that share the Apollo query + RBAC layer with the MCP tools. Cookie-authenticated, bookmarkable, and deliberately narrow: reads are GET-only and same-origin-enforced, and there is **no** generic tool-proxy endpoint. Each app card links the app's public browser-app-proxy URL (`https://<app>-<cube>.<apex>` — not Teleport's `…teleport.sh` address, which needs a Teleport account), and offers an inline **Unexpose** behind a confirmation that names the cube. Each app also links a **Logs** view at its own bookmarkable URL (`/console/cubes/<cube>/apps/<app>/logs`), backed by Groundcover — one query per Kubernetes workload the app owns, merged newest-first, and **near-live**: it polls every 10s incrementally (only rows newer than the last window), pauses when the tab is hidden, and offers a Live/Paused toggle. Streaming was deliberately not used — Groundcover exposes only search endpoints, so a WebSocket would still poll underneath while adding a second auth path. Each app row also offers **Expose**, which lists the cube's Kubernetes Services as candidates (auto-selecting the port when a Service has exactly one, and requiring a choice otherwise — a wrong port surfaces later as a Teleport timeout) and refuses any target the console did not itself enumerate. Expose and unexpose are the console's **only** mutations: it is its own route with enumerated args, gated on origin + a `X-Console-CSRF` synchronizer token, and it fails closed where the CSRF secret is unprovisioned — which is the case in prod today. Gated per-env by `console_enabled` (**dev only** today; off in prod, and off by default) — that flag takes down both the routes and the apex cookie allowlist together. See [docs/console-spec.md](docs/console-spec.md).

## Install

Requires [uv](https://docs.astral.sh/uv/) (or [pipx](https://pipx.pypa.io/)).

```bash
# Add to Claude Code
claude mcp add cube -- npx @edgescaleai/cube-mcp
```

Or run directly:

```bash
npx @edgescaleai/cube-mcp    # via Node (calls uvx under the hood)
uvx edgescaleai-cube-mcp     # via uv directly
```

## Getting Started

```
You: "Log me in"             → agent_login_browser (opens browser for Cognito login)
You: "Show Cube status"      → cube_status
You: "What pods are failing?" → kubectl_exec (runs kubectl server-side)
```

## Tools

### Auth

| Tool | Description |
|------|-------------|
| `agent_login_browser` | Log in via browser (Cognito) |
| `agent_logout` | Remove stored API key |
| `agent_status` | Check auth and connectivity |

### Kubernetes

| Tool | Description |
|------|-------------|
| `cube_list` | List available Cube clusters |
| `cube_status` | Get node status for a cluster |
| `kubectl_exec` | Run kubectl commands (server-side) |
| `list_services` | List Kubernetes Services in a cube's cluster, with app_uri suggestions for `expose` |

### Apps

| Tool | Description |
|------|-------------|
| `app_list` | List Teleport apps |
| `app_proxy` | Start local proxy tunnel to an app |
| `app_proxy_stop` | Stop running proxies |
| `app_proxy_status` | Show proxy status |

### App Exposure (Teleport)

| Tool | Description |
|------|-------------|
| `expose` | Expose a Kubernetes Service as an account-less Teleport app |
| `unexpose` | Remove an exposed app, or every app you can manage with `all=true` |
| `list_exposed` | List apps currently exposed on an environment (scoped to what you can manage) |

### Build & Registry

| Tool | Description |
|------|-------------|
| `build_and_publish_to_apollo` | Build image, package chart, push to ACR, and publish the manifest. Supports `build_args`, `dockerfile`, and `target` (multi-stage) for the image build; `platform` (defaults to `linux/amd64`, the cube architecture); and `build_mode` (`auto` uses local `docker buildx` if a daemon is present, else remote cube-cloud/kaniko — `local`/`remote` force one path). `show_build_log=true` surfaces the remote kaniko log tail on success. A failed image build aborts before publishing. |
| `acr_get_token` | Get Apollo Container Registry token |

> Remote (Docker-free) builds are driven server-side by three internal proxied tools — `build_image_prepare`, `build_image_start`, `build_image_status` — which `build_and_publish_to_apollo` orchestrates automatically. They are plumbing for the async submit→poll kaniko flow, not meant for direct invocation. See [docs/remote-build-spec.md](docs/remote-build-spec.md).
>
> The pure-Python chart packager does not fetch chart dependencies — vendor any subcharts (e.g. `helm dependency build`) into `charts/` before publishing, or Apollo rendering will fail on the missing dependency.

### Apollo Environments

| Tool | Description |
|------|-------------|
| `list_environments` | List/search Apollo environments |
| `install_entity` | Install a Helm chart entity |
| `uninstall_entity` | Uninstall an entity |
| `entity_health` | Get entity health and activity status |
| `plan_details` | Get plan tasks, events, and error logs |
| `cancel_active_plan` | Cancel (interrupt) the currently active plan for an entity |
| `update_entity_config` | Update entity config overrides |
| `enforce_entity_config` | Force re-apply entity configuration |

### Apollo Modules

| Tool | Description |
|------|-------------|
| `install_module` | Install a module on an environment |
| `uninstall_module` | Uninstall a module |
| `list_modules` | List modules in an environment |
| `update_module_variables` | Update module variables |

### Apollo Secrets

| Tool | Description |
|------|-------------|
| `create_secret` | Create a secret on an environment |
| `update_secret` | Update a secret value |

### Apollo Products & Release Channels

| Tool | Description |
|------|-------------|
| `list_products` | List available products |
| `compare_product_versions` | Compare versions of a product |
| `list_release_channels` | List release channels |
| `get_product_releases` | Get releases for a product |

### Groundcover (observability)

Read-only, per-cube observability. Access is scoped to the cubes your Apollo credentials can see, and every query is pinned server-side to a single cube — you cannot read another cube's data. See [docs/groundcover-integration-spec.md](docs/groundcover-integration-spec.md).

| Tool | Description |
|------|-------------|
| `groundcover_list_clusters` | List the cubes you're authorized to read that are reporting to Groundcover (start here) |
| `groundcover_list_workloads` | List workloads and their health (cpu, memory, rps, error rate, readiness, issues) for one cube |
| `groundcover_query_metrics` | Query metrics for one cube (PromQL built server-side, cluster-pinned); instant or range |
| `groundcover_list_metrics` | Discover metric names for one cube — families map when unfiltered, matching names when `filter` is given |
| `groundcover_search_logs` | Search logs for one cube (Groundcover log-search syntax) |
| `groundcover_search_traces` | Search distributed traces for one cube |
| `groundcover_k8s_events` | Search Kubernetes events (crashes, OOMKills, scheduling) for one cube |

## Local Development

```bash
# Install dependencies
uv sync --extra dev
uv pip install -e packages/cube-common -e packages/cube-cloud -e packages/cube-agent

# Run tests
uv run pytest packages/ -v

# Run cube-agent locally (for debugging)
uv run cube-agent
```

To test with Claude Code, point the MCP server at your local code:

```bash
claude mcp add cube-local -- uv run --directory /path/to/cube-mcp cube-agent
```

Reload after changes with `/mcp` in Claude Code.

`claude mcp add`'s `local` scope is keyed per project path, so if you work across multiple `git worktree` checkouts of this repo, each one needs its own `cube-local` entry (or they collide on the name). [`.claude/cube-local.mcp.json`](.claude/cube-local.mcp.json) avoids that by using `--directory .` instead of a hardcoded path, so the same file works unmodified from any checkout:

```bash
cd /path/to/cube-mcp  # or any worktree of it
claude --strict-mcp-config --mcp-config .claude/cube-local.mcp.json
```

`--strict-mcp-config` loads only the servers in that file, dropping any other MCP servers configured for the session — pass additional `--mcp-config <file>` arguments if you need those alongside it.

## Contributing

cube-mcp uses a deploy-on-merge model: **`main` is the release branch and deploys to `prod`; `dev-main` is a shared testing branch that deploys to the `dev` environment.** New work is built on a branch cut from `main`, validated on the `dev` stack, and merged to `main` only once it works — feature PRs target `main`, not `dev-main`.

### Branch and promotion flow

| Branch | Cut from | Deploys to | Role |
|--------|----------|------------|------|
| `feature/<desc>` / `fix/<desc>` | `main` | `dev`, by pushing to `dev-main` | the unit of work; its PR targets `main` |
| `dev-main` | long-lived | `dev` (`deploy-dev.yml`) | shared validation stack; a promotion path only when it holds a single feature |
| `main` | long-lived | `prod` (`deploy.yml` + `publish.yml`) | release branch — every merge ships |

**1. Branch from `main`.** Cut your branch off the latest `main` and name it `feature/<desc>` or `fix/<desc>`:

```bash
git fetch origin
git switch -c feature/my-change origin/main
```

**2. Open a PR into `main`.** Make your changes, run `uv run pytest packages/ -v`, then open a PR **targeting `main`** (not `dev-main`). It should get a review and green CI before merging.

**3. Validate on the `dev` stack first.** Before merging, push your branch's changes to `dev-main` to deploy them to the `dev` environment (`cube-mcp-dev-*`) via `deploy-dev.yml`, and exercise the change end-to-end against real infra. `dev-main` is a testing ground — catch deploy- and infra-level problems here, not in `prod`. The feature PR still targets `main`.

**4. Merge to `main` once verified.** Only after the change works on `dev`, merge the `main`-targeted PR. Merging deploys to `prod` (`deploy.yml`) and, for publish-path changes, auto-publishes to PyPI + npm (`publish.yml`).

**Promoting `dev-main` itself.** Opening a `dev-main` → `main` promotion PR is acceptable **only** when `dev-main` holds exactly one unmerged feature; with more than one it bundles unverified changes into a single prod merge, so land those through their own `main`-targeted PRs instead. Because `dev-main` is a shared, disposable testing branch, it can be reset to `main` when it drifts or accumulates stale test commits — coordinate with anyone else testing on `dev` before doing so.

## Operations & Ownership

This section exists so anyone inheriting cube-mcp can run it end-to-end without tribal knowledge. If something here is wrong or missing, fix it in this README — don't keep the truth in your head.

### Runbooks (point Claude Code at these)

| Doc | Use it when |
|-----|-------------|
| [docs/admin.md](docs/admin.md) | Day-to-day RBAC, profiles, API keys, Teleport, infra layout, troubleshooting. The single most important file in this repo. |
| [docs/adding-new-user.md](docs/adding-new-user.md) | Onboarding a new user (Cognito create + group assignment + verification). |
| [docs/rbac-architecture.md](docs/rbac-architecture.md) | Deeper architectural background on the RBAC model. |
| [resources/teleport.md](resources/teleport.md) | User-facing Teleport quickstart (`tsh login`, `tsh kube ls`, etc.). |
| [docs/groundcover-integration-spec.md](docs/groundcover-integration-spec.md) | How the read-only, per-cube Groundcover observability tools are wired and tenancy-scoped. |
| [docs/browser-app-proxy-spec.md](docs/browser-app-proxy-spec.md) | Account-less browser access to `expose`d in-cube web apps (subdomains, cookie auth, WebSocket proxying). |
| [docs/console-spec.md](docs/console-spec.md) | The browser UI at `/console` — where it lives, how cookie-auth is gated on the apex, the read/write CSRF contract, and the phase plan. Read §4.2 before touching `auth/middleware.py`. |
| [docs/remote-build-spec.md](docs/remote-build-spec.md) | Docker-free remote (kaniko) image builds — the server-side build path for `build_and_publish_to_apollo`. |
| [docs/tool-call-logging-spec.md](docs/tool-call-logging-spec.md) | Tool-call analytics telemetry (S3 Parquet + Groundcover OTLP spans). |

These docs are written in runbook style — open Claude Code in this repo and ask it to "follow `docs/adding-new-user.md` to add `alice@example.com` to the `lear-dev` group" and it will execute the AWS CLI calls itself.

### Where AWS resources live

Everything is in **AWS account `992382448282`**, region **`us-west-2`**. All resources are managed by Terraform in `infra/terraform/` — change them there, not in the console.

| Resource | Name / ARN suffix | Terraform file |
|----------|-------------------|----------------|
| Cognito user pool | `cube-mcp-prod-*` (look up with `aws cognito-idp list-user-pools`) | `infra/terraform/cognito.tf` |
| Cognito groups | validated: `admin`, `lear-sandbox`, `pltr-perception-dev`, `operations`, `strategists`; pending validation: `lear-dev`, `conagra-dev`, `pltr-dev`, `edgescaleai-dev` | base in `infra/terraform/cognito.tf`; script-provisioned profiles created live by `scripts/provision-access.py` |
| API keys table (DynamoDB) | `cube-mcp-prod-api-keys` | `infra/terraform/dynamodb.tf` |
| Profile credentials (Secrets Manager) | `cube-mcp/profiles/<profile>` | `infra/terraform/secrets.tf` |
| tbot config (Secrets Manager) | `cube-mcp/tbot-config` | `infra/terraform/tbot.tf` |
| Groundcover API key (Secrets Manager) | `cube-mcp/groundcover/api-key` (value: `{"api_key":"…","backend_id":"groundcover"}`; populated out-of-band, ECS task role granted read on `cube-mcp/groundcover/*`) | `infra/terraform/secrets.tf`, `iam.tf` |
| Tool-call logs (S3) | `cube-mcp-prod-tool-call-logs` (per-env; `cube-mcp-dev-tool-call-logs` is `force_destroy`, prod is not) | `infra/terraform/telemetry.tf` |
| Compaction Lambda + EventBridge | `cube-mcp-prod-tool-call-compaction` (nightly DuckDB rollup; zip built in CI from `infra/lambda/tool_call_compaction/build.sh`) | `infra/terraform/telemetry.tf` |
| OTLP secret (Secrets Manager) | `cube-mcp/telemetry/otlp` (Groundcover OTLP endpoint/key; populated out-of-band, ECS task role granted read) | `infra/terraform/telemetry.tf` |
| ECS task role | `cube-mcp-prod-ecs-task` | `infra/terraform/iam.tf` |
| ECS cluster | `cube-mcp-prod-cluster` (service: `cube-mcp-prod-service`) | `infra/terraform/ecs.tf` |
| CloudWatch Logs | `/ecs/cube-mcp-prod` (cube-cloud + tbot containers) | `infra/terraform/ecs.tf` |
| ALB + HTTPS listener | `cube.edgescaleai-cube.com` (apex cert also carries a `*.<domain>` SAN for browser app-proxy subdomains) | `infra/terraform/alb.tf` |
| Route53 records | `edgescaleai-cube.com` zone (`Z0571857327BEMX2EHNZU`) — **non-authoritative** (public DNS is served by Cloudflare, see below) | `infra/terraform/route53.tf` |
| ECR (cube-cloud + tbot images) | `cube-mcp/cube-cloud`, `cube-mcp/tbot` | `infra/terraform/ecr.tf`, `tbot.tf` |
| SES sender | `noreply@edgescaleai-cube.com` (identity `edgescaleai-cube.com`) | `infra/terraform/cognito.tf` |

Three telemetry flags gate the tool-call analytics stack — `telemetry_enabled` (S3 sink), `telemetry_compaction_enabled` (nightly Lambda), and `otlp_enabled` (Groundcover spans). All default off and are set per-env in `infra/terraform/environments/*.tfvars`; all three are `true` in prod.

**Outside AWS:**

- **Teleport:** `edgescaleai.teleport.sh` — bot is `cube-mcp-bot`, joins via IAM. See [docs/admin.md §4](docs/admin.md#4-teleport-access--rbac).
- **Apollo:** `edgescaleai.palantirapollo.com` — OAuth2 credentials per profile, stored in Secrets Manager.
- **Groundcover:** `api.groundcover.com` — one workspace/tenant for the whole fleet (each cube is a `cluster`); we enforce per-cube tenancy ourselves. Key in Secrets Manager (`cube-mcp/groundcover/api-key`).
- **Cloudflare (DNS):** the public domains (`cube.edgescaleai-cube.com` / `cube-dev.edgescaleai-cube.com`) are DNS-only CNAMEs → the ALB, **managed manually in Cloudflare, not Terraform** (the `aws_route53_record`s are no-ops). The browser app proxy additionally needs a **wildcard** `*.cube.edgescaleai-cube.com` DNS-only CNAME → the prod ALB; until it's added the feature ships **dormant** (subdomains won't resolve). See [docs/browser-app-proxy-spec.md](docs/browser-app-proxy-spec.md) Appendix B.
- **PyPI / npm:** auto-published from `main` via CI (GitHub Actions). The npm shim under `npm/` is a thin wrapper that shells to `uvx edgescaleai-cube-mcp`.

### CI/CD pipelines

All deploys go through GitHub Actions in `.github/workflows/`. AWS auth uses OIDC — no long-lived AWS keys are stored in GitHub.

| Workflow | Trigger | What it does |
|----------|---------|--------------|
| `test.yml` | Reusable, called by others | Runs `pytest packages/ -v`. |
| `deploy.yml` | Push to `main` (paths: `packages/cube-cloud`, `cube-common`, `src/`, `infra/`) | Runs tests → builds the tool-call compaction Lambda zip (`infra/lambda/tool_call_compaction/build.sh`) so `terraform apply` sees its `source_code_hash` → `terraform apply` (only if `infra/terraform/` changed) → builds & pushes `cube-mcp/cube-cloud:latest` and `cube-mcp/tbot:latest` to ECR → seeds new profile secrets → seeds `infra/tbot/tbot-config-prod.yaml` (base-only) into Secrets Manager **only if absent** (never overwrites; per-profile identities are written directly by `scripts/provision-access.py`) → `aws ecs update-service --force-new-deployment` and waits for `services-stable`. Also posts `terraform plan` as a PR comment for terraform PRs. |
| `deploy-dev.yml` | Push to `dev-main` branch | Same shape as `deploy.yml`, but targets `cube-mcp-dev-cluster` / `cube-mcp-dev-service` and uses TF state key `cube-cloud-dev/terraform.tfstate`. Images tagged `:dev`. Use this branch to test infra changes before merging to `main`. |
| `publish.yml` | Push to `main` (paths: `src/`, `npm/`, `pyproject.toml`) | Runs tests → bumps the patch version in `pyproject.toml` and `npm/package.json` → commits as `Release vX.Y.Z [skip ci]` and tags `vX.Y.Z` → publishes to PyPI (`twine`) and npm (`npm publish --access public`). |
| `sync-knowledge.yml` | Scheduled / manual | Syncs internal Claude Code knowledge from the `disco-projects` repo. Not on the critical deploy path — safe to ignore unless it breaks. |

**OIDC role** for all AWS-touching jobs: `arn:aws:iam::992382448282:role/github-actions`.

### Required CI secrets

Stored as GitHub repo secrets. To rotate any of these, generate a new value at the source and update via `gh secret set <NAME>` (or the repo Settings → Secrets UI).

| Secret | Where used | Source / how to rotate |
|--------|-----------|------------------------|
| `PYPI_TOKEN` | `publish.yml` | [pypi.org/manage/account/token/](https://pypi.org/manage/account/token/) — scope to the `cube-mcp` project. |
| `NPM_TOKEN` | `publish.yml` | npm account → Access Tokens → Automation token (must allow publish). |
| `PUBLISH_DEPLOY_KEY` | `publish.yml` | SSH deploy key on this repo with **write** access (used to push the auto-bumped version commit + tag back to `main`). Regenerate: create a new SSH keypair, add the public key as a repo Deploy Key with write access, paste the private key into the secret. |
| `DISCO_PROJECTS_TOKEN` | `sync-knowledge.yml` | Fine-grained PAT with read access to `EdgescaleAI/disco-projects`. |
| `ANTHROPIC_API_KEY` | `sync-knowledge.yml` | console.anthropic.com — used by the disco knowledge sync job. |

AWS credentials are **not** stored as secrets — `deploy.yml`/`deploy-dev.yml` assume `arn:aws:iam::992382448282:role/github-actions` via OIDC. To grant a new repo or change permissions, update that role's trust policy and inline policies in AWS IAM.

### Terraform state

Stored in S3, no DynamoDB lock table currently configured (single-writer assumption — CI is the only applier).

| Env | Bucket | Key |
|-----|--------|-----|
| Prod | `cube-mcp-terraform-state` (`us-west-2`) | `cube-cloud/terraform.tfstate` |
| Dev | `cube-mcp-terraform-state` (`us-west-2`) | `cube-cloud-dev/terraform.tfstate` |

**Local apply** (only if you really need to bypass CI):

```bash
cd infra/terraform
terraform init                                                    # prod
terraform init -backend-config="key=cube-cloud-dev/terraform.tfstate"  # dev
terraform plan
```

The S3 bucket has versioning enabled, so a corrupt state file can be rolled back to a previous version via the AWS console.

### Dev environment

A parallel stack exists in the same AWS account for testing infra changes before merging to `main`.

| | Prod | Dev |
|---|-----|-----|
| Branch | `main` | `dev-main` |
| ECS cluster | `cube-mcp-prod-cluster` | `cube-mcp-dev-cluster` |
| ECS service | `cube-mcp-prod-service` | `cube-mcp-dev-service` |
| Image tag | `:latest` | `:dev` |
| TF state key | `cube-cloud/terraform.tfstate` | `cube-cloud-dev/terraform.tfstate` |
| TF var file | (defaults) | `infra/terraform/environments/dev.tfvars` |

`dev-main` is a **testing environment first**: push your branch's changes there to deploy them to the dev stack and validate before they land on `main`. Feature PRs target `main` directly and are merged only after the change has been verified on the dev stack. Promoting `dev-main` → `main` as a single PR is acceptable **only** when `dev-main` holds exactly one unmerged feature; with more than one it would bundle unverified changes into a single prod merge, so promote those individually via their own `main`-targeted PRs. See [Contributing](#contributing) for the full branch-and-promotion flow.

### Versioning & releases

- **Scheme:** patch-bump only, automated. `publish.yml` reads `pyproject.toml`, increments the patch, writes it back to both `pyproject.toml` and `npm/package.json`, commits as `github-actions[bot]`, and tags `vX.Y.Z`.
- **Major/minor bumps:** edit `pyproject.toml` manually in a PR. The next merge to `main` that touches `src/` / `npm/` / `pyproject.toml` will publish from there.
- **PyPI ↔ npm sync:** the same version number is used for both packages. The `npm/` shim is a thin wrapper that shells to `uvx edgescaleai-cube-mcp`.
- **What "main" means for users:** every merge that touches the publish paths ships to PyPI + npm within minutes. There is no staging release.

### Rollback playbook

| Surface | How to roll back |
|---------|------------------|
| **ECS deploy (cube-cloud or tbot)** | Images are tagged `:latest` only — there is no previous-image tag to roll forward to. Rollback path: `git revert` the offending commit on `main`; the next `deploy.yml` run will rebuild and redeploy. For an emergency, manually re-tag a known-good ECR image as `:latest` (`aws ecr batch-get-image` → `put-image`) and `aws ecs update-service --force-new-deployment`. **Improvement worth making:** tag images with the git SHA so rollback is one CLI call. |
| **Terraform** | `git revert` the offending commit; `deploy.yml` re-runs `terraform apply` on next push. For state corruption, restore from S3 versioning on `cube-mcp-terraform-state`. |
| **PyPI / npm** | Cannot un-publish (yanking PyPI is reversible only in narrow windows; npm allows unpublish within 72h). The standard path is to push a new patch version with the fix. Users on `npx @edgescaleai/cube-mcp` and `uvx edgescaleai-cube-mcp` pick up the new version on next invocation. |
| **Cognito user / group change** | Cognito has no native rollback. Reverse the change manually: re-add the user, restore group membership. Audit trail is in CloudTrail (`cube-mcp-prod-trail` if enabled — verify). |
| **Secrets Manager profile credentials** | Each secret has versioning. Roll back with `aws secretsmanager update-secret-version-stage --secret-id cube-mcp/profiles/<profile> --version-stage AWSCURRENT --move-to <prev_version_id>`. |

### Monitoring & alerting

> **Alerting: not yet configured.** No CloudWatch alarms, SNS topics, Sentry, or paging are set up. Nothing pages anyone when the service degrades — that is still the biggest gap.
>
> **Observability that *does* exist:** CloudWatch Logs (`/ecs/cube-mcp-prod`) + ECS task health for the service itself; **tool-call analytics telemetry** (live in prod) recording every cloud tool call to two sinks — S3 Parquet (durable analytics, queryable with the DuckDB query pack in [docs/tool-call-analytics-queries.md](docs/tool-call-analytics-queries.md)) and Groundcover OTLP spans (hot/correlation tier). See [docs/tool-call-logging-spec.md](docs/tool-call-logging-spec.md). Per-cube observability (of the fleet, not this service) is available via the `groundcover_*` tools.

Day-to-day debugging:

```bash
# Live tail prod logs (cube-cloud + tbot)
aws logs tail /ecs/cube-mcp-prod --follow --region us-west-2

# Filter to just tbot
aws logs tail /ecs/cube-mcp-prod --filter-pattern "tbot" --follow

# Service health
aws ecs describe-services \
  --cluster cube-mcp-prod-cluster \
  --services cube-mcp-prod-service \
  --query 'services[0].{running:runningCount,desired:desiredCount,events:events[0:5]}'
```

**Recommended additions** for the next owner (none of these exist yet):
- CloudWatch alarm on ECS service `RunningTaskCount < 1` → SNS → email/PagerDuty.
- ALB 5xx alarm on the listener.
- DynamoDB throttling alarm on `cube-mcp-prod-api-keys`.
- Synthetic check that runs `agent_login_browser` end-to-end weekly.

### Stakeholders & contacts

> `TODO(owner)`: Vinayak (departing) is the only person with full context on these. Fill these in before he leaves so the next owner has someone to call.

| Role | Person | Contact |
|------|--------|---------|
| Internal product owner | `TODO` | |
| Eng escalation / on-call | `TODO` | |
| AWS account admin | `TODO` | |
| Teleport admin (can grant `tctl` access) | `TODO` | |
| Apollo admin (can issue per-tenant OAuth2 creds) | `TODO` | |
| Lear tenant POC | `TODO` | |
| Conagra tenant POC | `TODO` | |
| Palantir tenant POC | `TODO` | |
| EdgescaleAI internal dev (`edgescaleai-dev` profile) | `TODO` | |

### Cost & billing

> `TODO(owner)`: Add rough monthly run-rate and the account that gets billed.

- **AWS** (account `992382448282`): `TODO` — typical monthly spend, biggest line items, who pays the invoice.
- **Teleport** (`edgescaleai.teleport.sh`): `TODO` — plan tier, seat count, billing contact.
- **Apollo** (`edgescaleai.palantirapollo.com`): `TODO` — Palantir contract reference, who renews.
- **Domain** (`edgescaleai-cube.com`): `TODO` — registrar, expiry date, who has the login.
- **PyPI / npm**: free tiers, no recurring cost.

If costs need to be cut quickly: the dev ECS service (`cube-mcp-dev-service`) can be scaled to zero with no user impact.

### Access & disaster recovery

| Surface | Status / what the next owner should verify |
|---------|--------------------------------------------|
| GitHub repo admins | `TODO(owner)`: list everyone with admin/maintain on `EdgescaleAI/cube-mcp` and confirm branch protection on `main` (require PR + passing tests). |
| AWS root account access | `TODO(owner)`: confirm who holds the root credentials for account `992382448282` and that MFA is enforced. |
| Domain registrar login | `TODO(owner)`: who has the registrar account for `edgescaleai-cube.com`. |
| DynamoDB `cube-mcp-prod-api-keys` | Point-in-time recovery enabled (35-day window). |
| Secrets Manager | Per-secret versioning enabled by default — see Rollback playbook. |
| Terraform state | S3 bucket versioning enabled on `cube-mcp-terraform-state`. |
| ECR images | Lifecycle rules are scoped to **untagged** images only (retain 10 cube-cloud / 5 tbot untagged) — tagged images (`:latest`/`:dev`) are retained indefinitely. This was tightened after a `tagStatus: any` rule reaped the running prod `:latest` image and caused an ALB 503 outage. See `infra/terraform/ecr.tf` / `tbot.tf`. |

### Known tech debt / planned work

> Update this list as you ship or de-scope items so the next owner sees current state, not folklore.

- **Phase 2 RBAC:** per-profile Apollo RBAC is now implemented and validated — each scoped profile has its own Apollo SA + team + creds in Secrets Manager (validated end-to-end through MCP: `lear-sandbox`, `pltr-perception-dev`, `operations`). Remaining work: set up and validate `lear-dev` / `conagra-dev` / `pltr-dev` / `edgescaleai-dev` (real Teleport roles) through the MCP pipeline. Provisioning is done by `scripts/provision-access.py`.
- **ECR `:latest`-only tagging** makes ECS rollback awkward (see Rollback playbook). Tag images with the git SHA in `deploy.yml`.
- **No alerting.** See Monitoring section.
- **Teleport role management is manual `tctl`.** The future plan is the Teleport Terraform provider — see [docs/admin.md → Future](docs/admin.md#future-teleport-terraform-provider).
- **No DynamoDB lock on Terraform state.** Acceptable while CI is the only applier; add a lock table if local applies become routine.
- **Cognito drift: `esaiadmin` group.** Surfaced in [#93](https://github.com/EdgescaleAI/cube-mcp/pull/93). The prod Cognito pool has an `esaiadmin` group that isn't defined in `infra/terraform/cognito.tf` — it was created in the console. Either `terraform import` it or delete it; don't leave it as drift.
- **`TODO(owner)`:** anything else in flight at handoff time.

### First-day checklist for a new owner

1. **Read [docs/admin.md](docs/admin.md) end-to-end.** Then run `agent_login_browser` yourself so you've experienced the user flow.
2. **Get console access** to AWS account `992382448282` (`us-west-2`) with permissions for Cognito, DynamoDB, Secrets Manager, ECS, IAM, and CloudWatch Logs.
3. **Get Teleport admin** (`tctl`) on `edgescaleai.teleport.sh` — needed to manage the `cube-mcp-bot` and add roles. Ask the person listed in [Stakeholders & contacts](#stakeholders--contacts).
4. **Get Apollo admin** on `edgescaleai.palantirapollo.com` — needed to issue per-profile OAuth2 credentials when adding new tenants.
5. **Install [`cube-admin`](packages/cube-admin)** locally for API key management: `uv pip install -e packages/cube-admin`. This package is **internal-only and must never be published**.
6. **Verify CI is healthy** and you can rotate every secret in [Required CI secrets](#required-ci-secrets). Push a no-op commit to `dev-main` to confirm `deploy-dev.yml` succeeds end-to-end against the dev stack.
7. **Replace every `TODO(owner)` in this README** — Stakeholders, Cost, Access & DR, Tech debt. If you can't fill one in, write down who you asked and what they said. Don't leave them blank.

### Common ops tasks

- **Provision access (recommended):** `uv run python scripts/provision-access.py --email <email> --profile <name> --cubes <id1,id2> [--env prod|dev] [--apply]` — full end-to-end provisioner (Apollo SA/team/roles + Teleport role/bot + tbot identity in Secrets Manager + Cognito group), dry-run by default, idempotent.
- **Look up access:** `uv run python scripts/user-access.py show <email> [--env prod|dev]` — prints the user's groups, resolved profile, and Apollo-accessible cubes (`groups` / `grant` subcommands too).
- **Add a user:** [docs/adding-new-user.md](docs/adding-new-user.md).
- **Add a new tenant / role:** [docs/admin.md → Creating a new role / profile](docs/admin.md#creating-a-new-role--profile). Prefer `scripts/provision-access.py`; the six-step manual flow (Cognito group, SM secret, Teleport role, bot role grant, tbot output, merge) is kept as break-glass.
- **Revoke access:** [docs/admin.md → Revoking access](docs/admin.md#revoking-access).
- **Manage API keys:** `cube-admin keys {create,list,revoke}` — see [docs/admin.md → Managing API keys with cube-admin](docs/admin.md#managing-api-keys-with-cube-admin).
- **Debug a failing tool call:** check `aws logs tail /ecs/cube-mcp-prod --follow` for the cube-cloud and tbot containers.

### Repo layout cheatsheet

```
packages/
  cube-agent/    # Local MCP server (npx @edgescaleai/cube-mcp entrypoint)
  cube-cloud/    # FastAPI service on ECS — auth, RBAC, Apollo, Teleport relay
  cube-common/   # Shared types and helpers
  cube-admin/    # Internal-only API key CLI — DO NOT PUBLISH
infra/
  terraform/     # All AWS resources
  tbot/          # tbot Dockerfile, entrypoint, prod config
docs/            # Runbooks (start here)
resources/       # User-facing reference docs
npm/             # npm shim that calls uvx edgescaleai-cube-mcp
scripts/         # Release / publish helpers
```
