# AI Matrx Context Engine — v2 Specification

> ⚠️ **HISTORICAL DESIGN PROPOSAL — do NOT read the DDL below as current schema.**
> Written before the 2026 schema reorg: the `public.cx_*` tables it shows now live in the
> `chat` schema, `workspace_id` was never built, the `project_id` FKs it proposes are now
> FORBIDDEN (project membership is a `platform.associations` edge), and the owner column is
> `created_by`, not `user_id`. Kept for the reasoning, not the shapes.

> **Corrections from v1:** `ai_runs`, `ai_tasks`, `broker_values` are dead. Prompts/prompt_builtins = agents.
> The real entity landscape is cx_conversation-centric. This document supersedes v1.

---

## 1. The Entity Landscape (What Actually Exists)

### The Hierarchy (Context Scoping Layers)

```
User (auth.users + profiles)
│
└── Organization
    │
    └── Workspace (nestable via parent_workspace_id)
        │
        └── Project
            │
            └── Task (nestable via parent_task_id)
```

### The 9 Scoped Entities (Things Users Work With)

| # | Entity | Table(s) | Description | Rows |
|---|--------|----------|-------------|------|
| 1 | **Conversations** | `cx_conversation`, `cx_message`, `cx_user_request`, `cx_request`, `cx_tool_call` | ALL AI interaction. The primary vessel. | 1,095 |
| 2 | **Artifacts** | `cx_artifact` ← **NEW** | Structured outputs from conversations (SVGs, HTML, code blocks, generated files). Attached to conversations but independently referenceable. | — |
| 3 | **Notes** | `notes`, `note_versions`, `note_folders` | User text, documentation, anything they want to store. | 265 |
| 4 | **Files** | `files.files` + S3 | Binary assets in the canonical file system. | 0 |
| 5 | **Data** | `udt_datasets`, `udt_dataset_fields`, `udt_dataset_rows` | Tabular/structured data. Dynamic user-defined schemas. | 105 tables, 2,319 rows |
| 6 | **Knowledge** | `rs_topic`, `rs_source`, `rs_content`, `rs_analysis`, `rs_synthesis`, `rs_document` | Research content, uploaded sources, embeddings. From automated research or user uploads (e.g., a biology PDF). | 305 sources, 162 content |
| 7 | **Transcripts** | `transcripts` | Audio/video → text with segments and timestamps. | 9 |
| 8 | **Code** | TBD | User git repos, code files. Not fully integrated yet. | — |
| 9 | **Environments** | `app_instances` (local), `sandbox_instances` (cloud) | Compute environments. app_instance = AI Matrx running on user's machine. sandbox = temporary cloud environment with persistence. | 3 + 4 |

### The Agents (What Processes Things)

**`prompts` + `prompt_builtins` = AI Agents.** These are custom AI agents with:
- Specific system prompts / message templates
- Assigned tools they can invoke
- Variable access (auto-injected OR tool-accessible)
- Model + settings configuration

---

## 2. Current Schema Gaps

### Gap 1: `projects.workspace_id` Does Not Exist
Projects connect to organizations but skip workspaces entirely. The Org → Workspace → Project chain is broken.

### Gap 2: No Scope FKs on Any Scoped Entity
These tables only have `user_id`:
- `cx_conversation` — no org/ws/project/task FKs
- `notes` — no org/ws/project/task FKs
- `udt_datasets` — no org/ws/project/task FKs
- `user_files` — no org/ws/project/task FKs
- `transcripts` — no org/ws/project/task FKs
- `workflow` — no org/ws/project/task FKs

`rs_topic` already has `project_id` ✓ and `sandbox_instances` already has `project_id` ✓

### Gap 3: No `workspace_members`
Org and project have membership tables. Workspaces don't.

### Gap 4: No `cx_artifact` Table
Conversation outputs (generated SVGs, code blocks, structured results) have no dedicated table. `cx_media` handles file uploads but not generated artifacts.

### Gap 5: No Active Context Tracker
Nothing records "User X is working in Org A → Workspace B → Project C → Task D."

### Gap 6: No Formal Context Variable System
No dedicated, cascading key/value store for auto-injection into agents.

### Gap 7: `canvas_items` Uses Text IDs
`canvas_items.task_id` and `canvas_items.source_message_id` are `text` not `uuid`. No real FKs.

---

## 3. The Dual Context Delivery Model

This is what makes this system different from anything else. Context isn't just "shove everything in the system prompt." There are two tiers:

### Tier 1: Direct Injection (In the Prompt)
Small, critical context that's always present in the agent's system prompt:
- Current scope (org/workspace/project/task names and metadata)
- Key variables (tech stack, coding standards, project description)
- Task details (title, description, status, assignee, due date)
- User preferences (response style, preferred model)

**Token budget: ~500-2,000 tokens.** Always present. No tool call needed.

### Tier 2: Tool-Accessible Context (On Demand)
Larger context the agent can pull when it needs it:
- **`search_notes`** — search notes within scope
- **`query_data`** — query user tables within scope
- **`search_knowledge`** — search research content/knowledge base
- **`get_transcripts`** — retrieve transcript content
- **`read_file`** — access files within scope
- **`list_conversations`** — browse related conversations
- **`get_artifacts`** — retrieve generated artifacts
- **`summarize_context`** — have a sub-agent summarize large content before using it
- **`get_code`** — access code files/repos
- **`run_in_sandbox`** — execute code in the user's environment

**Token budget: Unlimited (on demand).** The agent decides what to pull.

### How Variables Control This

Each `context_variable` has an `inject_as` field:

| `inject_as` | Behavior |
|--------------|----------|
| `direct` | Tier 1 — baked into system prompt |
| `tool_accessible` | Tier 2 — available via tool call, agent chooses when to access |
| `searchable` | Tier 2 — available through search tools |
| `metadata` | Stored but not injected — for frontend/API use only |

---

## 4. Migration SQL

### 4a. Fix the Hierarchy: `projects.workspace_id`

```sql
ALTER TABLE public.projects
ADD COLUMN workspace_id uuid REFERENCES public.workspaces(id) ON DELETE SET NULL;

CREATE INDEX idx_projects_workspace_id ON public.projects(workspace_id)
WHERE workspace_id IS NOT NULL;

COMMENT ON COLUMN public.projects.workspace_id IS
  'Optional workspace scope. NULL = project lives directly under the org.';
```

### 4b. Create `workspace_members`

```sql
CREATE TABLE public.workspace_members (
    id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id uuid NOT NULL REFERENCES public.workspaces(id) ON DELETE CASCADE,
    user_id      uuid NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
    role         text NOT NULL DEFAULT 'member'
                 CHECK (role IN ('owner', 'admin', 'member', 'viewer')),
    joined_at    timestamptz NOT NULL DEFAULT now(),
    invited_by   uuid REFERENCES auth.users(id),
    UNIQUE(workspace_id, user_id)
);

ALTER TABLE public.workspace_members ENABLE ROW LEVEL SECURITY;
CREATE INDEX idx_ws_members_user ON public.workspace_members(user_id);
CREATE INDEX idx_ws_members_ws ON public.workspace_members(workspace_id);
```

### 4c. Add Scope Columns to ALL Scoped Entities

Every scoped entity gets the same 4 nullable FKs. All nullable because an entity can live at any level (or be user-level only).

```sql
-- ═══════════════════════════════════════════════════
-- cx_conversation
-- ═══════════════════════════════════════════════════
ALTER TABLE public.cx_conversation
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

CREATE INDEX IF NOT EXISTS idx_cx_conv_scope
ON public.cx_conversation(organization_id, workspace_id, project_id, task_id)
WHERE organization_id IS NOT NULL OR workspace_id IS NOT NULL
   OR project_id IS NOT NULL OR task_id IS NOT NULL;

-- ═══════════════════════════════════════════════════
-- notes
-- ═══════════════════════════════════════════════════
ALTER TABLE public.notes
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

CREATE INDEX IF NOT EXISTS idx_notes_scope
ON public.notes(organization_id, workspace_id, project_id, task_id)
WHERE organization_id IS NOT NULL OR workspace_id IS NOT NULL
   OR project_id IS NOT NULL OR task_id IS NOT NULL;

-- ═══════════════════════════════════════════════════
-- udt_datasets (Data)
-- ═══════════════════════════════════════════════════
ALTER TABLE public.udt_datasets
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

-- ═══════════════════════════════════════════════════
-- user_files (Files)
-- ═══════════════════════════════════════════════════
ALTER TABLE public.user_files
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

-- ═══════════════════════════════════════════════════
-- transcripts
-- ═══════════════════════════════════════════════════
ALTER TABLE public.transcripts
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

-- ═══════════════════════════════════════════════════
-- workflow
-- ═══════════════════════════════════════════════════
ALTER TABLE public.workflow
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

-- ═══════════════════════════════════════════════════
-- prompts (agents)
-- ═══════════════════════════════════════════════════
ALTER TABLE public.prompts
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

-- ═══════════════════════════════════════════════════
-- sandbox_instances (already has project_id, add the rest)
-- ═══════════════════════════════════════════════════
ALTER TABLE public.sandbox_instances
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;

-- ═══════════════════════════════════════════════════
-- app_instances (local apps)
-- ═══════════════════════════════════════════════════
ALTER TABLE public.app_instances
ADD COLUMN IF NOT EXISTS organization_id uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS project_id      uuid REFERENCES public.projects(id) ON DELETE SET NULL,
ADD COLUMN IF NOT EXISTS task_id         uuid REFERENCES public.tasks(id) ON DELETE SET NULL;
```

### 4d. Create `cx_artifact`

```sql
CREATE TABLE public.cx_artifact (
    id                uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    conversation_id   uuid NOT NULL REFERENCES public.cx_conversation(id) ON DELETE CASCADE,
    message_id        uuid REFERENCES public.cx_message(id) ON DELETE SET NULL,
    user_id           uuid NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,

    -- What is this artifact?
    artifact_type     text NOT NULL
                      CHECK (artifact_type IN (
                          'code', 'svg', 'html', 'markdown', 'json',
                          'image', 'document', 'chart', 'diagram',
                          'table', 'file', 'component', 'other'
                      )),
    title             text NOT NULL DEFAULT '',
    description       text NOT NULL DEFAULT '',
    content           text NOT NULL DEFAULT '',
    content_json      jsonb NOT NULL DEFAULT '{}'::jsonb,
    language          text NOT NULL DEFAULT '',           -- for code: python, typescript, sql, etc.
    mime_type         text NOT NULL DEFAULT '',

    -- If stored as a file
    file_path         text NOT NULL DEFAULT '',
    file_size_bytes   bigint NOT NULL DEFAULT 0,

    -- Scope (inherited from conversation or set independently)
    organization_id   uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
    workspace_id      uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
    project_id        uuid REFERENCES public.projects(id) ON DELETE SET NULL,
    task_id           uuid REFERENCES public.tasks(id) ON DELETE SET NULL,

    -- Metadata
    tags              text[] NOT NULL DEFAULT '{}',
    is_pinned         boolean NOT NULL DEFAULT false,
    is_archived       boolean NOT NULL DEFAULT false,
    version           integer NOT NULL DEFAULT 1,
    parent_artifact_id uuid REFERENCES public.cx_artifact(id) ON DELETE SET NULL,

    created_at        timestamptz NOT NULL DEFAULT now(),
    updated_at        timestamptz NOT NULL DEFAULT now(),
    deleted_at        timestamptz
);

ALTER TABLE public.cx_artifact ENABLE ROW LEVEL SECURITY;

CREATE INDEX idx_cx_artifact_conv ON public.cx_artifact(conversation_id);
CREATE INDEX idx_cx_artifact_user ON public.cx_artifact(user_id);
CREATE INDEX idx_cx_artifact_type ON public.cx_artifact(artifact_type);
CREATE INDEX idx_cx_artifact_scope
ON public.cx_artifact(organization_id, workspace_id, project_id, task_id)
WHERE organization_id IS NOT NULL OR workspace_id IS NOT NULL
   OR project_id IS NOT NULL OR task_id IS NOT NULL;

COMMENT ON TABLE public.cx_artifact IS
  'Structured outputs from AI conversations. SVGs, code blocks, HTML, generated docs.
   Attached to conversations but independently queryable and scopeable.';
```

### 4e. Add `settings` to `tasks` (if missing)

```sql
ALTER TABLE public.tasks
ADD COLUMN IF NOT EXISTS settings jsonb NOT NULL DEFAULT '{}'::jsonb;
```

---

## 5. The Context Variable System

### 5a. `context_variables` — The Cascading KV Store

```sql
CREATE TABLE public.context_variables (
    id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),

    -- SCOPE: exactly one of these is non-null
    user_id         uuid REFERENCES auth.users(id) ON DELETE CASCADE,
    organization_id uuid REFERENCES public.organizations(id) ON DELETE CASCADE,
    workspace_id    uuid REFERENCES public.workspaces(id) ON DELETE CASCADE,
    project_id      uuid REFERENCES public.projects(id) ON DELETE CASCADE,
    task_id         uuid REFERENCES public.tasks(id) ON DELETE CASCADE,

    -- THE VARIABLE
    key             text NOT NULL,
    value           jsonb NOT NULL,
    value_type      text NOT NULL DEFAULT 'string'
                    CHECK (value_type IN (
                        'string', 'number', 'boolean', 'json',
                        'secret', 'file_ref', 'model_ref',
                        'prompt_ref', 'tool_ref', 'template'
                    )),

    -- HOW IT'S DELIVERED TO THE AGENT
    inject_as       text NOT NULL DEFAULT 'direct'
                    CHECK (inject_as IN (
                        'direct',           -- Tier 1: baked into system prompt
                        'tool_accessible',  -- Tier 2: agent can pull via tool
                        'searchable',       -- Tier 2: available via search tools
                        'metadata'          -- Not injected, frontend/API only
                    )),

    -- METADATA
    description     text NOT NULL DEFAULT '',
    is_system       boolean NOT NULL DEFAULT false,  -- auto-populated by platform
    is_secret       boolean NOT NULL DEFAULT false,  -- never sent to client
    is_active       boolean NOT NULL DEFAULT true,
    tags            text[] NOT NULL DEFAULT '{}',

    created_at      timestamptz NOT NULL DEFAULT now(),
    updated_at      timestamptz NOT NULL DEFAULT now(),
    created_by      uuid REFERENCES auth.users(id),

    -- CONSTRAINTS
    CONSTRAINT exactly_one_scope CHECK (
        (CASE WHEN user_id IS NOT NULL THEN 1 ELSE 0 END +
         CASE WHEN organization_id IS NOT NULL THEN 1 ELSE 0 END +
         CASE WHEN workspace_id IS NOT NULL THEN 1 ELSE 0 END +
         CASE WHEN project_id IS NOT NULL THEN 1 ELSE 0 END +
         CASE WHEN task_id IS NOT NULL THEN 1 ELSE 0 END) = 1
    ),

    CONSTRAINT unique_key_per_user UNIQUE (user_id, key),
    CONSTRAINT unique_key_per_org UNIQUE (organization_id, key),
    CONSTRAINT unique_key_per_ws UNIQUE (workspace_id, key),
    CONSTRAINT unique_key_per_proj UNIQUE (project_id, key),
    CONSTRAINT unique_key_per_task UNIQUE (task_id, key)
);

ALTER TABLE public.context_variables ENABLE ROW LEVEL SECURITY;

CREATE INDEX idx_ctx_var_user ON public.context_variables(user_id) WHERE user_id IS NOT NULL;
CREATE INDEX idx_ctx_var_org ON public.context_variables(organization_id) WHERE organization_id IS NOT NULL;
CREATE INDEX idx_ctx_var_ws ON public.context_variables(workspace_id) WHERE workspace_id IS NOT NULL;
CREATE INDEX idx_ctx_var_proj ON public.context_variables(project_id) WHERE project_id IS NOT NULL;
CREATE INDEX idx_ctx_var_task ON public.context_variables(task_id) WHERE task_id IS NOT NULL;
CREATE INDEX idx_ctx_var_key ON public.context_variables(key);
CREATE INDEX idx_ctx_var_inject ON public.context_variables(inject_as) WHERE is_active = true;

COMMENT ON TABLE public.context_variables IS
  'Cascading key/value store. Variables defined at any hierarchy layer override
   the same key from broader layers. Task > Project > Workspace > Org > User.';
```

### 5b. `user_active_context` — Where Is the User Now?

```sql
CREATE TABLE public.user_active_context (
    user_id          uuid PRIMARY KEY REFERENCES auth.users(id) ON DELETE CASCADE,

    -- Current position in the hierarchy
    organization_id  uuid REFERENCES public.organizations(id) ON DELETE SET NULL,
    workspace_id     uuid REFERENCES public.workspaces(id) ON DELETE SET NULL,
    project_id       uuid REFERENCES public.projects(id) ON DELETE SET NULL,
    task_id          uuid REFERENCES public.tasks(id) ON DELETE SET NULL,

    -- What entity are they actively working with?
    active_entity_type  text,    -- 'cx_conversation', 'note', 'workflow', 'data', etc.
    active_entity_id    uuid,

    -- Session info
    app_source       text NOT NULL DEFAULT 'web'
                     CHECK (app_source IN ('web', 'mobile', 'desktop', 'extension', 'api')),
    last_activity    timestamptz NOT NULL DEFAULT now(),
    updated_at       timestamptz NOT NULL DEFAULT now()
);

ALTER TABLE public.user_active_context ENABLE ROW LEVEL SECURITY;

COMMENT ON TABLE public.user_active_context IS
  'Single row per user. Updated on navigation. Agents read this to auto-resolve context.';
```

---

## 6. The Resolution Engine

### 6a. PostgreSQL Function: `resolve_context_variables`

Walks the hierarchy from broad → narrow. Higher-specificity layers override lower for the same key.

```sql
CREATE OR REPLACE FUNCTION public.resolve_context_variables(
    p_user_id         uuid,
    p_organization_id uuid DEFAULT NULL,
    p_workspace_id    uuid DEFAULT NULL,
    p_project_id      uuid DEFAULT NULL,
    p_task_id         uuid DEFAULT NULL,
    p_inject_as       text DEFAULT NULL,      -- filter: 'direct', 'tool_accessible', etc. NULL = all
    p_include_secrets  boolean DEFAULT false
)
RETURNS jsonb
LANGUAGE plpgsql SECURITY DEFINER STABLE
AS $$
DECLARE
    result jsonb := '{}'::jsonb;
    sources jsonb := '{}'::jsonb;
    rec record;
    var_entry jsonb;
BEGIN
    FOR rec IN (
        SELECT
            cv.key,
            cv.value,
            cv.value_type,
            cv.inject_as,
            cv.description,
            cv.is_secret,
            cv.is_system,
            cv.tags,
            CASE
                WHEN cv.task_id IS NOT NULL THEN 5
                WHEN cv.project_id IS NOT NULL THEN 4
                WHEN cv.workspace_id IS NOT NULL THEN 3
                WHEN cv.organization_id IS NOT NULL THEN 2
                WHEN cv.user_id IS NOT NULL THEN 1
                ELSE 0
            END AS priority,
            CASE
                WHEN cv.task_id IS NOT NULL THEN 'task'
                WHEN cv.project_id IS NOT NULL THEN 'project'
                WHEN cv.workspace_id IS NOT NULL THEN 'workspace'
                WHEN cv.organization_id IS NOT NULL THEN 'organization'
                WHEN cv.user_id IS NOT NULL THEN 'user'
                ELSE 'system'
            END AS source_scope
        FROM public.context_variables cv
        WHERE cv.is_active = true
          AND (p_inject_as IS NULL OR cv.inject_as = p_inject_as)
          AND (
              cv.user_id = p_user_id
              OR cv.organization_id = p_organization_id
              OR cv.workspace_id = p_workspace_id
              OR cv.project_id = p_project_id
              OR cv.task_id = p_task_id
              OR (cv.is_system = true AND cv.user_id IS NULL
                  AND cv.organization_id IS NULL AND cv.workspace_id IS NULL
                  AND cv.project_id IS NULL AND cv.task_id IS NULL)
          )
        ORDER BY priority ASC  -- low priority first, overwritten by higher
    )
    LOOP
        IF rec.is_secret AND NOT p_include_secrets THEN
            var_entry := jsonb_build_object(
                'value', '***REDACTED***',
                'type', rec.value_type,
                'inject_as', rec.inject_as,
                'source', rec.source_scope,
                'is_secret', true
            );
        ELSE
            var_entry := jsonb_build_object(
                'value', rec.value,
                'type', rec.value_type,
                'inject_as', rec.inject_as,
                'source', rec.source_scope,
                'description', rec.description
            );
        END IF;

        result := result || jsonb_build_object(rec.key, var_entry);
        sources := sources || jsonb_build_object(rec.key, rec.source_scope);
    END LOOP;

    RETURN jsonb_build_object(
        'variables', result,
        'sources', sources,
        'context', jsonb_build_object(
            'user_id', p_user_id,
            'organization_id', p_organization_id,
            'workspace_id', p_workspace_id,
            'project_id', p_project_id,
            'task_id', p_task_id
        ),
        'resolved_at', extract(epoch from now())
    );
END;
$$;

COMMENT ON FUNCTION public.resolve_context_variables IS
  'Resolves all context variables for a given scope, with cascading override.
   Task > Project > Workspace > Org > User > System.';
```

### 6b. Quick Resolution from Active Context

```sql
CREATE OR REPLACE FUNCTION public.resolve_active_context(
    p_user_id uuid,
    p_inject_as text DEFAULT NULL
)
RETURNS jsonb
LANGUAGE plpgsql SECURITY DEFINER STABLE
AS $$
DECLARE
    ctx public.user_active_context%ROWTYPE;
BEGIN
    SELECT * INTO ctx FROM public.user_active_context WHERE user_id = p_user_id;

    IF NOT FOUND THEN
        RETURN public.resolve_context_variables(p_user_id, p_inject_as := p_inject_as);
    END IF;

    RETURN public.resolve_context_variables(
        p_user_id,
        ctx.organization_id,
        ctx.workspace_id,
        ctx.project_id,
        ctx.task_id,
        p_inject_as
    );
END;
$$;
```

---

## 7. Agent Context Injection (AI Dream / Python Side)

### 7a. The Two-Tier Builder

```python
"""
context_engine.py — Builds agent context blocks for system prompt injection.
"""
from __future__ import annotations
import json
from typing import Any


async def build_agent_context(
    user_id: str,
    supabase: Any,
    override_scope: dict | None = None,
) -> AgentContext:
    """
    Main entry point. Returns both tiers of context.
    """
    # Resolve all variables (both tiers)
    if override_scope:
        all_vars = await supabase.rpc(
            'resolve_context_variables',
            {
                'p_user_id': user_id,
                'p_organization_id': override_scope.get('organization_id'),
                'p_workspace_id': override_scope.get('workspace_id'),
                'p_project_id': override_scope.get('project_id'),
                'p_task_id': override_scope.get('task_id'),
            }
        ).execute()
    else:
        all_vars = await supabase.rpc(
            'resolve_active_context',
            {'p_user_id': user_id}
        ).execute()

    resolved = all_vars.data
    variables = resolved.get('variables', {})
    context_scope = resolved.get('context', {})

    # Split by tier
    tier1_direct = {}
    tier2_tools = {}
    tier2_searchable = {}

    for key, var_data in variables.items():
        inject_as = var_data.get('inject_as', 'direct')
        if inject_as == 'direct':
            tier1_direct[key] = var_data
        elif inject_as == 'tool_accessible':
            tier2_tools[key] = var_data
        elif inject_as == 'searchable':
            tier2_searchable[key] = var_data

    return AgentContext(
        scope=context_scope,
        direct_variables=tier1_direct,
        tool_variables=tier2_tools,
        searchable_variables=tier2_searchable,
    )


class AgentContext:
    """Container for resolved context, split by delivery tier."""

    def __init__(
        self,
        scope: dict,
        direct_variables: dict,
        tool_variables: dict,
        searchable_variables: dict,
    ):
        self.scope = scope
        self.direct_variables = direct_variables
        self.tool_variables = tool_variables
        self.searchable_variables = searchable_variables

    def build_system_prompt_block(self) -> str:
        """
        Builds the XML block injected directly into the agent's system prompt.
        This is Tier 1 — always present, small, critical.
        """
        lines = ['<agent_context>']

        # Current scope
        lines.append('  <scope>')
        for key in ('organization_id', 'workspace_id', 'project_id', 'task_id'):
            val = self.scope.get(key)
            if val:
                label = key.replace('_id', '')
                lines.append(f'    {label}: {val}')
        lines.append('  </scope>')

        # Direct-inject variables
        if self.direct_variables:
            lines.append('  <variables>')
            for key, var_data in self.direct_variables.items():
                value = var_data.get('value')
                source = var_data.get('source', '')
                # Flatten jsonb value to string for injection
                if isinstance(value, (dict, list)):
                    value = json.dumps(value)
                elif isinstance(value, str):
                    # Already a string via jsonb, strip quotes
                    value = value.strip('"')
                lines.append(f'    {key}: {value}  [{source}]')
            lines.append('  </variables>')

        # Notify agent about available Tier 2 context
        if self.tool_variables or self.searchable_variables:
            lines.append('  <available_context>')
            if self.tool_variables:
                tool_keys = ', '.join(self.tool_variables.keys())
                lines.append(
                    f'    Tool-accessible variables: {tool_keys}'
                )
                lines.append(
                    '    Use get_context_variable(key) to access these.'
                )
            if self.searchable_variables:
                search_keys = ', '.join(self.searchable_variables.keys())
                lines.append(
                    f'    Searchable context: {search_keys}'
                )
                lines.append(
                    '    Use search_context(query) to find relevant content.'
                )
            lines.append('  </available_context>')

        lines.append('</agent_context>')
        return '\n'.join(lines)

    def get_tool_variable(self, key: str) -> Any:
        """Called when an agent uses the get_context_variable tool."""
        var = self.tool_variables.get(key)
        if var:
            return var.get('value')
        return None

    def get_all_tool_variables(self) -> dict:
        """Returns all Tier 2 tool-accessible variables."""
        return {
            k: v.get('value')
            for k, v in self.tool_variables.items()
        }
```

### 7b. What the Agent Sees — Example

User is working in: **Acme Corp → Engineering → Backend Rewrite → Auth Migration**

**Tier 1 (always in system prompt):**

```xml
<agent_context>
  <scope>
    organization: <uuid>
    workspace: <uuid>
    project: <uuid>
    task: <uuid>
  </scope>
  <variables>
    org_name: Acme Corporation  [organization]
    tech_stack: Python, FastAPI, React, PostgreSQL  [organization]
    coding_standards: PEP 8, strict typing, comprehensive docstrings  [organization]
    project_name: Backend Rewrite  [project]
    project_description: Full rewrite of legacy monolith to microservices  [project]
    target_framework: FastAPI 0.115  [project]
    task_title: Auth Migration  [task]
    task_description: Migrate auth from session-based to JWT + OAuth 2.1  [task]
    task_status: in_progress  [task]
    task_priority: high  [task]
    task_assignee: Arman  [task]
    response_style: concise, code-first  [user]
  </variables>
  <available_context>
    Tool-accessible variables: api_docs_url, db_schema, existing_auth_code, test_fixtures
    Use get_context_variable(key) to access these.
    Searchable context: project_notes, research_findings, meeting_transcripts
    Use search_context(query) to find relevant content.
  </available_context>
</agent_context>
```

**Tier 2 (agent pulls on demand):**

Agent sees `api_docs_url` is available → calls `get_context_variable("api_docs_url")` → gets the URL.

Agent needs to understand the current DB schema → calls `get_context_variable("db_schema")` → gets a large JSON doc describing the schema.

Agent wants to check if there are notes about auth decisions → calls `search_context("auth decision JWT reasoning")` → gets relevant notes from the project scope.

---

## 8. System-Populated Variables

These are auto-computed by AI Dream when resolving context. They don't live in `context_variables` — they're derived from the actual hierarchy tables and injected alongside.

| Key | Source | Layer |
|-----|--------|-------|
| `org_name` | `organizations.name` | org |
| `org_slug` | `organizations.slug` | org |
| `workspace_name` | `workspaces.name` | workspace |
| `workspace_path` | Computed from parent chain | workspace |
| `project_name` | `projects.name` | project |
| `project_slug` | `projects.slug` | project |
| `project_description` | `projects.description` | project |
| `task_title` | `tasks.title` | task |
| `task_description` | `tasks.description` | task |
| `task_status` | `tasks.status` | task |
| `task_priority` | `tasks.priority` | task |
| `task_assignee` | `profiles.display_name` via `tasks.assignee_id` | task |
| `task_due_date` | `tasks.due_date` | task |
| `user_display_name` | `profiles.display_name` | user |
| `user_role_in_org` | `organization_members.role` | org |
| `user_role_in_project` | `project_members.role` | project |
| `conversations_in_scope` | COUNT of recent `cx_conversation` at deepest scope | computed |
| `notes_in_scope` | COUNT of `notes` at deepest scope | computed |
| `active_sandbox` | `sandbox_instances.id` if one is running for this project | computed |

---

## 9. Example User-Defined Variables by Layer

### User Level
| Key | Type | inject_as | Example |
|-----|------|-----------|---------|
| `response_style` | string | direct | `"concise, code-first"` |
| `preferred_language` | string | direct | `"python"` |
| `timezone` | string | direct | `"America/Los_Angeles"` |
| `communication_tone` | string | direct | `"technical, no fluff"` |

### Organization Level
| Key | Type | inject_as | Example |
|-----|------|-----------|---------|
| `tech_stack` | string | direct | `"Python, FastAPI, React, Supabase"` |
| `coding_standards` | string | direct | `"PEP 8, strict typing"` |
| `deployment_platform` | string | direct | `"AWS via Coolify"` |
| `org_api_key` | secret | tool_accessible | `"sk-..."` |

### Workspace Level
| Key | Type | inject_as | Example |
|-----|------|-----------|---------|
| `database_type` | string | direct | `"PostgreSQL 17 via Supabase"` |
| `git_org` | string | direct | `"github.com/acme-eng"` |
| `ci_cd_pipeline` | string | tool_accessible | `"GitHub Actions → Coolify deploy"` |

### Project Level
| Key | Type | inject_as | Example |
|-----|------|-----------|---------|
| `target_framework` | string | direct | `"FastAPI 0.115"` |
| `db_schema` | json | tool_accessible | `{large schema doc}` |
| `api_docs_url` | string | tool_accessible | `"https://..."` |
| `design_doc` | string | searchable | `{large design document}` |

### Task Level
| Key | Type | inject_as | Example |
|-----|------|-----------|---------|
| `branch_name` | string | direct | `"feature/auth-migration"` |
| `pr_url` | string | direct | `"https://github.com/..."` |
| `blocked_by` | string | direct | `"Waiting on DB schema approval"` |
| `acceptance_criteria` | string | tool_accessible | `{detailed criteria}` |
| `test_fixtures` | json | tool_accessible | `{test data}` |

---

## 10. How the 9 Entities Interact with Context

| Entity | Scoped? | Agent Can... |
|--------|---------|-------------|
| **Conversations** | ✓ (org/ws/proj/task) | Search conversations in scope, read history |
| **Artifacts** | ✓ (inherits from conversation + independent scope) | Retrieve artifacts by type, search by tags |
| **Notes** | ✓ (org/ws/proj/task) | Search notes, read full content, summarize |
| **Files** | ✓ (org/ws/proj/task) | List files, read content, reference paths |
| **Data** | ✓ (org/ws/proj/task) | Query tables, read schemas, analyze data |
| **Knowledge** | ✓ (via rs_topic.project_id) | Search research, read analyses, cite sources |
| **Transcripts** | ✓ (org/ws/proj/task) | Search transcripts, read segments, extract quotes |
| **Code** | TBD | Access repos, read files, search codebase |
| **Environments** | ✓ (sandbox has project_id, app_instance gets scope cols) | Execute code, access file systems, manage processes |

---

## 11. Tools the Agent Gets for Tier 2 Access

These are the tools registered in the `tools` table that enable Tier 2 context access:

```python
CONTEXT_TOOLS = [
    {
        "name": "get_context_variable",
        "description": "Retrieve a specific context variable by key. Use when you see a variable listed in <available_context>.",
        "parameters": {"key": {"type": "string", "required": True}}
    },
    {
        "name": "search_notes",
        "description": "Search notes within the current scope. Returns matching notes with snippets.",
        "parameters": {
            "query": {"type": "string", "required": True},
            "scope": {"type": "string", "enum": ["task", "project", "workspace", "org", "all"], "default": "project"}
        }
    },
    {
        "name": "query_data_table",
        "description": "Query a user data table within the current scope.",
        "parameters": {
            "table_name": {"type": "string", "required": True},
            "filter": {"type": "object"},
            "limit": {"type": "integer", "default": 50}
        }
    },
    {
        "name": "search_knowledge",
        "description": "Search the knowledge base (research content, uploaded docs, analyses) within scope.",
        "parameters": {
            "query": {"type": "string", "required": True},
            "content_types": {"type": "array", "items": {"type": "string"}, "default": ["all"]}
        }
    },
    {
        "name": "search_transcripts",
        "description": "Search transcripts within scope. Returns matching segments with timestamps.",
        "parameters": {
            "query": {"type": "string", "required": True}
        }
    },
    {
        "name": "list_artifacts",
        "description": "List artifacts (code, SVGs, docs) from conversations in scope.",
        "parameters": {
            "artifact_type": {"type": "string"},
            "conversation_id": {"type": "string"}
        }
    },
    {
        "name": "search_conversations",
        "description": "Search past AI conversations within scope.",
        "parameters": {
            "query": {"type": "string", "required": True},
            "scope": {"type": "string", "default": "project"}
        }
    },
    {
        "name": "summarize_context",
        "description": "Have a sub-agent summarize large context (a note, research doc, transcript, etc.) before using it.",
        "parameters": {
            "entity_type": {"type": "string", "required": True},
            "entity_id": {"type": "string", "required": True},
            "focus": {"type": "string", "description": "What aspect to focus the summary on"}
        }
    }
]
```

---

## 12. Migration Execution Order

```
Phase 1: Fix the hierarchy
  1. workspace_members (new table)
  2. projects.workspace_id (new FK)
  3. tasks.settings (new column if missing)

Phase 2: Scope the entities
  4. Scope columns on cx_conversation
  5. Scope columns on notes
  6. Scope columns on udt_datasets
  7. Scope columns on user_files
  8. Scope columns on transcripts
  9. Scope columns on workflow
  10. Scope columns on prompts
  11. Scope columns on sandbox_instances (add org/ws/task)
  12. Scope columns on app_instances

Phase 3: New context tables
  13. cx_artifact (new table)
  14. context_variables (new table)
  15. user_active_context (new table)

Phase 4: Resolution engine
  16. resolve_context_variables function
  17. resolve_active_context function

Phase 5: RLS policies on all new tables

Phase 6: Backfill
  18. Set organization_id on existing entities via user_id → organization_members
  19. Set project_id on existing entities where inferable from metadata
```

---

## 13. The VS Code Analogy (Complete)

| VS Code | AI Matrx | How It's Delivered |
|---------|----------|-------------------|
| Open file in editor | Active cx_conversation | Tier 1: direct |
| Project root / workspace | Org → Workspace → Project → Task | Tier 1: scope block |
| `.env` variables | context_variables (cascading) | Tier 1: direct inject |
| `tsconfig.json` / `package.json` | Project settings + tech_stack vars | Tier 1: direct |
| Git branch / status | Task branch_name, PR URL | Tier 1: direct |
| File tree | Scoped notes, files, artifacts | Tier 2: tool search |
| Terminal / debugger | sandbox_instances, app_instances | Tier 2: run_in_sandbox |
| Search across files | search_notes, search_knowledge | Tier 2: tool search |
| Extensions / tools | Tools registry per agent | Agent config |
| User settings (global) | User-level context_variables | Tier 1: direct |
| Workspace settings (local) | Project/task-level overrides | Tier 1: cascading override |
| Copilot's codebase index | Knowledge base (rs_ system) | Tier 2: search_knowledge |
| Copilot's @workspace | summarize_context tool | Tier 2: sub-agent |
| Opened terminal CWD | user_active_context | Tier 1: scope block |
