# Workflow Trace Tab Design

## Architecture Overview
The tracing feature spans backend instrumentation, persistence, API exposure, and frontend visualization.

1. **OpenTelemetry Instrumentation**
   - Introduce a trace provider within the backend execution engine to create a root span per workflow execution.
   - Child spans represent node executions; they capture attributes for node ID, display name, status, token usage, latency, and artifact references.
   - Instrumentation uses the OpenTelemetry SDK with configurable exporters (OTLP/HTTP by default) and supports runtime-configurable sampling.
   - Span attributes follow the schema documented in [Span Schema](#span-schema).

2. **Trace Persistence & Metadata**
   - Extend the run history models to store trace IDs and lightweight span metadata (execution ID, trace ID, latest span timestamp, status).
   - Persist detailed span data via the OpenTelemetry exporter to an external collector; the API fetches data either from the collector or an internal cache, depending on deployment mode.
   - Update repository interfaces to propagate trace identifiers so UI can link to traces immediately after run creation.

3. **Trace Retrieval API**
   - Add REST endpoints under `/executions/{execution_id}/trace` providing:
     - Execution-level metadata: status, started/finished timestamps, total token counts.
     - A hierarchical span tree with attributes, links to artifacts, and optionally correlation IDs to external systems.
     - Pagination or depth limiting for large trace trees.
   - The API returns JSON that matches the schema in [Trace API Response](#trace-api-response).
   - For live runs, integrate with WebSocket channels to push incremental span updates using existing execution channels (augment message schema with trace payloads described in [WebSocket Message Schema](#websocket-message-schema)).

4. **Canvas Trace Tab**
   - Expand Canvas layout to include a `Trace` tab, registered alongside `Editor`, `Execution`, `Readiness`, and `Settings`.
   - Ensure the tab bar remains readable on narrow widths by reusing the responsive overflow behavior in `workflow-tabs.tsx`; if space becomes constrained, fall back to grouping Trace under Execution as a sub-tab.
   - Implement a trace viewer component that renders a collapsible tree with duration bars, status badges, token metrics, and artifact actions.
   - Provide a detail panel showing span attributes, prompts/responses, and resource usage.
   - Hook into execution selection state so the Trace tab updates when users switch runs; support loading states and error handling.

5. **External Integrations**
   - Surface optional links to external observability dashboards (e.g., Grafana Tempo, Honeycomb) based on deployment configuration.
   - Allow administrators to configure exporter endpoints and authentication via environment variables or admin UI (documented in configuration guides).
   - Instrumentation hooks piggyback on existing execution streaming logic in `workflow_execution.py`, ensuring spans are emitted when node lifecycle callbacks run.

## Span Schema

Spans adopt a consistent attribute set so downstream visualizations can make strong assumptions about available metadata.

| Attribute Key | Type | Description |
| --- | --- | --- |
| `orcheo.execution.id` | string | UUID of the workflow execution (root span only). |
| `orcheo.node.id` | string | Node identifier within the workflow graph. |
| `orcheo.node.display_name` | string | Human-friendly name displayed in Canvas. |
| `orcheo.node.kind` | string | Node type (e.g., `llm`, `tool`, `branch`). |
| `orcheo.node.status` | string | `success`, `error`, or `skipped`. |
| `orcheo.node.latency_ms` | int | Duration of the node execution in milliseconds. |
| `orcheo.token.input` | int | Prompt token count attributed to the node. |
| `orcheo.token.output` | int | Completion token count attributed to the node. |
| `orcheo.artifact.ids` | array<string> | IDs of artifacts generated by the node. |
| `orcheo.error.code` | string | Optional machine-readable error code when status is `error`. |

Token usage deltas larger than 1,000 tokens trigger a span event named `token.chunk` with payload `{ "input": int, "output": int, "reason": string }`. Prompt/response text is attached via `message` span events with redacted content as needed.

## Trace API Response

The REST endpoint responds with JSON conforming to the following shape:

```json
{
  "execution": {
    "id": "uuid",
    "status": "success",
    "started_at": "2024-05-04T12:32:14Z",
    "finished_at": "2024-05-04T12:32:42Z",
    "token_usage": { "input": 1200, "output": 900 },
    "trace_id": "0af7651916cd43dd8448eb211c80319c"
  },
  "spans": [
    {
      "span_id": "b7ad6f...",
      "parent_span_id": null,
      "name": "Workflow Execution",
      "start_time": "2024-05-04T12:32:14.120Z",
      "end_time": "2024-05-04T12:32:42.981Z",
      "attributes": { "orcheo.execution.id": "uuid" },
      "events": [],
      "status": { "code": "OK" }
    },
    {
      "span_id": "e2f3...",
      "parent_span_id": "b7ad6f...",
      "name": "LLM Node",
      "start_time": "2024-05-04T12:32:15.001Z",
      "end_time": "2024-05-04T12:32:19.442Z",
      "attributes": {
        "orcheo.node.id": "node-3",
        "orcheo.node.display_name": "Draft Answer",
        "orcheo.token.input": 450,
        "orcheo.token.output": 512
      },
      "events": [
        { "name": "message", "time": "2024-05-04T12:32:16.001Z", "attributes": { "role": "user" } }
      ],
      "links": []
    }
  ],
  "page_info": {
    "has_next_page": false,
    "cursor": null
  }
}
```

Spans are returned as a flat array coupled with `parent_span_id` pointers to avoid deeply nested payloads. Clients reconstruct the tree in-memory. Pagination is cursor-based using the `page_info.cursor`; clients request additional spans by supplying `?cursor=<token>`. Large traces may also provide a `depth` query parameter to limit traversal.

## WebSocket Message Schema

Live updates reuse the existing execution WebSocket channel. New messages adopt the following envelope to minimize UI flicker:

```json
{
  "type": "trace:update",
  "execution_id": "uuid",
  "trace_id": "0af765...",
  "spans": [
    {
      "span_id": "e2f3...",
      "parent_span_id": "b7ad6f...",
      "name": "LLM Node",
      "attributes": { "orcheo.node.status": "running" },
      "status": { "code": "UNSET" }
    }
  ],
  "complete": false
}
```

Partial span payloads only include fields that changed since the previous message. The frontend applies updates idempotently by merging attributes and status using span IDs. A message with `complete: true` signals that the trace is finalized.

## Data Flow
1. User triggers workflow execution.
2. Backend execution engine starts root span and records trace ID in run metadata.
3. Node executions create child spans, emitting prompts, responses, token usage, and artifact metadata via OpenTelemetry attributes/events.
4. Spans are exported to the configured collector; metadata is persisted in the Orcheo database.
5. Canvas requests `/executions/{id}/trace`; backend aggregates span tree (via collector query or cached span store) and returns JSON payload.
6. Canvas renders trace tree; listens for WebSocket messages to append or update spans for live runs.
7. Users can download artifacts directly from the Trace tab (reusing existing artifact endpoints) and follow links to external dashboards if available.

## Security & Privacy
- Ensure secrets and sensitive content (e.g., API keys) are redacted before being attached to spans.
- Enforce authorization checks on the trace API to match execution access controls.
- Provide configuration flags to disable prompt/response capture for compliance-sensitive environments.

## Performance Considerations
- Use sampling controls to limit trace volume in high-throughput environments.
- Implement pagination or lazy loading for spans when trace depth exceeds configured thresholds.
- Cache recent spans in memory to reduce collector round-trips for live updates.

## Artifact Handling

Artifact payloads larger than 64 KiB are referenced via signed URLs exposed by the existing artifact download endpoint. Metadata-only artifacts (e.g., tool results) under that threshold are embedded directly as span events for quick inspection. The decision avoids bloating span payloads while keeping small context readily available.

## Retention Policy

Trace metadata stored in Orcheo follows the same retention window as execution history (default 30 days, configurable per workspace). Exported spans inside the external collector obey that system's retention policies. When a run is deleted, associated trace IDs are purged from Orcheo and a deletion request is issued to the collector to maintain GDPR compliance.
