---
title: "Tracing"
---

Tracing records the **full chain** of an agent's execution and is the core of observability in enterprise-grade agent applications. It turns a request's complete flow—user input, model reasoning, tool calls, memory and knowledge-base access, response generation—into analyzable, traceable structured data, providing a basis for debugging, monitoring, and optimization.

## Enabling Observability in VeADK

To enable full-chain observability, attach an `OpentelemetryTracer` with one or more exporters to your agent. The example below uses the APMPlus exporter:

<Tabs items={['Code', 'Required Environment Variables']}>
  <Tab value="Code">
    ```python title="examples/tracing/tracing_exporter.py"
    import asyncio
    from veadk import Agent, Runner
    from veadk.tracing.telemetry.exporters.apmplus_exporter import APMPlusExporter
    from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer

    # Initialize the tracing exporter
    exporters = [APMPlusExporter()]
    tracer = OpentelemetryTracer(exporters=exporters)

    # Define the agent and attach the tracer
    agent = Agent(tracers=[tracer])

    runner = Runner(agent=agent)
    response = asyncio.run(runner.run(messages="hi!"))
    print(response)
    ```
  </Tab>
  <Tab value="Required Environment Variables">
    Configure the exporter via environment variables:

    - `OBSERVABILITY_OPENTELEMETRY_APMPLUS_API_KEY`: The API key of the APM service
    - `OBSERVABILITY_OPENTELEMETRY_APMPLUS_ENDPOINT`: The endpoint of the APM service, e.g. `http://apmplus-cn-beijing.volces.com:4317`
    - `OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME`: The service name for APM, e.g. `python_coder_agent`
    - `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT`: Whether to collect Agent, LLM, and tool input/output content, default `true`

    Or define them in `config.yaml`:

    ```yaml title="config.yaml"
    observability:
      opentelemetry:
        trace_content: true
        apmplus:
          endpoint: ...
          api_key: ...
          service_name: ...
    ```
  </Tab>
</Tabs>

Once enabled, the IDs of the related tracing data are printed in the logs:

![TraceID illustration](/assets/images/observation/show_traceid.jpeg)

For the full integration steps per platform, see [Observe on Volcengine](/en/docs/framework/ve-tracing).

## Why Tracing

With tracing in place, you can:

- **Debug and locate problems**: Quickly pinpoint where logic errors, tool-call failures, or abnormal model output occur;
- **Analyze performance**: Discover bottlenecks from per-module latency and optimize model-call and tool-execution strategies;
- **Audit and comply**: Keep a complete record of requests, responses, and decisions;
- **Coordinate multi-agent systems**: Analyze invocation relationships and data flow between agents in collaboration scenarios.

## Tracing Data Structure

Tracing data consists of a series of **spans**, each recording the name, IDs, start/end time, and attributes of one operation (such as calling the model or executing a tool). Below is an example of a single tool-call span:

```json title="tracing.json"
{
  "name": "execute_tool get_city_weather",
  "span_id": 15420762184471404328,
  "trace_id": 195590910357197178730434145750344919939,
  "start_time": 1762770336065962000,
  "end_time": 1762770336066380000,
  "attributes": {
    "gen_ai.operation.name": "execute_tool",
    "gen_ai.tool.name": "get_city_weather",
    "gen_ai.tool.input": "{\"name\": \"get_city_weather\"}",
    "gen_ai.tool.output": "{\"response\": {\"result\": \"Sunny, 25°C\"}}",
    "cozeloop.input": "{\"name\": \"get_city_weather\"}",
    "cozeloop.output": "{\"response\": {\"result\": \"Sunny, 25°C\"}}"
  },
  "parent_span_id": 18301436667977577407
}
```

See [Span Attributes](/en/docs/framework/span-attributes) for what each attribute means.

In a multi-agent scenario, spans are linked through `parent_span_id`, showing the full chain of `coding_agent` dispatching to `python_coder`:

```json title="tracing.json"
[
  {
    "name": "execute_tool transfer_to_agent",
    "span_id": 8839983747734433620,
    "trace_id": 161011438572144016825706884868588853689,
    "attributes": {
      "agent.name": "coding_agent",
      "gen_ai.operation.name": "execute_tool",
      "gen_ai.tool.name": "transfer_to_agent"
    },
    "parent_span_id": 5680387922600458423
  },
  {
    "name": "call_llm",
    "span_id": 6376035117066938110,
    "trace_id": 161011438572144016825706884868588853689,
    "attributes": {
      "agent.name": "python_coder",
      "gen_ai.request.model": "openai/doubao-seed-1-8-251228",
      "gen_ai.request.type": "chat",
      "gen_ai.span.kind": "llm",
      "gen_ai.prompt.0.role": "user",
      "gen_ai.prompt.0.content": "Please write a quicksort implementation in Python."
    },
    "parent_span_id": 11028930176666491606
  },
  {
    "name": "invoke_agent python_coder",
    "span_id": 11028930176666491606,
    "trace_id": 161011438572144016825706884868588853689,
    "attributes": {
      "gen_ai.span.kind": "agent",
      "gen_ai.operation.name": "invoke_agent",
      "gen_ai.agent.description": "Skilled at solving problems using the Python programming language.",
      "gen_ai.system": "openai",
      "agent.name": "python_coder"
    },
    "parent_span_id": 5680387922600458423
  },
  {
    "name": "invoke_agent coding_agent",
    "span_id": 987150150265236585,
    "trace_id": 161011438572144016825706884868588853689,
    "attributes": {
      "gen_ai.span.kind": "agent",
      "gen_ai.operation.name": "invoke_agent",
      "gen_ai.agent.description": "Can invoke a suitable agent to solve user problems.",
      "gen_ai.system": "openai",
      "agent.name": "coding_agent"
    },
    "parent_span_id": 11431457420615823859
  },
  {
    "name": "invocation",
    "span_id": 11431457420615823859,
    "trace_id": 161011438572144016825706884868588853689,
    "attributes": {
      "gen_ai.operation.name": "chain",
      "gen_ai.span.kind": "workflow",
      "gen_ai.system": "openai",
      "agent.name": "python_coder"
    },
    "parent_span_id": null
  }
]
```

<Callout type="info" title="OpenTelemetry Compatibility">
  VeADK's field naming follows the OpenTelemetry generative-AI conventions, so you can import tracing data directly into any OpenTelemetry-compatible system for analysis and visualization.
</Callout>
