Metadata-Version: 2.4
Name: bigquery-agent-analytics
Version: 0.4.0
Summary: SDK for analyzing and evaluating agent traces stored in BigQuery.
Author: Google LLC
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: google-adk>=1.0.0
Requires-Dist: google-cloud-bigquery>=3.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.9.0
Provides-Extra: all
Requires-Dist: bigframes>=1.0.0; extra == 'all'
Requires-Dist: google-cloud-aiplatform>=1.148.0; extra == 'all'
Requires-Dist: google-genai>=1.0.0; extra == 'all'
Requires-Dist: isort>=5.0; extra == 'all'
Requires-Dist: jsonschema>=4.0; extra == 'all'
Requires-Dist: pandas>=2.0.0; extra == 'all'
Requires-Dist: pyink>=24.3.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.21; extra == 'all'
Requires-Dist: pytest>=7.0; extra == 'all'
Requires-Dist: python-dotenv>=1.0.0; extra == 'all'
Requires-Dist: rdflib>=7.0; extra == 'all'
Provides-Extra: bigframes
Requires-Dist: bigframes>=1.0.0; extra == 'bigframes'
Provides-Extra: cli
Provides-Extra: dev
Requires-Dist: isort>=5.0; extra == 'dev'
Requires-Dist: jsonschema>=4.0; extra == 'dev'
Requires-Dist: pyink>=24.3.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: improvement
Requires-Dist: google-cloud-aiplatform>=1.148.0; extra == 'improvement'
Requires-Dist: google-genai>=1.0.0; extra == 'improvement'
Requires-Dist: pandas>=2.0.0; extra == 'improvement'
Requires-Dist: python-dotenv>=1.0.0; extra == 'improvement'
Provides-Extra: llm
Requires-Dist: google-genai>=1.0.0; extra == 'llm'
Provides-Extra: owl
Requires-Dist: rdflib>=7.0; extra == 'owl'
Description-Content-Type: text/markdown

# BigQuery Agent Analytics SDK

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12%20|%203.13%20|%203.14-blue)](pyproject.toml)
[![CI](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/actions/workflows/ci.yml/badge.svg)](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/actions/workflows/ci.yml)

An open-source Python SDK for analyzing, evaluating, and curating agent traces
stored in BigQuery. Built on top of the
[BigQuery Agent Analytics](https://adk.dev/integrations/bigquery-agent-analytics/), it provides
a consumption-layer toolkit for agent observability, analysis, evaluation, and advanced capabilities like the Agent Context Graph — extracting decision traces from your agent's context graph — at scale.

## Overview

The BigQuery Agent Analytics SDK connects your AI agent telemetry in BigQuery to
a rich set of evaluation, observability, and analytics capabilities. It is
designed for ML engineers, data scientists, and platform teams who run agents in
production and need to understand agent behavior, measure quality, and detect
regressions — all through BigQuery SQL or Python.

## Key Features

**Observability**
- Trace reconstruction and DAG visualization
- Per-event-type BigQuery views
- Observability dashboards (SQL and BigFrames)

**Evaluation**
- Code-based metrics (latency, turn count, error rate, token efficiency, cost)
- LLM-as-Judge scoring (correctness, hallucination, sentiment)
- Trajectory matching (exact, in-order, any-order)
- Multi-trial evaluation with pass@k / pass^k
- Grader composition (weighted, binary, majority strategies)
- Eval suite lifecycle management with graduation and saturation detection
- Static quality validation (ambiguous tasks, class imbalance, suspicious thresholds)

**AI/ML Integration**
- BigQuery AI.GENERATE, AI.EMBED, AI.CLASSIFY
- Anomaly detection and latency forecasting
- Categorical (Hatteras-style) evaluation via BigFrames

**Advanced Analytics**
- Agent Context Graph — extract decision traces from your agent's context graph: the requests an agent handled, the options it weighed, and the outcomes it committed, materialized into a queryable BigQuery property graph (GQL traversal, scheduled refresh via `bqaa context-graph`)
- Long-horizon cross-session memory
- Multi-stage agent insights pipeline
- Drift detection for golden vs production question distributions

**CLI** (`bq-agent-sdk`)
- 12+ commands for diagnostics, evaluation, and CI/CD integration

**Deployment Surfaces**
- Remote Function (BigQuery SQL via Cloud Run)
- Python UDF scoring kernels
- Streaming evaluation (Cloud Scheduler + Cloud Run)
- Continuous query templates

**Usage Telemetry**
- Every job the SDK submits is labeled (`sdk`, `sdk_version`,
  `sdk_surface`, `sdk_feature`, and `sdk_ai_function` where relevant)
  so operators can attribute spend, latency, and adoption directly
  from `INFORMATION_SCHEMA.JOBS_BY_PROJECT`. No extra telemetry
  pipeline is required. See [docs/sdk_usage_tracking.md](docs/sdk_usage_tracking.md)
  for the label schema and ready-to-run tracking queries.

## Prerequisites

- Python 3.10+
- A Google Cloud project with BigQuery enabled
- Agent traces stored in BigQuery via the
  [ADK BigQuery Trace Exporter](https://github.com/google/adk-python/tree/main/contributing/extensions/bigquery_trace_exporter)

## Installation

```bash
pip install bigquery-agent-analytics
```

With optional LLM judge support:

```bash
pip install bigquery-agent-analytics[llm]
```

With BigFrames support:

```bash
pip install bigquery-agent-analytics[bigframes]
```

## Quick Start

```python
from bigquery_agent_analytics import Client

client = Client(project_id="my-project", dataset_id="analytics")
trace = client.get_trace("trace-abc-123")
trace.render()
```

See [SDK.md](SDK.md) for the full API walkthrough with code examples for every
feature.

### Try it: extract decision traces (Agent Context Graph, ~10 minutes)

Deploy a context graph, seed sample agent events, extract the decision
traces, and query one in GQL — entirely from your terminal:

```bash
export PROJECT_ID="your-project" DATASET="agent_analytics_demo"
gcloud config set project "$PROJECT_ID"
bq --location=US mk --dataset "$PROJECT_ID:$DATASET"

# 1. Deploy the context graph (one-time DDL: tables, then the property graph).
cd examples/context_graph/codelab
envsubst < table_ddl.sql      | bq query --use_legacy_sql=false
envsubst < property_graph.sql | bq query --use_legacy_sql=false

# 2. Seed five sample agent sessions into agent_events.
bqaa seed-events --project-id "$PROJECT_ID" --dataset-id "$DATASET" --sessions 5

# 3. Extract decision traces from the deployed graph
#    (read back via INFORMATION_SCHEMA.PROPERTY_GRAPHS — no SQL file passed).
bqaa context-graph --project-id "$PROJECT_ID" --dataset-id "$DATASET" \
    --graph agent_decisions_graph --lookback-hours 24 --format json

# 4. Query a decision trace: what did the agent weigh, and how did it resolve?
bq query --use_legacy_sql=false "
SELECT * FROM GRAPH_TABLE(
  $DATASET.agent_decisions_graph
  MATCH (req:DecisionRequest)-[eo:evaluatesOption]->(opt:DecisionOption),
        (req)-[ri:resultedIn]->(out:DecisionOutcome)
  COLUMNS (req.request_text AS question, opt.option_label AS considered,
           out.status AS outcome, out.rationale AS rationale))"
```

Expect `"ok": true` with 5 sessions materialized, and fifteen GQL rows — three
options weighed per request, each with the committed outcome and rationale.
The [Agent Context Graph codelab](docs/codelabs/periodic_materialization.md) is the
guided version of these steps (plus backfill and production scheduling), and
[`examples/context_graph/`](examples/context_graph/) is the worked example
with a runnable ADK agent.

## Documentation

| Resource | Description |
|----------|-------------|
| [SDK Feature Reference](SDK.md) | Complete API walkthrough with working code examples |
| [Agent Context Graph Codelab](docs/codelabs/periodic_materialization.md) | Extract decision traces from your agent's context graph, end to end (~35 min) |
| [Scheduled Deploy Runbook](docs/guides/scheduled-context-graph-deploy.md) | Keep the context graph fresh on a Cloud Run + Cloud Scheduler cron |
| [Design Documents](docs/README.md) | Architecture decisions and design rationale |
| [Examples](examples/README.md) | Notebooks, SQL scripts, and demos |
| [Deployment Guides](deploy/README.md) | Four deployment surfaces for Google Cloud |

## Architecture

```
src/bigquery_agent_analytics/
│
├── Core
│   ├── client.py                  # High-level SDK client
│   ├── trace.py                   # Trace reconstruction & visualization
│   ├── views.py                   # Per-event-type BigQuery view management
│   ├── event_semantics.py         # Canonical event type helpers & predicates
│   ├── serialization.py           # Uniform serialization layer
│   └── formatter.py               # Output formatting (json/text/table)
│
├── Evaluation
│   ├── evaluators.py              # SystemEvaluator + LLMAsJudge
│   ├── trace_evaluator.py         # Trajectory matching & replay
│   ├── multi_trial.py             # Multi-trial runner + pass@k
│   ├── grader_pipeline.py         # Grader composition pipeline
│   ├── eval_suite.py              # Eval suite lifecycle management
│   └── eval_validator.py          # Static validation checks
│
├── AI/ML
│   ├── ai_ml_integration.py       # BigQuery AI/ML capabilities
│   ├── bigframes_evaluator.py     # BigFrames DataFrame evaluator
│   ├── categorical_evaluator.py   # Hatteras categorical evaluation
│   └── categorical_views.py       # Categorical metric views
│
├── Analytics
│   ├── insights.py                # Multi-stage insights pipeline
│   ├── feedback.py                # Drift detection & question distribution
│   └── memory_service.py          # Long-horizon agent memory
│
├── Agent Context Graph
│   ├── context_graph.py           # Decision-trace extraction & GQL traversal
│   ├── materialize_window.py      # Scheduled materialization (bqaa context-graph)
│   └── property_graph_spec.py     # Derive the spec from your deployed property graph
│
└── CLI & Deploy
    ├── cli.py                     # CLI entry point (bq-agent-sdk)
    ├── udf_kernels.py             # Python UDF scoring kernels
    └── udf_sql_templates.py       # UDF SQL generation
```

## Related Projects

- [Google ADK](https://github.com/google/adk-python) — Agent Development Kit
  for building AI agents
- [BigQuery](https://cloud.google.com/bigquery) — Google Cloud analytics data
  warehouse
- [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-application-overview) —
  AI.GENERATE, AI.EMBED, AI.CLASSIFY, and more

## Development

```bash
# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Format code
pyink --config pyproject.toml src/ tests/
isort src/ tests/
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

Apache License 2.0 — see [LICENSE](LICENSE) for details.
