# MCA SDK Examples

This directory contains working examples demonstrating MCA SDK features using the GCP development environment.

## Overview

All examples follow a consistent pattern:
1. **GCP Authentication** - Verify Application Default Credentials
2. **Collector Connectivity** - Test connection to GCP dev collector
3. **Connect to Registry API** - Create registry client with GCP auth
4. **Filter Models by Type** - Discover available models (demonstrates registry filtering)
5. **Fetch Specific Model Config** - Get configuration for your model
6. **SDK Initialization** - Set up MCA SDK with registry config
7-10. **Feature Demonstrations** - Show specific SDK capabilities
11. **Telemetry Flush** - Send all data to collector
12. **Verification** - Confirm telemetry in GCP Console

## Prerequisites

All examples require:

### 1. Install SDK

```bash
pip install mca-sdk[gcp,genai]>=0.6.1
```

### 2. GCP Authentication

```bash
gcloud auth application-default login
```

### 3. VPN Connection (if outside GKE cluster)

The GCP dev collector (`10.164.76.55`) requires VPC access. Connect to VPN if working outside the GKE cluster.

## Available Examples

All three core examples demonstrate **complete Registry API integration** including:
- Connecting to Registry API with GCP authentication
- Filtering models by type (demonstrates discovery capability)
- Fetching specific model configuration
- Using registry config to initialize SDK
- Complete telemetry flow with verification

---

### 1. internal-model/ - Clinical Prediction Model

**Use Case:** Traditional ML model (classification, regression)

**Features:**
- ✅ Registry filtering by `model_type="classification"`
- ✅ Input/output capture with `@predict` decorator
- ✅ Manual prediction recording with threshold checking
- ✅ Custom operation tracing with `trace()` context manager
- ✅ Custom metrics (counter, gauge, histogram)
- ✅ Error tracking with context

**Run:**
```bash
cd internal-model
python instrumented_model.py
```

**Best For:** Data scientists instrumenting scikit-learn, XGBoost, or TensorFlow models

---

### 2. internal-genai/ - LLM Clinical Documentation Assistant

**Use Case:** GenAI/LLM applications (GPT-4, Claude, etc.)

**Features:**
- ✅ Registry filtering by `model_type="generative"`
- ✅ `create_genai_client()` for LLM-specific initialization
- ✅ `@predict` decorator for LLM functions
- ✅ `track_llm_completion()` for token/cost tracking
- ✅ `estimate_openai_cost()` for cost calculation
- ✅ Cost threshold checking from registry config
- ✅ Custom GenAI metrics (cache hits, rate limits, quality scores)
- ✅ LLM error tracking with context

**Run:**
```bash
cd internal-genai
python litellm_instrumented.py
```

**Best For:** Data scientists using LLMs for clinical note summarization, triage, or decision support

---

### 3. internal-agentic/ - Medical Research Assistant Agent

**Use Case:** Agentic AI with multi-step reasoning and tool use

**Features:**
- ✅ Registry filtering by `model_type="agentic"`
- ✅ Goal tracking with `record_goal_started/completed()`
- ✅ Tool execution tracking with `track_tool()`
- ✅ Multi-step reasoning spans with `reasoning_step()`
- ✅ Human intervention recording
- ✅ Policy violation tracking with `record_policy_violation()`
- ✅ Error tracking with goal context

**Run:**
```bash
cd internal-agentic
python agent_instrumented.py
```

**Best For:** Data scientists building AI agents that use multiple tools or APIs to complete complex tasks

---

## GCP Development Environment

All examples use the GCP dev environment by default:

| Service | Endpoint | Port |
|---------|----------|------|
| OTel Collector (HTTP) | `http://10.164.76.55` | 4318 |
| OTel Collector (gRPC) | `10.164.76.8` | 4317 |
| Registration API | `https://emms-registration-api-dev-rlaptra7zq-ue.a.run.app` | 443 |

**VPN Required:** The collector IP `10.164.76.55` is only accessible from within the VPC or via VPN.

## Quick Start

1. **Install SDK and authenticate:**
   ```bash
   pip install mca-sdk[gcp,genai]>=0.6.1
   gcloud auth application-default login
   ```

2. **Connect to VPN** (if outside GKE cluster)

3. **Run any example:**
   ```bash
   cd internal-model
   python instrumented_model.py
   ```

4. **Verify telemetry in GCP Console:**
   - **Logs:** https://console.cloud.google.com/logs/query
   - **Traces:** https://console.cloud.google.com/traces/list

---

## SDK API Quick Reference

### Core SDK Methods (All Model Types)

| Method | Use When | Example Location |
|--------|----------|------------------|
| `@predict(capture_input, capture_output)` | Wrapping prediction functions | `internal-model`, `internal-genai` |
| `record_prediction(latency, prediction_id, **attrs)` | Manually logging predictions | `internal-model` |
| `record_error(error, error_type, severity, context)` | Tracking errors with context | All examples |
| `record_counter(name, value, **attrs)` | Incrementing counts (cache hits, requests) | All examples |
| `record_gauge(name, value, **attrs)` | Point-in-time values (queue size, rate limits) | All examples |
| `record_metric(name, value, metric_type, **attrs)` | Histograms (latency, confidence distributions) | All examples |
| `trace(operation_name)` | Custom operation spans | `internal-model` |

### GenAI-Specific Methods

| Method | Use When | Example Location |
|--------|----------|------------------|
| `create_genai_client(...)` | Initializing SDK for LLM models | `internal-genai` |
| `track_llm_completion(model, prompt_tokens, completion_tokens, latency, status)` | Tracking LLM API calls | `internal-genai` |
| `estimate_openai_cost(model, prompt_tokens, completion_tokens)` | Calculating LLM costs | `internal-genai` |

### Agentic AI Methods

| Method | Use When | Example Location |
|--------|----------|------------------|
| `record_goal_started(goal_description, goal_type, **attrs)` | Starting multi-step agent tasks | `internal-agentic` |
| `record_goal_completed(goal_id, status, **attrs)` | Completing agent goals | `internal-agentic` |
| `track_tool(tool_name, goal_id, **attrs)` | Tracking tool/API usage in agents | `internal-agentic` |
| `reasoning_step(step_type, step_description)` | LLM reasoning/planning steps | `internal-agentic` |
| `record_human_intervention(intervention_type, reason, **attrs)` | Human-in-the-loop events | `internal-agentic` |
| `record_policy_violation(policy_type)` | AI safety/guardrail violations | `internal-agentic` |

### Registry API Methods

| Method | Use When | Example Location |
|--------|----------|------------------|
| `RegistryClient(url, use_gcp_auth=True)` | Connecting to Registry API | All examples |
| `list_configs(limit, team_name, model_category, model_type)` | Browsing/filtering models | All examples |
| `fetch_model_config(model_id)` | Getting model configuration | All examples |
| `client.thresholds` | Accessing registry thresholds | `internal-model`, `internal-genai` |

### Where to Use Each Feature in Your Code

```python
# 1. STARTUP: Initialize SDK with Registry
from mca_sdk import MCAClient
from mca_sdk.registry import RegistryClient

registry_client = RegistryClient(url=REGISTRY_URL, use_gcp_auth=True)
model_config = registry_client.fetch_model_config(model_id)

client = MCAClient(
    service_name=model_config.service_name,
    team_name=model_config.team_name,
    model_id=model_id,
    collector_endpoint=COLLECTOR_ENDPOINT,
    use_gcp_auth=True
)

# 2. PREDICTION: Wrap your prediction function
@client.predict(capture_input=True, capture_output=True)
def predict(features: dict) -> dict:
    # Your model logic here
    return prediction

# OR manually record predictions
latency = time.time() - start
client.record_prediction(latency=latency, prediction_id="pred-001")

# 3. CUSTOM METRICS: Business metrics
client.record_counter("model.cache_hits", 10)  # Incrementing values
client.record_gauge("model.queue_size", 5)     # Current state
client.record_metric("model.confidence", 0.87, metric_type="histogram")  # Distributions

# 4. LLM TRACKING: For GenAI models
from mca_sdk.integrations import track_llm_completion, estimate_openai_cost

cost = estimate_openai_cost("gpt-4", prompt_tokens=100, completion_tokens=50)
track_llm_completion(client, model="gpt-4", prompt_tokens=100,
                     completion_tokens=50, latency=0.5, status="success")

# 5. AGENT TRACKING: For multi-step agents
goal_id = client.record_goal_started("Research medical question", goal_type="research")

with client.track_tool("pubmed_search", goal_id=goal_id):
    results = search_pubmed(query)

with client.reasoning_step(step_type="planning", step_description="Planning approach"):
    plan = create_plan()

client.record_goal_completed(goal_id, status="success")

# 6. ERROR TRACKING: When things go wrong
try:
    result = risky_operation()
except Exception as e:
    client.record_error(error=e, error_type="validation_error",
                       severity="warning", context={"patient_id": "123"})

# 7. THRESHOLD CHECKING: Use registry thresholds
latency_threshold = client.thresholds.get('latency_ms', 1000)
if latency * 1000 > latency_threshold:
    print(f"Warning: High latency {latency*1000}ms")

# 8. CUSTOM OPERATIONS: Trace any operation
with client.trace("data_preprocessing"):
    preprocessed_data = preprocess(raw_data)

# 9. POLICY VIOLATIONS: AI safety
if prohibited_content_detected:
    client.record_policy_violation(policy_type="content_safety")

# 10. SHUTDOWN: Flush telemetry
client.shutdown(timeout_millis=5000)
registry_client.close()
```

### Feature Matrix: Which Example Shows What?

| Feature | internal-model | internal-genai | internal-agentic |
|---------|:--------------:|:--------------:|:----------------:|
| `@predict` decorator | ✅ | ✅ | - |
| `record_prediction()` | ✅ | - | - |
| `record_error()` | ✅ | ✅ | ✅ |
| `record_counter/gauge/metric()` | ✅ | ✅ | - |
| `trace()` context manager | ✅ | - | - |
| Registry filtering | ✅ | ✅ | ✅ |
| Threshold checking | ✅ | ✅ | - |
| `create_genai_client()` | - | ✅ | - |
| `track_llm_completion()` | - | ✅ | - |
| `estimate_openai_cost()` | - | ✅ | - |
| `record_goal_started/completed()` | - | - | ✅ |
| `track_tool()` | - | - | ✅ |
| `reasoning_step()` | - | - | ✅ |
| `record_human_intervention()` | - | - | ✅ |
| `record_policy_violation()` | - | - | ✅ |

---

## Shared Utilities

The `shared/` directory provides reusable utilities used by all examples:

```python
from shared import (
    print_gcp_auth_status,           # Verify ADC
    print_collector_status,          # Test collector connectivity
    COLLECTOR_HTTP_ENDPOINT,         # Standard endpoints
    REGISTRY_URL,                    # Registry API URL
    DEFAULT_MODEL_ID,                # Default model for demos
)
```

## Verification

After running any example, verify telemetry in GCP:

### Cloud Logging

Visit: https://console.cloud.google.com/logs/query

Filter:
```
logName="projects/bhsf-ai-team-projects/logs/emms-model-telemetry"
AND labels."service.name"="your-service-name"
```

### Cloud Trace

Visit: https://console.cloud.google.com/traces/list

Search: `service.name="your-service-name"`

### Google-Managed Prometheus (via Cloud Monitoring)

Contact DevOps for Cloud Monitoring dashboard URLs.

## Troubleshooting

### Cannot connect to collector

**Symptoms:**
```
✗ OTLP HTTP (4318): CLOSED
✗ OTLP gRPC (4317): CLOSED
```

**Solutions:**
1. Connect to VPN
2. Verify you can ping `10.164.76.55`
3. Check firewall rules

### ADC not configured

**Symptoms:**
```
✗ ADC not configured: Could not automatically determine credentials
```

**Solution:**
```bash
gcloud auth application-default login
```

### No telemetry in Cloud Logging

**Causes:**
- Telemetry not yet propagated (wait 30-60 seconds)
- Service name mismatch in filter
- Time range doesn't include execution time

**Solution:**
1. Wait 1-2 minutes after running example
2. Refresh Cloud Logging console
3. Verify service name matches exactly

### Registry API timeout

**Symptoms:**
```
✗ Failed to fetch config: Registry request timed out
```

**Solutions:**
1. Verify internet connectivity
2. Check Registry API is online: `curl https://emms-registration-api-dev-rlaptra7zq-ue.a.run.app/health`
3. Increase timeout: `registry_client = RegistryClient(url=..., timeout=30.0)`

## Security Notes

⚠️ **Sensitive Data Protection:**
- All examples use mock data only
- In production, sanitize all Sensitive Data before passing to SDK
- SDK does NOT automatically mask Sensitive Data
- Application is responsible for Sensitive Data sanitization

⚠️ **Credentials:**
- Never log tokens, API keys, or passwords
- Use environment variables for secrets
- ADC tokens are handled securely by google-auth

## Other Examples

### vendor-bridge/
Demonstrates the vendor API bridge pattern for integrating with third-party ML platforms. Uses mock vendor API for portability. This example serves a different use case (vendor integration) and is kept for reference.

## Next Steps

1. **Run all examples** to understand different SDK features
2. **Review the notebook** at `notebooks/end-to-end-demo.ipynb` for additional patterns
3. **Adapt to your model** - Copy the example closest to your use case
4. **Deploy to production** with proper Sensitive Data safeguards and configuration

## Support

- **SDK Documentation:** `../README.md`
- **API Contracts:** `../../docs/api-contracts.md`
- **Architecture:** `../../docs/architecture.md`
- **GCP Dev Environment:** `../../docs/gcp-dev-environment.md`

## Key Design Patterns

### Pattern 1: GCP Auth and Connectivity

```python
from shared import print_gcp_auth_status, print_collector_status

if not print_gcp_auth_status():
    sys.exit(1)

if not print_collector_status():
    sys.exit(1)
```

### Pattern 2: Registry Integration (Filter by Type + Fetch)

```python
from mca_sdk.registry import RegistryClient

# Step 1: Create registry client
registry_client = RegistryClient(
    url=REGISTRY_URL,
    use_gcp_auth=True,
    timeout=10.0
)

# Step 2: Filter by model type (demonstrates discovery)
filtered_models = registry_client.list_configs(
    model_category="internal",
    model_type="classification",  # or "generative", "agentic"
    limit=5
)
print(f"Found {filtered_models['total']} classification models")

# Step 3: Fetch specific model config
model_config = registry_client.fetch_model_config(model_id)
```

### Pattern 3: SDK Initialization

```python
from mca_sdk import MCAClient

client = MCAClient(
    service_name=model_config.service_name,
    team_name=model_config.team_name,
    model_id=model_id,
    collector_endpoint=COLLECTOR_HTTP_ENDPOINT,
    allow_insecure_collector=True,
    registry_url=REGISTRY_URL,
    use_gcp_auth=True
)
```

### Pattern 4: Telemetry Recording

```python
# Predictions
client.record_prediction(latency=0.15, prediction_id="pred-001")

# Custom metrics
client.record_counter("model.cache_hits", 10)
client.record_gauge("model.queue_size", 5)
client.record_metric("model.confidence", 0.89, metric_type="histogram")

# Errors
client.record_error(error=e, error_type="validation_error", severity="warning")
```

### Pattern 5: Flush and Verify

```python
# Flush
client.shutdown(timeout_millis=10000)
registry_client.close()

# Wait for propagation
time.sleep(30)

# Verify in GCP Console
print("Verify at: https://console.cloud.google.com/logs/query")
```

---

**Remember:** All examples use the GCP dev environment and require VPN access if working outside the GKE cluster.
