# GenAI/LLM Example - Clinical Documentation Assistant

Demonstrates GenAI-specific SDK features with complete Registry API integration for LLM-based applications.

## Overview

This example shows how to instrument LLM applications using the MCA SDK, with focus on:
- Registry filtering for GenAI models
- Explicit token and cost tracking with `track_llm_completion`
- Token and cost tracking with threshold enforcement
- Custom GenAI metrics
- Error tracking for LLM API failures

## Features Demonstrated

- ✅ GCP Authentication (Application Default Credentials)
- ✅ OTel Collector connectivity verification
- ✅ Registry API integration (filtering by model type + config fetch)
- ✅ `create_genai_client()` for LLM-specific initialization
- ✅ Simulated LiteLLM integration and OpenTelemetry automatic instrumentation
- ✅ `track_llm_completion()` for token and cost tracking
- ✅ `estimate_openai_cost()` for cost calculation
- ✅ Cost threshold checking from registry config
- ✅ Custom GenAI metrics (cache hits, rate limits, quality scores)
- ✅ LLM error tracking with context
- ✅ Telemetry verification in GCP services

## File

- **`litellm_instrumented.py`** - Main example demonstrating manual GenAI instrumentation
- **`requirements.txt`** - Python dependencies
- **`README.md`** - This file

## Prerequisites

### 1. Install Dependencies

```bash
pip install mca-sdk[gcp,genai]>=0.6.1
```

### 2. Configure GCP Authentication

```bash
gcloud auth application-default login
```

### 3. VPN Connection (if outside GKE cluster)

The GCP dev OTel Collector (`10.164.76.55`) is only accessible from inside the GKE cluster or via VPN.

## Usage

### Run Against GCP Dev Collector (Default)

```bash
python litellm_instrumented.py
```

**Note:** Requires VPN access if running outside the GKE cluster.

### Run Against Local Collector (Optional)

```bash
docker-compose up -d otel-collector
export MCA_COLLECTOR_ENDPOINT=http://localhost:4318
python litellm_instrumented.py
```

## Expected Output

The script will:
1. Verify GCP authentication and collector connectivity
2. Filter registry for generative models (demonstrates discovery)
3. Fetch specific model configuration with cost thresholds
4. Initialize GenAI client with registry config
5. Demonstrate `@predict` decorator for LLM functions
6. Process clinical notes with token and cost tracking
7. Check cost against registry thresholds
8. Record custom GenAI metrics (cache hits, rate limits, quality)
9. Track LLM errors with context
10. Flush telemetry to collector
11. Print verification URLs for GCP Console

## Running with OpenTelemetry Collector

To see telemetry exported to a real collector:

### Step 1: Start the Collector

```bash
cd mca-prototype
docker-compose up -d otel-collector
```

This starts the OpenTelemetry Collector on:
- OTLP/HTTP: `http://localhost:4318`
- OTLP/gRPC: `http://localhost:4317`

### Step 2: Run the Example

```bash
cd sdk-examples/internal-genai
python litellm_callback_example.py
```

## Using GCP Development Infrastructure

To test against the real GCP dev environment instead of a local collector:

```bash
# Use Docker Compose override
docker-compose -f ../../docker-compose.yml -f ../../docker-compose.gcp-dev.yml up genai-assistant

# Or set environment variables for local testing
export MCA_COLLECTOR_ENDPOINT=http://10.164.76.55:4318
export MCA_ALLOW_INSECURE_COLLECTOR=true
python litellm_callback_example.py
```

See [GCP Dev Environment Guide](../../../docs/gcp-dev-environment.md) for complete setup instructions and troubleshooting.

### Verification

After running against GCP dev, verify telemetry in GCP Console:

```bash
# View logs
gcloud logging read "logName=projects/bhsf-ai-team-projects/logs/emms-model-telemetry" --limit=10

# View traces
gcloud trace list --project=bhsf-ai-team-projects --limit=10
```

Or use the GCP Console:
- **Logs**: [https://console.cloud.google.com/logs/query?project=bhsf-ai-team-projects](https://console.cloud.google.com/logs/query?project=bhsf-ai-team-projects)
- **Traces**: [https://console.cloud.google.com/traces/list?project=bhsf-ai-team-projects](https://console.cloud.google.com/traces/list?project=bhsf-ai-team-projects)

### Troubleshooting GCP Connectivity

**Cannot connect to 10.164.76.55:**
- Verify VPC connectivity or VPN access
- Test with: `curl http://10.164.76.55:4318/`

**Telemetry not appearing in GCP Console:**
- Verify collector endpoint is set correctly
- Check collector health: `curl http://10.164.76.55:13133/health/status`
- Wait 15 seconds for batch processing

### Step 3: Verify Telemetry

Check the collector logs:

```bash
docker-compose logs -f otel-collector
```

You should see OTLP exports with GenAI metrics and traces.

## Verification

### Verify Metrics Are Emitted

When running the example, you should see:

1. **Console Output:** Each iteration logs the prompt and response
2. **Structured Logs:** MCA SDK logs LLM requests with token/cost/latency data
3. **Collector Logs:** OTLP exports visible in `docker-compose logs otel-collector`

### Verify Traces Are Created

Traces are exported to the OpenTelemetry Collector and can be viewed in:
- **Cloud Trace** (Google Cloud)
- **Jaeger** (open-source tracing UI)
- **Zipkin** (open-source tracing UI)

Each trace span will have:
- Span name: `llm.{model}` (e.g., `llm.gpt-4`)
- Attributes: Model, provider, tokens, cost, latency
- Status: OK or ERROR

### Verify Cost Calculation

**Mock mode limitation:** Cost calculation is unavailable because LiteLLM's mock responses lack the required pricing metadata fields. This is a fundamental limitation of mock mode and cannot be fully addressed without real API responses.

**For production:** Real API calls provide accurate cost calculation using LiteLLM's pricing data and actual usage information from the provider.

## Mock Mode vs Production Mode

| Feature | Mock Mode | Production Mode |
|---------|-----------|-----------------|
| API Calls | No (simulated) | Yes (real API) |
| Token Counts | Simulated | Real from API |
| Cost Calculation | $0.00 or None | Accurate USD cost |
| Latency | Simulated | Real API latency |
| API Keys Required | No | Yes |
| Use Case | Testing, demos | Production apps |

## Troubleshooting

### Issue: "MCALiteLLMCallback requires the 'litellm' package"

**Solution:** Install the GenAI extra:

```bash
pip install mca-sdk[genai]
```

Or install LiteLLM directly:

```bash
pip install litellm>=1.61.15
```

### Issue: "Connection refused" to OpenTelemetry Collector

**Symptoms:**
```
Failed to export telemetry: Connection refused (localhost:4318)
```

**Solutions:**
1. **Start the collector:**
   ```bash
   cd mca-prototype
   docker-compose up -d otel-collector
   ```

2. **Verify collector is running:**
   ```bash
   docker-compose ps
   curl http://localhost:4318/v1/traces
   ```

3. **SDK continues running:** The SDK uses graceful degradation - if the collector is unavailable, the example continues without crashing.

### Issue: Cost is $0.00 in mock mode

**Explanation:** Mock mode limitation - LiteLLM's simulated responses lack the pricing metadata required for cost calculation. This is expected behavior and not a bug.

**Verification:** Run with a real API key to see accurate costs:
```bash
export OPENAI_API_KEY="sk-..."
# Do NOT set MOCK_LLM (production mode)
python3 litellm_callback_example.py
```

### Issue: How to switch between mock and production mode?

**Mock Mode** (no API keys needed):
```bash
export MOCK_LLM=true
python3 litellm_callback_example.py
# Console shows: 🔵 Running in MOCK mode (no real API calls)
```

**Production Mode** (requires API keys):
```bash
export OPENAI_API_KEY="sk-..."
# Do NOT set MOCK_LLM, or set it to false
python3 litellm_callback_example.py
# Console shows: 🟢 Running in PRODUCTION mode (real API calls)
```

**How it works internally:**
- When `MOCK_LLM=true`, the script sets `LITELLM_MODE=COMPLETION` internally
- This is the official LiteLLM approach for mock/test mode
- LiteLLM returns simulated responses without making API calls
- All telemetry is still captured normally (tokens, latency, traces)
- No code edits required to switch modes!

### Issue: "Multiple MCALiteLLMCallback instances detected"

**Symptoms:**
```
WARNING: Multiple MCALiteLLMCallback instances detected in litellm.callbacks
```

**Explanation:** LiteLLM uses a global callback registry. If you create multiple callback instances, they will all fire for every request, causing duplicate metrics.

**Solution:** Only create ONE callback instance per application process:

```python
# GOOD (one callback, using append)
callback = MCALiteLLMCallback(client)
litellm.callbacks.append(callback)

# ALSO GOOD (check for duplicates first)
if not any(isinstance(cb, MCALiteLLMCallback) for cb in litellm.callbacks):
    litellm.callbacks.append(callback)

# BAD (overwrites existing callbacks)
litellm.callbacks = [callback]  # Destroys any pre-existing callbacks

# BAD (multiple callbacks)
callback1 = MCALiteLLMCallback(client1)
callback2 = MCALiteLLMCallback(client2)  # Will cause duplicates!
litellm.callbacks.extend([callback1, callback2])
```

### Issue: Traces not appearing in Cloud Trace

**Checklist:**
1. Verify collector is exporting to Google Cloud:
   ```bash
   # Check collector config
   cat config/otel-collector-config.yaml
   ```

2. Verify GCP credentials are set:
   ```bash
   echo $GOOGLE_APPLICATION_CREDENTIALS
   ```

3. Check collector logs for export errors:
   ```bash
   docker-compose logs otel-collector | grep -i error
   ```

### Issue: Import error "cannot import name 'MCALiteLLMCallback'"

**Solution:** Ensure you're using MCA SDK 0.3.0 or later:

```bash
pip install --upgrade mca-sdk[genai]
```

Verify version:

```bash
python -c "import mca_sdk; print(mca_sdk.__version__)"
```

## Architecture

### How MCALiteLLMCallback Works

```
┌─────────────────────────────────────────────────────────────────┐
│ YOUR APPLICATION                                                │
│   import litellm                                                │
│   from mca_sdk.integrations import MCALiteLLMCallback          │
│                                                                  │
│   callback = MCALiteLLMCallback(client)                        │
│   litellm.callbacks = [callback]  ← Register once              │
│                                                                  │
│   response = litellm.completion(...)  ← Automatically tracked! │
└─────────────────┬───────────────────────────────────────────────┘
                  │
                  │ (1) log_pre_api_call → Start trace span
                  │ (2) LLM API call (OpenAI, Claude, etc.)
                  │ (3) log_success_event → Record metrics, end span
                  │
                  ▼
┌─────────────────────────────────────────────────────────────────┐
│ MCA SDK (OpenTelemetry)                                         │
│   - Record token counters                                       │
│   - Record cost histogram                                       │
│   - Record latency histogram                                    │
│   - Create trace span with attributes                           │
│   - Emit structured logs                                        │
└─────────────────┬───────────────────────────────────────────────┘
                  │
                  │ OTLP/HTTP or OTLP/gRPC
                  ▼
┌─────────────────────────────────────────────────────────────────┐
│ OPENTELEMETRY COLLECTOR                                         │
│   - Batch processor                                             │
│   - Attributes processor (add gcp.region, environment, etc.)    │
│   - Exporters (GMP, Cloud Trace, Cloud Logging, Langfuse)       │
└─────────────────────────────────────────────────────────────────┘
```

### Callback Lifecycle

1. **Registration:**
   ```python
   callback = MCALiteLLMCallback(client)
   litellm.callbacks.append(callback)  # Use append to respect existing callbacks
   ```

2. **Pre-API Call (`log_pre_api_call`):**
   - Extract model, temperature, max_tokens
   - Start trace span with attributes
   - Store span in active spans dict

3. **API Call:**
   - LiteLLM calls the provider (OpenAI, Anthropic, etc.)
   - Response includes token usage

4. **Post-API Call (`log_success_event` or `log_failure_event`):**
   - Extract token usage from response
   - Calculate cost using `litellm.completion_cost()`
   - Calculate latency
   - Record metrics (tokens, cost, latency)
   - End trace span with attributes
   - Emit structured log

5. **Export (Asynchronous):**
   - MCA SDK queues telemetry data for export
   - OTel exporters send data to collector in background
   - Collector forwards to backends (Google-Managed Prometheus for metrics, Cloud Logging for logs, Cloud Trace for traces, Langfuse for LLM observability)

## Known Limitations

### 1. Time To First Token (TTFT) Not Supported

TTFT tracking requires streaming mode and the `log_stream_event` hook, which is not currently implemented in `MCALiteLLMCallback`. This callback only supports non-streaming (batch) completion calls.

**Workaround:** For streaming use cases, use the manual instrumentation approach shown in `litellm_instrumented.py`.

### 2. Cost Calculation May Fail for Custom Models

The `litellm.completion_cost()` function relies on LiteLLM's internal pricing tables. If you're using:
- Custom fine-tuned models
- Local models (Ollama, LMStudio)
- New models not yet in LiteLLM's pricing data

Cost calculation may return `None` or `$0.00`. This is expected and the callback will gracefully skip the cost metric.

**Workaround:** Implement custom cost calculation logic outside the callback if needed.

### 3. Global Callback Registry

LiteLLM uses a global callback registry (`litellm.callbacks`), which means:
- Only ONE `MCALiteLLMCallback` instance should be registered per process
- Multiple instances cause duplicate metrics
- Cannot have different configurations for different LLM calls in the same process

This is a fundamental LiteLLM design constraint that cannot be fully mitigated without changes to LiteLLM itself.

## Security Considerations

### Sensitive Data Protection

The callback is **configured for data privacy by default to support HIPAA compliance requirements**:
- Prompts are NOT logged (may contain patient data)
- Completions are NOT logged (may contain Sensitive Data)
- Error messages are NOT logged (may contain user input)
- Only metadata is captured (model, tokens, cost, latency)

**Note:** HIPAA compliance depends on organizational usage and implementation. Software configuration alone cannot guarantee compliance.

**NEVER enable `record_prompts`, `record_completions`, or `record_error_messages` in production HIPAA environments** unless you have:
- Explicit approval from compliance team
- Proper Sensitive Data de-identification in place
- Secure log storage with access controls

### Credential Management

API keys should be set via environment variables, never hardcoded:

```python
# GOOD
export OPENAI_API_KEY="sk-..."
response = litellm.completion(model="gpt-4", ...)

# BAD (never hardcode)
response = litellm.completion(
    model="gpt-4",
    api_key="sk-hardcoded-key",  # NEVER DO THIS
    ...
)
```

## Additional Examples

### Example: Error Handling

The callback automatically tracks failures:

```python
try:
    response = litellm.completion(
        model="invalid-model",  # Will fail
        messages=[{"role": "user", "content": "Hello"}]
    )
except Exception as e:
    print(f"Error: {e}")
    # Error is automatically tracked by callback:
    # - genai.requests_total (status=error)
    # - genai.requests_failed_total
    # - Trace span with error status
```

### Example: Custom Attributes

Add custom attributes to the MCA client (not directly to the callback):

```python
client = create_genai_client(
    service_name="clinical-summarizer",
    llm_provider="openai",
    llm_model="gpt-4",
    team_name="clinical-ai",
    collector_endpoint="http://localhost:4318",
)

# Custom attributes on the client propagate to all metrics
client.meter.create_counter("custom.metric").add(1, attributes={
    "use_case": "patient_summarization",
    "department": "cardiology"
})
```

### Example: Multiple Models

The callback tracks all models used:

```python
# Register callback once (use append to respect existing callbacks)
callback = MCALiteLLMCallback(client)
litellm.callbacks.append(callback)

# All models are automatically tracked
response1 = litellm.completion(model="gpt-4", messages=[...])
response2 = litellm.completion(model="claude-3", messages=[...])
response3 = litellm.completion(model="gemini-pro", messages=[...])

# Metrics are tagged with model name for filtering
```

## Next Steps

1. **Run the example in mock mode** to verify setup
2. **Start OpenTelemetry Collector** to see telemetry exports
3. **Configure for production** with real API keys
4. **Integrate into your application** by copying the callback setup
5. **Monitor in production** using Cloud Monitoring dashboards for metrics

## References

- **MCA SDK Documentation:** `mca-prototype/README.md`
- **LiteLLM Documentation:** https://docs.litellm.ai/
- **OpenTelemetry Python:** https://opentelemetry.io/docs/languages/python/
- **MCA SDK Integrations:** `mca_sdk/integrations/`

## Support

For issues or questions:
- Review `mca-prototype/README.md`
- Check `tests/test_litellm_callback.py` for usage examples
- See `docs/api-contracts.md` for API specifications
