# Vendor Model Bridge Example

This example demonstrates how to create a bridge to poll a third-party vendor's API, transform the data into OTLP metrics, and send them to the OpenTelemetry Collector.

## ⚠️ When to Use This Example

**Use this example ONLY if:**
- ✅ Your model is deployed on a **third-party vendor platform** (AWS SageMaker, Azure ML, Google Vertex AI, etc.)
- ✅ The vendor platform **does NOT support OpenTelemetry** natively
- ✅ You need to **poll the vendor's REST API** to retrieve metrics
- ✅ You want to normalize vendor metrics into your observability stack

**Do NOT use this example if:**
- ❌ You're instrumenting an **internal BHSF model** (use `internal-model/`, `internal-genai/`, or `internal-agentic/` instead)
- ❌ Your model runs in **your own infrastructure** (use direct SDK instrumentation)
- ❌ You need to use the **BHSF Registration API** (this example doesn't integrate with it)

**Quick Guide:**
- **Internal models at BHSF** → Use `internal-model/`, `internal-genai/`, or `internal-agentic/`
- **External vendor platforms** → Use this `vendor-bridge/` example

## Overview

The vendor bridge pattern is used when:
- You have a model deployed on a third-party platform (AWS SageMaker, Azure ML, etc.)
- The platform doesn't natively support OpenTelemetry
- You need to poll the vendor's API to retrieve metrics
- You want to normalize vendor metrics into your observability stack

## Architecture Difference

**Vendor Bridge (this example):**
```
Vendor API (AWS/Azure/GCP) → Poll every 30s → Transform JSON → OTLP → Collector
```

**Direct Instrumentation (other examples):**
```
Your Model Code → MCA SDK → OTLP → Collector → GCP Services
                    ↓
              Registry API (config)
```

## Features

- **Automatic Polling**: Configurable poll interval (default: 30s)
- **Retry Logic**: Exponential backoff for transient failures
- **State Persistence**: SQLite-based storage for cumulative metrics
- **Health Checks**: Comprehensive health monitoring
- **Graceful Shutdown**: Clean resource cleanup on exit
- **Thread-Safe**: Background polling with proper synchronization
- **No Registry Integration**: Designed for external vendors, not BHSF internal registry

## Running the Example

### 1. Start the Mock Vendor API and Collector

```bash
cd mca-prototype
docker-compose up -d mock-vendor-api otel-collector
```

This starts:
- Mock vendor API on `http://localhost:8080`
- OTel Collector on `http://localhost:4318` (OTLP/HTTP)

### 2. Run the Bridge

```bash
# From repository root
python mca-prototype/sdk-examples/vendor-bridge/api_to_otlp_bridge.py
```

### 3. Observe Metrics

The bridge will poll the vendor API every 30 seconds, transform JSON metrics to OTLP format, and send them to the collector.

## Using GCP Development Infrastructure

To test against the real GCP dev environment instead of a local collector:

```bash
# Use Docker Compose override
docker-compose -f ../../docker-compose.yml -f ../../docker-compose.gcp-dev.yml up vendor-bridge

# Or set environment variables for local testing
export MCA_COLLECTOR_ENDPOINT=http://10.164.76.55:4318
export MCA_ALLOW_INSECURE_COLLECTOR=true
export MCA_VENDOR_API_URL=http://localhost:8080/metrics
python api_to_otlp_bridge.py
```

See [GCP Dev Environment Guide](../../../docs/gcp-dev-environment.md) for complete setup instructions and troubleshooting.

### Verification

After running against GCP dev, verify telemetry in GCP Console:

```bash
# View logs
gcloud logging read "logName=projects/bhsf-ai-team-projects/logs/emms-model-telemetry" --limit=10

# View traces
gcloud trace list --project=bhsf-ai-team-projects --limit=10
```

Or use the GCP Console:
- **Logs**: [https://console.cloud.google.com/logs/query?project=bhsf-ai-team-projects](https://console.cloud.google.com/logs/query?project=bhsf-ai-team-projects)
- **Traces**: [https://console.cloud.google.com/traces/list?project=bhsf-ai-team-projects](https://console.cloud.google.com/traces/list?project=bhsf-ai-team-projects)

### Troubleshooting GCP Connectivity

**Cannot connect to 10.164.76.55:**
- Verify VPC connectivity or VPN access
- Test with: `curl http://10.164.76.55:4318/`

**Telemetry not appearing in GCP Console:**
- Verify collector endpoint is set correctly
- Check collector health: `curl http://10.164.76.55:13133/health/status`
- Wait 15 seconds for batch processing

## Configuration

Configure via environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `MCA_VENDOR_API_URL` | `http://localhost:8080/metrics` | Vendor API endpoint |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTel Collector endpoint |
| `MCA_VENDOR_BRIDGE_STATE_FILE` | None (in-memory) | Path to SQLite state database |
| `MCA_VENDOR_BRIDGE_MAX_ITERATIONS` | None (continuous) | Max polls before stopping |
| `MCA_ALLOW_INSECURE_COLLECTOR` | `false` | Allow HTTP for non-localhost |

### Example: Production Configuration

```bash
export MCA_VENDOR_API_URL="https://api.vendor.com/models/fraud-detection/metrics"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel-collector.mycompany.com:4318"
export MCA_VENDOR_BRIDGE_STATE_FILE="/var/lib/mca/vendor_state.db"

python api_to_otlp_bridge.py
```

## State Persistence

### Why State Persistence Matters

Vendor APIs often return **delta metrics** (changes since last poll). To convert these to **cumulative counters** (required by OTLP), the bridge must:
1. Load previous cumulative values from storage
2. Add the delta to the cumulative value
3. Save the new cumulative value
4. Export the cumulative value to OTLP

**Without state persistence**, cumulative metrics reset to zero on restart, breaking dashboards and alerts.

### SQLite State Manager

The example uses `SQLiteStateManager` from the MCA SDK for production-grade persistence:

```python
from mca_sdk.integrations import SQLiteStateManager

manager = SQLiteStateManager("/var/lib/mca/state.db")
manager.connect()

# Load previous state
state = manager.load_state()
predictions = state.get("cumulative_predictions", 0)

# Update with delta
predictions += delta_from_api

# Save new state
manager.save_state({"cumulative_predictions": predictions})
manager.close()
```

**Features:**
- **Atomic writes**: Uses SQLite transactions
- **Crash recovery**: WAL mode for durability
- **Thread-safe**: Can be used from multiple threads
- **Automatic schema**: Creates table on first use

### In-Memory vs Persistent Storage

| Mode | Use Case | Data Loss Risk |
|------|----------|----------------|
| **In-Memory** (`state_file=None`) | Development, testing | High - lost on restart |
| **SQLite** (`state_file="/path/to/db"`) | Production | Low - survives restarts |

**⚠️ WARNING**: In-memory storage is **NOT suitable for production**. Always configure `MCA_VENDOR_BRIDGE_STATE_FILE` in production environments.

## Production Considerations

### 1. State Persistence (Critical)

**Always configure state file in production:**
```bash
export MCA_VENDOR_BRIDGE_STATE_FILE="/var/lib/mca/state.db"
```

**Use persistent volumes in Kubernetes:**
```yaml
volumeMounts:
  - name: state-storage
    mountPath: /var/lib/mca
volumes:
  - name: state-storage
    persistentVolumeClaim:
      claimName: vendor-bridge-state
```

### 2. HTTPS Enforcement

Production endpoints must use HTTPS:
```bash
# ✅ Good
export MCA_VENDOR_API_URL="https://api.vendor.com/metrics"

# ❌ Bad (will fail unless allow_insecure=true)
export MCA_VENDOR_API_URL="http://api.vendor.com/metrics"
```

### 3. Error Handling

The bridge includes:
- **Retry logic**: 3 attempts with exponential backoff
- **Circuit breaker**: Backs off after consecutive failures
- **Health checks**: Detects and reports unhealthy state

## Docker Deployment

### Development vs Production Images

The included `Dockerfile` serves **dual purposes** for demonstration:
1. Running the mock vendor API (development/testing)
2. Running the vendor bridge (production pattern)

**For production deployments**, create separate, minimal images:

#### Production Bridge Image (Recommended)

```dockerfile
FROM python:3.11-slim

WORKDIR /app

# Install only production dependencies
COPY requirements-base.txt .
RUN pip install --no-cache-dir -r requirements-base.txt && \
    pip install --no-cache-dir mca-sdk[vendor]

# Copy only bridge code (not mock API)
COPY api_to_otlp_bridge.py .

# Use non-root user
RUN useradd -m -u 1000 bridge && chown -R bridge:bridge /app
USER bridge

# State persistence volume
VOLUME ["/app/state"]

CMD ["python", "api_to_otlp_bridge.py"]
```

#### Development Image (Current Dockerfile)

The current `Dockerfile` includes both the mock API and bridge for local testing. This is intentional for the example but should not be used in production.

**Key differences:**

| Aspect | Development Image | Production Image |
|--------|------------------|------------------|
| **Size** | ~500MB (includes dev tools) | ~200MB (minimal) |
| **Contents** | Mock API + Bridge + SDK source | Bridge only + PyPI SDK |
| **Use Case** | Local testing, CI/CD | Production deployment |
| **Security** | Runs as root (acceptable for dev) | Non-root user required |

### Production Deployment Example

```yaml
# kubernetes/vendor-bridge-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vendor-bridge
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: bridge
        image: myregistry/vendor-bridge:1.0.0
        env:
        - name: MCA_VENDOR_API_URL
          value: "https://api.vendor.com/metrics"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "https://otel-collector:4318"
        - name: MCA_VENDOR_BRIDGE_STATE_FILE
          value: "/app/state/vendor_state.db"
        volumeMounts:
        - name: state
          mountPath: /app/state
      volumes:
      - name: state
        persistentVolumeClaim:
          claimName: vendor-bridge-state
```

## See Also

- [VendorBridge API Documentation](../../docs/api/mca_sdk.html#VendorBridge)
- [SQLiteStateManager API Documentation](../../docs/api/mca_sdk.html#SQLiteStateManager)
- [MCA SDK Configuration Guide](../../docs/configuration.md)
