# Internal Model Example - Hospital Readmission Predictor

Demonstrates comprehensive MCA SDK usage for a traditional ML classification model with complete Registry API integration. Uses Diabetes 130-US Hospitals dataset for 30-day readmission prediction.

## Features Demonstrated

- ✅ GCP Authentication (Application Default Credentials)
- ✅ OTel Collector connectivity verification
- ✅ Registry API integration (filtering by model type + config fetch)
- ✅ Input/output capture with `@predict` decorator
- ✅ Manual prediction recording with threshold checking
- ✅ Custom operation tracing with `trace()` context manager
- ✅ Custom business metrics (counter, gauge, histogram)
- ✅ Error tracking and reporting
- ✅ Telemetry verification in GCP services

## Prerequisites

### 1. Install Dependencies

```bash
pip install mca-sdk[gcp,genai]>=0.6.1
pip install ucimlrepo  # For dataset download
```

### 2. Download Training Dataset

This example uses the **Diabetes 130-US Hospitals dataset** (101,766 hospital encounters) for training a 30-day readmission prediction model.

#### Option A: Automatic Download (Recommended)

The training script will automatically attempt to download the dataset using the `ucimlrepo` package:

```bash
python train_model.py
```

If successful, the dataset will be fetched directly from the UCI ML Repository.

#### Option B: Manual Download (For Network-Restricted Environments)

If automatic download fails (e.g., SSL certificate issues, firewall restrictions, Google Workbench):

1. **Download the dataset manually:**
   - Visit: [https://archive.ics.uci.edu/dataset/296/diabetes+130-us+hospitals+for+years+1999-2008](https://archive.ics.uci.edu/dataset/296/diabetes+130-us+hospitals+for+years+1999-2008)
   - Click "Download" button to get `diabetes+130-us+hospitals+for+years+1999-2008.zip`

2. **For Google Workbench:**
   - Upload the ZIP file to your Workbench instance
   - Open a terminal in Workbench
   - Navigate to the example directory:
     ```bash
     cd ~/sdk/mca-prototype/sdk-examples/internal-model
     ```
   - Extract the dataset:
     ```bash
     unzip diabetes*.zip -d dataset_diabetes/
     ```

3. **For Local Development:**
   - Place the downloaded ZIP in the `internal-model/` directory
   - Extract:
     ```bash
     unzip diabetes+130-us+hospitals+for+years+1999-2008.zip -d dataset_diabetes/
     ```

4. **Verify extraction:**
   ```bash
   ls -la dataset_diabetes/
   # Should show: diabetic_data.csv and IDS_mapping.csv
   ```

5. **Train the model:**
   ```bash
   python train_model.py
   ```

The script will detect the local dataset automatically and load it instead of attempting to download.

**Note:** If the dataset cannot be downloaded or found locally, the script will fall back to synthetic data (10,000 samples) for demonstration purposes. However, the synthetic data is not representative of real clinical patterns and will result in lower model performance.

### 3. Configure GCP Authentication

```bash
gcloud auth application-default login
```

This creates Application Default Credentials (ADC) that the SDK uses to:
- Authenticate with Registry API
- Verify GCP project access

### 4. VPN Connection (if outside GKE cluster)

The GCP dev OTel Collector (`10.164.76.55`) is only accessible from:
- Inside the GKE cluster
- Via VPN connection

**Connect to VPN before running this example if you're working outside the cluster.**

## Usage

### Step 1: Train the Model (First Time Only)

Before running the example, train the readmission prediction model:

```bash
python train_model.py
```

This will:
1. Load the Diabetes 130-US Hospitals dataset (local or download)
2. Train a Random Forest classifier
3. Save the model to `readmission_model.pkl`

Expected output:
```
=== Training Hospital Readmission Prediction Model ===

Loading dataset from local file: dataset_diabetes/diabetic_data.csv
✓ Loaded 101766 hospital encounters from local dataset
Using 10000 encounters for training
Features: time_in_hospital, num_lab_procedures, num_procedures, num_medications, number_diagnoses, age_numeric
Readmission rate (30-day): 11.2%

Training Random Forest classifier...

=== Model Performance ===
Accuracy:  0.892
Precision: 0.687
Recall:    0.423
F1 Score:  0.524

✓ Model saved to readmission_model.pkl
```

**Generated Files:**
The training script creates three files:
1. `readmission_model.pkl` - Trained Random Forest model (~4.8 MB)
2. `inference_dataset_with_actuals.csv` - Test set with ground truth labels (~2,000 samples)
3. `dataset_info.json` - Dataset metadata and model metrics

## Datasets

### Training Dataset
**Source:** Diabetes 130-US Hospitals dataset (UCI ML Repository #296)
- **Records:** 101,766 hospital encounters (1999-2008)
- **File:** `dataset_diabetes/diabetic_data.csv` (95 MB)
- **Features Used:**
  - `time_in_hospital`: Length of stay (1-14 days)
  - `num_lab_procedures`: Number of lab procedures (1-132)
  - `num_procedures`: Number of procedures (0-6)
  - `num_medications`: Number of medications (1-81)
  - `number_diagnoses`: Number of diagnoses (1-16)
  - `age_numeric`: Patient age (5-95 years)
- **Target:** `readmitted` ('<30', '>30', 'NO') converted to binary `readmitted_30` (1=readmitted within 30 days)

### Inference Dataset with Actuals
**File:** `inference_dataset_with_actuals.csv` (auto-generated by `train_model.py`)
- **Records:** ~2,000 test samples (20% stratified split)
- **Purpose:** Make predictions on deployed model and compare against ground truth
- **Columns:**
  - 6 feature columns (same as training)
  - `actual_readmission` - Ground truth label (0 or 1)
  - `encounter_id` - Unique identifier

**Use Cases:**
- Validate deployed model performance
- Compare predictions to actuals for accuracy calculation
- Populate monitoring dashboards with realistic data
- Detect data drift in production

**Example Usage:**
```python
import pandas as pd
from sklearn.metrics import accuracy_score

# Load inference dataset
df = pd.read_csv('inference_dataset_with_actuals.csv')

# Separate features and actuals
features = df[['time_in_hospital', 'num_lab_procedures', 'num_procedures',
               'num_medications', 'number_diagnoses', 'age_numeric']]
actuals = df['actual_readmission']

# Make predictions via deployed endpoint
for _, row in features.iterrows():
    prediction = endpoint.predict(instances=[{"features": row.to_dict()}])
    # Compare prediction to actual

# Calculate accuracy
accuracy = accuracy_score(actuals, predictions)
print(f"Deployed model accuracy: {accuracy:.3f}")
```

### Dataset Info
**File:** `dataset_info.json` (auto-generated)
- Training/test split sizes
- Feature names
- Readmission rates (train and test)
- Baseline model metrics

### Step 2: Run Against GCP Dev Collector (Default)

The example connects to the GCP dev collector by default:

```bash
python instrumented_model.py
```

**Note:** Requires VPN access if running outside the GKE cluster.

### Step 3: Run Against Local Collector (Optional)

For local development without GCP access:

```bash
docker-compose up -d otel-collector
export MCA_COLLECTOR_ENDPOINT=http://localhost:4318
python instrumented_model.py
```

## Using GCP Development Infrastructure

To test against the real GCP dev environment instead of a local collector:

```bash
# Use Docker Compose override
docker-compose -f ../../docker-compose.yml -f ../../docker-compose.gcp-dev.yml up internal-model

# Or set environment variables for local testing
export MCA_COLLECTOR_ENDPOINT=http://10.164.76.55:4318
export MCA_ALLOW_INSECURE_COLLECTOR=true
python instrumented_model.py
```

See [GCP Dev Environment Guide](../../../docs/gcp-dev-environment.md) for complete setup instructions and troubleshooting.

### Verification

After running against GCP dev, verify telemetry in GCP Console:

```bash
# View logs
gcloud logging read "logName=projects/bhsf-ai-team-projects/logs/emms-model-telemetry" --limit=10

# View traces
gcloud trace list --project=bhsf-ai-team-projects --limit=10
```

Or use the GCP Console:
- **Logs**: [https://console.cloud.google.com/logs/query?project=bhsf-ai-team-projects](https://console.cloud.google.com/logs/query?project=bhsf-ai-team-projects)
- **Traces**: [https://console.cloud.google.com/traces/list?project=bhsf-ai-team-projects](https://console.cloud.google.com/traces/list?project=bhsf-ai-team-projects)

### Troubleshooting GCP Connectivity

**Cannot connect to 10.164.76.55:**
- Verify VPC connectivity or VPN access
- Test with: `curl http://10.164.76.55:4318/`

**Telemetry not appearing in GCP Console:**
- Verify collector endpoint is set correctly
- Check collector health: `curl http://10.164.76.55:13133/health/status`
- Wait 15 seconds for batch processing

## Expected Output

The script will:
1. Verify GCP authentication and collector connectivity
2. Filter registry for classification models (demonstrates discovery)
3. Fetch specific model configuration
4. Initialize SDK with registry config
5. Demonstrate input/output capture with `@predict` decorator
6. Record manual predictions with threshold checking
7. Create custom operation spans with `trace()`
8. Record custom metrics (counter, gauge, histogram)
9. Track errors with context
10. Flush telemetry to collector
11. Print verification URLs for GCP Console

## Monitoring

View metrics and traces in your configured backend:
- Google-Managed Prometheus (metrics storage)
- Cloud Monitoring (dashboards)
- Cloud Logging (logs)
- Cloud Trace (traces)

## Files

- `instrumented_model.py` - Main example script with full SDK integration
- `train_model.py` - Training script for the readmission prediction model
- `requirements.txt` - Python dependencies
- `Dockerfile` - Containerized version
- `dataset_diabetes/` - Dataset directory (created after extraction)
- `readmission_model.pkl` - Trained model file (created after training)

## Troubleshooting

### Dataset Download Issues

**SSL Certificate Errors in Google Workbench:**
- Use manual download method (Option B above)
- Upload ZIP file directly to Workbench

**"Error connecting to server" from ucimlrepo:**
- Network firewall may be blocking UCI repository
- Use manual download method
- Script will fall back to synthetic data if download fails

**Synthetic data warning appears:**
- Ensure `dataset_diabetes/diabetic_data.csv` exists
- Re-run dataset extraction steps
- Verify file permissions with `ls -la dataset_diabetes/`

### Model Training Errors

**"FileNotFoundError: readmission_model.pkl":**
- Run `python train_model.py` first to create the model
- Ensure training completed successfully

### Connection Refused Error

If you see "Connection refused" errors:
- Ensure OpenTelemetry Collector is running
- Check collector endpoint: `http://localhost:4318`
- Verify Docker Compose services are up

### Import Errors

Ensure all dependencies are installed:
```bash
pip install -r requirements.txt
```
