# Testing Java MCA SDK in Google Workbench

## Prerequisites
- Google Workbench instance with Java 11+ installed
- Maven 3.6+ available
- Docker and Docker Compose available
- Repository cloned to Workbench

## Step 1: Start OTel Collector

```bash
cd /home/coder/emms/sdk/mca-prototype
docker-compose up -d otel-collector

# Verify collector is running
curl http://localhost:13133/
# Should return: Alive!

# Check collector logs
docker-compose logs otel-collector
```

## Step 2: Build Java SDK

```bash
cd /home/coder/emms/sdk/mca-prototype/mca-sdk-java

# Clean and compile
mvn clean compile

# Expected output:
# [INFO] BUILD SUCCESS
# [INFO] Total time: ~15s
```

## Step 3: Run Unit Tests

```bash
# Run all unit tests
mvn test

# Run with coverage report
mvn test jacoco:report

# View coverage report
open target/site/jacoco/index.html

# Expected output:
# Tests run: 70+, Failures: 0, Errors: 0, Skipped: 0
# Coverage: >= 85%
```

## Step 4: Basic Integration Test

Create `src/test/java/integration/BasicIntegrationTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.RecordPredictionRequest;

/**
 * Basic integration test for MCA SDK with OTel Collector.
 */
public class BasicIntegrationTest {

    public static void main(String[] args) throws InterruptedException {
        System.out.println("=== Java MCA SDK Basic Integration Test ===\n");

        // Initialize client with try-with-resources
        try (MCAClient client = new MCAClient.Builder()
                .serviceName("java-test-model")
                .modelId("mdl-java-001")
                .modelVersion("1.0.0")
                .teamName("test-team")
                .otelEndpoint("http://localhost:4318")
                .build()) {

            System.out.println("✓ MCAClient initialized successfully");

            // Test 1: Record prediction with latency
            client.recordPrediction(RecordPredictionRequest.builder()
                    .latency(0.123)
                    .predictionId("pred-001")
                    .attribute("model_version", "v1.0")
                    .build());
            System.out.println("✓ Recorded prediction with latency");

            // Test 2: Record prediction without latency (defaults to 0.0)
            client.recordPrediction(RecordPredictionRequest.builder()
                    .predictionId("pred-002")
                    .attribute("custom_attribute", "test-value")
                    .build());
            System.out.println("✓ Recorded prediction without latency");

            // Test 3: Record with multiple attributes
            client.recordPrediction(RecordPredictionRequest.builder()
                    .latency(0.089)
                    .predictionId("pred-003")
                    .attribute("environment", "test")
                    .attribute("patient_age", 65)
                    .attribute("confidence", 0.92)
                    .build());
            System.out.println("✓ Recorded prediction with multiple attributes");

            // Test 4: Multiple rapid predictions
            for (int i = 0; i < 5; i++) {
                client.recordPrediction(RecordPredictionRequest.builder()
                        .latency(Math.random() * 0.5)
                        .predictionId("pred-batch-" + i)
                        .build());
            }
            System.out.println("✓ Recorded 5 batch predictions");

            System.out.println("\n⏳ Waiting 10 seconds for metrics to export...");
            Thread.sleep(10000);

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            e.printStackTrace();
            System.exit(1);
        }

        System.out.println("✓ Client shutdown successfully\n");
        System.out.println("Integration test completed!");
    }
}
```

### Run Basic Integration Test

```bash
# Compile and run
mvn compile exec:java -Dexec.mainClass="integration.BasicIntegrationTest"

# Expected output:
# === Java MCA SDK Basic Integration Test ===
# ✓ MCAClient initialized successfully
# ✓ Recorded prediction with latency
# ✓ Recorded prediction without latency
# ✓ Recorded prediction with multiple attributes
# ✓ Recorded 5 batch predictions
# ⏳ Waiting 10 seconds for metrics to export...
# ✓ Client shutdown successfully
# Integration test completed!
```

## Step 5: Advanced Instrumentation Test

Create `src/test/java/integration/AdvancedInstrumentationTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.instrumentation.SpanManager;

import java.util.HashMap;
import java.util.Map;

/**
 * Test advanced instrumentation features: tracing and custom metrics.
 */
public class AdvancedInstrumentationTest {

    public static void main(String[] args) {
        System.out.println("=== Advanced Instrumentation Test ===\n");

        try (MCAClient client = new MCAClient.Builder()
                .serviceName("advanced-test")
                .modelId("mdl-adv-001")
                .teamName("platform-team")
                .otelEndpoint("http://localhost:4318")
                .build()) {

            // Test 1: Manual tracing with SpanManager
            System.out.println("1. Testing manual tracing...");
            Map<String, Object> spanAttrs = new HashMap<>();
            spanAttrs.put("operation", "data_preprocessing");
            spanAttrs.put("user_id", "user-123");

            try (SpanManager span = client.trace("model.preprocess", spanAttrs)) {
                // Simulate work
                Thread.sleep(100);

                // Add dynamic attributes
                span.setAttribute("records_processed", 1000L)
                    .setAttribute("data_quality", "high")
                    .setAttribute("processing_time_ms", 100L);

                System.out.println("   ✓ Created span with dynamic attributes");
            }
            System.out.println("   ✓ Span automatically ended (try-with-resources)\n");

            // Test 2: Custom counter metric
            System.out.println("2. Testing custom counter metrics...");
            Map<String, Object> counterAttrs = new HashMap<>();
            counterAttrs.put("cache_type", "redis");
            counterAttrs.put("region", "us-east-1");

            client.recordCounter("model.cache_hits", 150.0, counterAttrs);
            client.recordCounter("model.api_requests_total", 500.0, null);
            System.out.println("   ✓ Recorded counter metrics\n");

            // Test 3: Custom histogram metric
            System.out.println("3. Testing custom histogram metrics...");
            Map<String, Object> histoAttrs = new HashMap<>();
            histoAttrs.put("model_version", "2.0");

            client.recordMetric("model.prediction_confidence", 0.92, "histogram", histoAttrs);
            client.recordMetric("model.data_quality_score", 0.87, "histogram", null);
            System.out.println("   ✓ Recorded histogram metrics\n");

            // Test 4: Custom gauge metric
            System.out.println("4. Testing custom gauge metrics...");
            Map<String, Object> gaugeAttrs = new HashMap<>();
            gaugeAttrs.put("queue_name", "predictions");

            client.recordGauge("model.queue_depth", 42.0, gaugeAttrs);
            client.recordGauge("model.memory_usage_mb", 512.0, null);
            System.out.println("   ✓ Recorded gauge metrics\n");

            // Test 5: Combined tracing + metrics
            System.out.println("5. Testing combined tracing + metrics...");
            Map<String, Object> predAttrs = new HashMap<>();
            predAttrs.put("model_version", "2.0");

            try (SpanManager span = client.trace("model.predict", predAttrs)) {
                long startTime = System.currentTimeMillis();
                Thread.sleep(50);
                long endTime = System.currentTimeMillis();
                double latency = (endTime - startTime) / 1000.0;

                client.recordCounter("model.predictions_total", 1.0, predAttrs);
                client.recordMetric("model.prediction_latency_seconds", latency, "histogram", predAttrs);

                span.setAttribute("prediction", "positive")
                    .setAttribute("confidence", 0.95)
                    .setAttribute("latency_ms", endTime - startTime);

                System.out.println("   ✓ Recorded metrics within span context");
            }
            System.out.println("   ✓ Span ended with complete context\n");

            Thread.sleep(5000);
            System.out.println("=== Advanced Instrumentation Test Complete ===");

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            e.printStackTrace();
            System.exit(1);
        }
    }
}
```

### Run Advanced Instrumentation Test

```bash
mvn compile exec:java -Dexec.mainClass="integration.AdvancedInstrumentationTest"

# Expected output:
# === Advanced Instrumentation Test ===
# 1. Testing manual tracing...
#    ✓ Created span with dynamic attributes
#    ✓ Span automatically ended (try-with-resources)
# 2. Testing custom counter metrics...
#    ✓ Recorded counter metrics
# 3. Testing custom histogram metrics...
#    ✓ Recorded histogram metrics
# 4. Testing custom gauge metrics...
#    ✓ Recorded gauge metrics
# 5. Testing combined tracing + metrics...
#    ✓ Recorded metrics within span context
#    ✓ Span ended with complete context
# === Advanced Instrumentation Test Complete ===
```

## Step 6: GenAI Integration Test

Create `src/test/java/integration/GenAIIntegrationTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.integrations.GenAIIntegration;

import java.util.HashMap;
import java.util.Map;

/**
 * Test GenAI integration features: token tracking, cost estimation.
 */
public class GenAIIntegrationTest {

    public static void main(String[] args) {
        System.out.println("=== GenAI Integration Test ===\n");

        try (MCAClient client = new MCAClient.Builder()
                .serviceName("genai-test")
                .modelId("mdl-genai-001")
                .teamName("ai-team")
                .otelEndpoint("http://localhost:4318")
                .build()) {

            GenAIIntegration genai = client.getGenAIIntegration();

            // Test 1: Record token usage
            System.out.println("1. Testing token usage tracking...");
            Map<String, Object> tokenAttrs = new HashMap<>();
            tokenAttrs.put("model", "gpt-4");
            tokenAttrs.put("provider", "openai");

            genai.recordTokenUsage(150, 75, tokenAttrs);
            System.out.println("   ✓ Recorded token usage (150 prompt, 75 completion)\n");

            // Test 2: Record cost
            System.out.println("2. Testing cost estimation...");
            Map<String, Object> costAttrs = new HashMap<>();
            costAttrs.put("model", "gpt-4");
            costAttrs.put("request_id", "req-123");

            genai.recordCost(0.045, costAttrs);
            System.out.println("   ✓ Recorded cost ($0.045)\n");

            // Test 3: Record request with failure
            System.out.println("3. Testing request tracking...");
            Map<String, Object> reqAttrs = new HashMap<>();
            reqAttrs.put("endpoint", "/v1/chat/completions");
            reqAttrs.put("status", "success");

            genai.recordRequest(reqAttrs);
            genai.recordFailedRequest(Map.of("error_type", "rate_limit"));
            System.out.println("   ✓ Recorded requests (1 success, 1 failure)\n");

            // Test 4: Multiple LLM calls simulation
            System.out.println("4. Testing multiple LLM call simulation...");
            for (int i = 0; i < 5; i++) {
                Map<String, Object> attrs = new HashMap<>();
                attrs.put("model", "gpt-3.5-turbo");
                attrs.put("call_number", i);

                long promptTokens = 50 + (long)(Math.random() * 100);
                long completionTokens = 30 + (long)(Math.random() * 50);
                double cost = (promptTokens * 0.0000015) + (completionTokens * 0.000002);

                genai.recordTokenUsage(promptTokens, completionTokens, attrs);
                genai.recordCost(cost, attrs);
                genai.recordRequest(attrs);
            }
            System.out.println("   ✓ Recorded 5 LLM calls with tokens and costs\n");

            Thread.sleep(5000);
            System.out.println("=== GenAI Integration Test Complete ===");

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            e.printStackTrace();
            System.exit(1);
        }
    }
}
```

### Run GenAI Integration Test

```bash
mvn compile exec:java -Dexec.mainClass="integration.GenAIIntegrationTest"

# Expected output:
# === GenAI Integration Test ===
# 1. Testing token usage tracking...
#    ✓ Recorded token usage (150 prompt, 75 completion)
# 2. Testing cost estimation...
#    ✓ Recorded cost ($0.045)
# 3. Testing request tracking...
#    ✓ Recorded requests (1 success, 1 failure)
# 4. Testing multiple LLM call simulation...
#    ✓ Recorded 5 LLM calls with tokens and costs
# === GenAI Integration Test Complete ===
```

## Step 7: Agentic AI Integration Test

Create `src/test/java/integration/AgenticTrackingTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.integrations.AgenticTracking;

import java.util.HashMap;
import java.util.Map;

/**
 * Test Agentic AI integration: goal lifecycle and tool execution tracking.
 */
public class AgenticTrackingTest {

    public static void main(String[] args) {
        System.out.println("=== Agentic AI Integration Test ===\n");

        try (MCAClient client = new MCAClient.Builder()
                .serviceName("agentic-test")
                .modelId("mdl-agent-001")
                .teamName("agent-team")
                .otelEndpoint("http://localhost:4318")
                .build()) {

            AgenticTracking agentic = client.getAgenticTracking();

            // Test 1: Goal lifecycle tracking
            System.out.println("1. Testing goal lifecycle...");
            Map<String, Object> goalAttrs = new HashMap<>();
            goalAttrs.put("goal_type", "information_retrieval");
            goalAttrs.put("user_id", "user-456");

            String goalId = agentic.startGoal("Retrieve patient medical history", goalAttrs);
            System.out.println("   ✓ Started goal: " + goalId);

            Thread.sleep(500);

            agentic.completeGoal(goalId, Map.of("status", "success", "records_found", 5));
            System.out.println("   ✓ Completed goal with results\n");

            // Test 2: Tool execution tracking
            System.out.println("2. Testing tool execution...");
            Map<String, Object> toolAttrs = new HashMap<>();
            toolAttrs.put("tool_name", "database_query");
            toolAttrs.put("query_type", "SELECT");

            double toolLatency = agentic.executeToolTimed("database_query", toolAttrs, () -> {
                try {
                    Thread.sleep(100);
                    return null;
                } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                }
            });
            System.out.println("   ✓ Executed tool with latency: " + toolLatency + "s\n");

            // Test 3: Nested goal with multiple tools
            System.out.println("3. Testing nested goal with tools...");
            Map<String, Object> nestedGoalAttrs = new HashMap<>();
            nestedGoalAttrs.put("goal_type", "data_analysis");

            String analysisGoalId = agentic.startGoal("Analyze patient cohort", nestedGoalAttrs);
            System.out.println("   ✓ Started analysis goal: " + analysisGoalId);

            // Execute multiple tools within goal context
            String[] tools = {"fetch_data", "preprocess", "compute_stats", "generate_report"};
            for (String tool : tools) {
                Map<String, Object> tAttrs = new HashMap<>();
                tAttrs.put("tool_name", tool);
                tAttrs.put("goal_id", analysisGoalId);

                agentic.executeToolTimed(tool, tAttrs, () -> {
                    try {
                        Thread.sleep(50 + (int)(Math.random() * 100));
                        return null;
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                });
                System.out.println("   ✓ Executed tool: " + tool);
            }

            agentic.completeGoal(analysisGoalId, Map.of("status", "success", "cohort_size", 150));
            System.out.println("   ✓ Completed analysis goal\n");

            Thread.sleep(5000);
            System.out.println("=== Agentic AI Integration Test Complete ===");

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            e.printStackTrace();
            System.exit(1);
        }
    }
}
```

### Run Agentic Tracking Test

```bash
mvn compile exec:java -Dexec.mainClass="integration.AgenticTrackingTest"

# Expected output:
# === Agentic AI Integration Test ===
# 1. Testing goal lifecycle...
#    ✓ Started goal: goal-<uuid>
#    ✓ Completed goal with results
# 2. Testing tool execution...
#    ✓ Executed tool with latency: 0.1xx s
# 3. Testing nested goal with tools...
#    ✓ Started analysis goal: goal-<uuid>
#    ✓ Executed tool: fetch_data
#    ✓ Executed tool: preprocess
#    ✓ Executed tool: compute_stats
#    ✓ Executed tool: generate_report
#    ✓ Completed analysis goal
# === Agentic AI Integration Test Complete ===
```

## Step 8: GCP Exporters Test

**Prerequisites:**
- GCP project with Cloud Trace and Cloud Logging APIs enabled
- Service account with permissions:
  - `roles/cloudtrace.agent` (Cloud Trace Writer)
  - `roles/logging.logWriter` (Cloud Logging Writer)
- `GOOGLE_APPLICATION_CREDENTIALS` environment variable set
- Optional dependencies added to pom.xml (already included as optional)

### Setup GCP Credentials

```bash
# Set GCP project ID
export GCP_PROJECT_ID="your-gcp-project-id"

# Set credentials path
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Verify credentials
gcloud auth application-default print-access-token
```

### Run GCP Exporter Example

```bash
mvn compile exec:java -Dexec.mainClass="examples.GCPExporterExample"

# Expected output:
# === MCA SDK GCP Exporter Example ===
# Example 1: Default OTLP export
#   Exported to OTLP endpoint: http://localhost:4318
# Example 2: GCP Cloud Trace export
#   Exported to: OTLP + GCP Cloud Trace
#   View traces: https://console.cloud.google.com/traces/list?project=...
# Example 3: GCP Cloud Logging export
#   Exported to: OTLP + GCP Cloud Logging
#   View logs: https://console.cloud.google.com/logs/query?project=...
# Example 4: All GCP exporters enabled
#   Exported to:
#     - OTLP: http://localhost:4318
#     - GCP Cloud Trace: https://console.cloud.google.com/traces?project=...
#     - GCP Cloud Logging: https://console.cloud.google.com/logs?project=...
# === Example Complete ===
```

### Verify GCP Cloud Trace

```bash
# Query traces via gcloud CLI
gcloud trace list --project=$GCP_PROJECT_ID --limit=10

# Or view in console:
# https://console.cloud.google.com/traces/list?project=<your-project-id>
```

### Verify GCP Cloud Logging

```bash
# Query logs via gcloud CLI
gcloud logging read "resource.type=global AND jsonPayload.service.name=\"example-model\"" \
  --project=$GCP_PROJECT_ID \
  --limit=10 \
  --format=json

# Or view in console:
# https://console.cloud.google.com/logs/query?project=<your-project-id>
```

## Step 9: Registry Integration Test

Create `src/test/java/integration/RegistryIntegrationTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.RecordPredictionRequest;

/**
 * Test registry integration with background refresh.
 */
public class RegistryIntegrationTest {

    public static void main(String[] args) {
        System.out.println("=== Registry Integration Test ===\n");

        // Note: Requires a running registry API
        String registryUrl = System.getenv("MCA_REGISTRY_URL");
        if (registryUrl == null) {
            registryUrl = "http://localhost:8080";  // Default local registry
        }

        try (MCAClient client = new MCAClient.Builder()
                .serviceName("registry-test")
                .modelId("mdl-reg-001")
                .otelEndpoint("http://localhost:4318")
                .registryUrl(registryUrl)
                .registryToken("test-token")
                .build()) {

            System.out.println("✓ Client initialized with registry integration");
            System.out.println("  Registry URL: " + registryUrl);

            // Record predictions
            for (int i = 0; i < 3; i++) {
                client.recordPrediction(RecordPredictionRequest.builder()
                        .latency(Math.random() * 0.3)
                        .predictionId("pred-reg-" + i)
                        .build());
            }
            System.out.println("✓ Recorded 3 predictions\n");

            // Background refresh happens automatically every 10 minutes (default)
            System.out.println("Background registry refresh is enabled");
            System.out.println("Config will be refreshed automatically every 10 minutes\n");

            Thread.sleep(5000);
            System.out.println("=== Registry Integration Test Complete ===");

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            System.err.println("Note: Registry integration requires a running registry API");
            e.printStackTrace();
        }
    }
}
```

### Run Registry Integration Test

```bash
# With registry URL
export MCA_REGISTRY_URL="http://localhost:8080"
export MCA_REGISTRY_TOKEN="your-token"

mvn compile exec:java -Dexec.mainClass="integration.RegistryIntegrationTest"

# Expected output (with registry):
# === Registry Integration Test ===
# ✓ Client initialized with registry integration
#   Registry URL: http://localhost:8080
# ✓ Recorded 3 predictions
# Background registry refresh is enabled
# Config will be refreshed automatically every 10 minutes
# === Registry Integration Test Complete ===

# Expected output (without registry):
# Note: Registry integration requires a running registry API
```

## Step 10: Performance Validation

Create `src/test/java/integration/PerformanceTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.RecordPredictionRequest;

/**
 * Performance test to validate NFR1 (<50ms overhead per prediction).
 */
public class PerformanceTest {

    public static void main(String[] args) {
        System.out.println("=== Performance Test ===\n");

        try (MCAClient client = new MCAClient.Builder()
                .serviceName("perf-test")
                .modelId("mdl-perf-001")
                .otelEndpoint("http://localhost:4318")
                .build()) {

            int iterations = 1000;
            System.out.println("Running " + iterations + " predictions...\n");

            long startTime = System.nanoTime();

            for (int i = 0; i < iterations; i++) {
                client.recordPrediction(RecordPredictionRequest.builder()
                        .latency(Math.random() * 0.1)
                        .predictionId("pred-perf-" + i)
                        .build());
            }

            long endTime = System.nanoTime();
            double durationSeconds = (endTime - startTime) / 1_000_000_000.0;
            double avgLatencyMs = (durationSeconds * 1000.0) / iterations;
            double throughput = iterations / durationSeconds;

            System.out.println("Performance Test Results:");
            System.out.println("  Total predictions: " + iterations);
            System.out.println("  Total time: " + String.format("%.2f", durationSeconds * 1000) + "ms");
            System.out.println("  Average latency: " + String.format("%.2f", avgLatencyMs) + "ms per prediction");
            System.out.println("  Throughput: " + String.format("%.0f", throughput) + " predictions/sec");
            System.out.println();

            // NFR1 requirement: SDK overhead <50ms (p95 <100ms)
            if (avgLatencyMs < 50) {
                System.out.println("✓ PASS: Average latency " + String.format("%.2f", avgLatencyMs) + "ms < 50ms target");
            } else {
                System.out.println("✗ FAIL: Average latency " + String.format("%.2f", avgLatencyMs) + "ms exceeds 50ms target");
            }

            Thread.sleep(5000);
            System.out.println("\n=== Performance Test Complete ===");

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            e.printStackTrace();
            System.exit(1);
        }
    }
}
```

### Run Performance Test

```bash
mvn compile exec:java -Dexec.mainClass="integration.PerformanceTest"

# Expected output:
# === Performance Test ===
# Running 1000 predictions...
#
# Performance Test Results:
#   Total predictions: 1000
#   Total time: 150.00ms
#   Average latency: 0.15ms per prediction
#   Throughput: 6667 predictions/sec
#
# ✓ PASS: Average latency 0.15ms < 50ms target
#
# === Performance Test Complete ===
```

## Step 11: Thread Safety Test

Create `src/test/java/integration/ThreadSafetyTest.java`:

```java
package integration;

import com.bhsf.mca.sdk.MCAClient;
import com.bhsf.mca.sdk.RecordPredictionRequest;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * Thread safety test for concurrent predictions.
 */
public class ThreadSafetyTest {

    public static void main(String[] args) {
        System.out.println("=== Thread Safety Test ===\n");

        try (MCAClient client = new MCAClient.Builder()
                .serviceName("thread-test")
                .modelId("mdl-thread-001")
                .otelEndpoint("http://localhost:4318")
                .build()) {

            int threadCount = 10;
            int predictionsPerThread = 100;
            AtomicInteger successCount = new AtomicInteger(0);
            AtomicInteger errorCount = new AtomicInteger(0);

            System.out.println("Starting " + threadCount + " threads...");
            System.out.println("Each thread will record " + predictionsPerThread + " predictions\n");

            ExecutorService executor = Executors.newFixedThreadPool(threadCount);
            long startTime = System.nanoTime();

            for (int i = 0; i < threadCount; i++) {
                final int threadId = i;
                executor.submit(() -> {
                    for (int j = 0; j < predictionsPerThread; j++) {
                        try {
                            client.recordPrediction(RecordPredictionRequest.builder()
                                    .latency(Math.random() * 0.2)
                                    .predictionId("pred-t" + threadId + "-" + j)
                                    .attribute("thread_id", threadId)
                                    .build());
                            successCount.incrementAndGet();
                        } catch (Exception e) {
                            errorCount.incrementAndGet();
                            System.err.println("Error in thread " + threadId + ": " + e.getMessage());
                        }
                    }
                });
            }

            executor.shutdown();
            executor.awaitTermination(60, TimeUnit.SECONDS);

            long endTime = System.nanoTime();
            double durationSeconds = (endTime - startTime) / 1_000_000_000.0;

            System.out.println("Thread Safety Test Results:");
            System.out.println("  Threads: " + threadCount);
            System.out.println("  Total predictions: " + (threadCount * predictionsPerThread));
            System.out.println("  Successful: " + successCount.get());
            System.out.println("  Errors: " + errorCount.get());
            System.out.println("  Duration: " + String.format("%.2f", durationSeconds) + "s");
            System.out.println("  Throughput: " + String.format("%.0f", successCount.get() / durationSeconds) + " predictions/sec");
            System.out.println();

            if (errorCount.get() == 0) {
                System.out.println("✓ PASS: No errors in concurrent execution");
            } else {
                System.out.println("✗ FAIL: " + errorCount.get() + " errors detected");
            }

            Thread.sleep(5000);
            System.out.println("\n=== Thread Safety Test Complete ===");

        } catch (Exception e) {
            System.err.println("✗ Error: " + e.getMessage());
            e.printStackTrace();
            System.exit(1);
        }
    }
}
```

### Run Thread Safety Test

```bash
mvn compile exec:java -Dexec.mainClass="integration.ThreadSafetyTest"

# Expected output:
# === Thread Safety Test ===
# Starting 10 threads...
# Each thread will record 100 predictions
#
# Thread Safety Test Results:
#   Threads: 10
#   Total predictions: 1000
#   Successful: 1000
#   Errors: 0
#   Duration: 0.25s
#   Throughput: 4000 predictions/sec
#
# ✓ PASS: No errors in concurrent execution
#
# === Thread Safety Test Complete ===
```

## Step 12: Verify Telemetry Data

### Check OTel Collector Logs

```bash
# Check for Java test service metrics
docker-compose logs otel-collector | grep -i "java-test-model"

# Check for specific metrics
docker-compose logs otel-collector | grep -i "model.predictions_total"
docker-compose logs otel-collector | grep -i "model.latency_seconds"

# Check for GenAI metrics
docker-compose logs otel-collector | grep -i "genai.tokens_total"
docker-compose logs otel-collector | grep -i "genai.cost_dollars"

# Check for Agentic metrics
docker-compose logs otel-collector | grep -i "agentic.goals_started_total"
docker-compose logs otel-collector | grep -i "agentic.tools_executed_total"

# Look for:
# - OTLP/HTTP requests received
# - Metrics exported
# - No errors
```

### Query Prometheus (if configured)

```bash
# Check prediction counter
curl -s http://localhost:9090/api/v1/query?query=model_predictions_total | jq .

# Check latency histogram
curl -s http://localhost:9090/api/v1/query?query=model_latency_seconds_bucket | jq .

# Check GenAI tokens
curl -s http://localhost:9090/api/v1/query?query=genai_tokens_total | jq .

# Check Agentic goals
curl -s http://localhost:9090/api/v1/query?query=agentic_goals_started_total | jq .
```

## Step 13: API Parity Validation with Python SDK

Run both SDKs side-by-side to verify equivalent behavior:

### Python SDK Test

```bash
cd /home/coder/emms/sdk/mca-prototype
python3 << 'EOF'
from mca_sdk import MCAClient

client = MCAClient(
    service_name="python-parity-test",
    model_id="mdl-python-001",
    model_version="1.0.0",
    team_name="test-team",
    model_type="internal",
    otel_endpoint="http://localhost:4318",
    strict_validation=False
)

client.record_prediction(latency=0.123, prediction_id="pred-py-001")
print("✓ Python SDK prediction recorded")

client.shutdown()
print("✓ Python SDK shutdown complete")
EOF
```

### Java SDK Test

```bash
cd /home/coder/emms/sdk/mca-prototype/mca-sdk-java

mvn compile exec:java -Dexec.mainClass="integration.BasicIntegrationTest"
```

### Compare Outputs

```bash
# Check that both SDKs produce similar OTLP payloads
docker-compose logs otel-collector | grep "python-parity-test" | tail -20
docker-compose logs otel-collector | grep "java-test-model" | tail -20

# Verify:
# ✓ Same metric names (model.predictions_total, model.latency_seconds)
# ✓ Same resource attributes (service.name, model.id, model.version, team.name)
# ✓ Similar payload structure
```

## Step 14: Run All Examples

```bash
cd /home/coder/emms/sdk/mca-prototype/mca-sdk-java

# Run Internal Model Example
mvn compile exec:java -Dexec.mainClass="examples.InternalModelExample"

# Run Advanced Instrumentation Example
mvn compile exec:java -Dexec.mainClass="examples.AdvancedInstrumentationExample"

# Run GCP Exporter Example (requires GCP setup)
export GCP_PROJECT_ID="your-project-id"
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/creds.json"
mvn compile exec:java -Dexec.mainClass="examples.GCPExporterExample"
```

## Troubleshooting

### Issue: "Cannot connect to collector"

```bash
# Check collector is running
docker-compose ps otel-collector

# Check collector endpoint
curl http://localhost:4318/

# Check port availability
netstat -tuln | grep 4318

# Restart collector
docker-compose restart otel-collector
```

### Issue: "No metrics appearing"

```bash
# Check export interval (default batch timeout is 30s for metrics, 5s for logs)
# Wait at least 35 seconds after recording metrics

# Check collector config
cat mca-prototype/config/otel-collector-config.yaml

# Verify OTLP receiver on port 4318
docker-compose logs otel-collector | grep "4318"

# Check for export errors
docker-compose logs otel-collector | grep -i "error"
```

### Issue: "Maven build fails"

```bash
# Clean Maven cache
mvn clean

# Rebuild with debug output
mvn clean compile -X

# Check Java version
java -version
# Should be Java 11 or higher

# Check Maven version
mvn -version
# Should be Maven 3.6+
```

### Issue: "GCP authentication fails"

```bash
# Check credentials file exists
ls -la $GOOGLE_APPLICATION_CREDENTIALS

# Test credentials
gcloud auth application-default print-access-token

# Check service account permissions
gcloud projects get-iam-policy $GCP_PROJECT_ID \
  --flatten="bindings[].members" \
  --format="table(bindings.role)" \
  --filter="bindings.members:serviceAccount:*"

# Required roles:
# - roles/cloudtrace.agent
# - roles/logging.logWriter
```

### Issue: "Tests timing out"

```bash
# Increase JUnit timeout in test annotations
@Test
@Timeout(value = 30, unit = TimeUnit.SECONDS)
public void testName() { ... }

# Or set global timeout in pom.xml surefire plugin
<configuration>
  <forkedProcessTimeoutInSeconds>60</forkedProcessTimeoutInSeconds>
</configuration>
```

### Issue: "OutOfMemoryError"

```bash
# Increase Maven heap size
export MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=512m"

# Or in mvn command
mvn test -DargLine="-Xmx2048m"
```

## Environment Variables Reference

| Variable | Default | Description |
|----------|---------|-------------|
| `MCA_SERVICE_NAME` | None | Service/model name (required) |
| `MCA_MODEL_ID` | None | Model identifier |
| `MCA_TEAM_NAME` | None | Team name |
| `MCA_MODEL_VERSION` | `1.0.0` | Model version |
| `MCA_OTEL_ENDPOINT` | `http://localhost:4318` | OTel Collector endpoint |
| `MCA_REGISTRY_URL` | None | Registry API URL (optional) |
| `MCA_REGISTRY_TOKEN` | None | Registry API token (optional) |
| `GCP_PROJECT_ID` | None | GCP project ID (for exporters) |
| `GOOGLE_APPLICATION_CREDENTIALS` | None | Path to GCP service account JSON |

## Success Criteria Checklist

- [ ] OTel Collector running and accessible on port 4318
- [ ] All unit tests passing (70+ tests)
- [ ] Test coverage >= 85% (verified with JaCoCo report)
- [ ] Basic integration test connects to collector
- [ ] Metrics visible in collector logs (model.predictions_total, model.latency_seconds)
- [ ] Advanced instrumentation working (SpanManager, custom metrics)
- [ ] GenAI integration tracking tokens and costs
- [ ] Agentic integration tracking goals and tools
- [ ] GCP exporters working (Cloud Trace, Cloud Logging)
- [ ] Registry integration with background refresh
- [ ] Performance meets NFR1 (<50ms overhead per prediction)
- [ ] Thread safety validated (1000 concurrent predictions, no errors)
- [ ] API parity with Python SDK (same metric names, attributes, behavior)
- [ ] Custom attributes included in telemetry payloads
- [ ] Shutdown flushes all pending data
- [ ] Try-with-resources (AutoCloseable) works correctly

## Next Steps

1. Run all integration tests in Workbench
2. Validate telemetry data in collector logs
3. Compare with Python SDK output for API parity
4. Performance test to verify NFR1 compliance
5. Test GCP exporters with real GCP project
6. Update story status to "review" in sprint-status.yaml
7. Create PR with any changes

## Additional Resources

- **README**: `mca-prototype/mca-sdk-java/README.md`
- **Maven POM**: `mca-prototype/mca-sdk-java/pom.xml`
- **Examples**: `mca-prototype/mca-sdk-java/examples/`
- **Tests**: `mca-prototype/mca-sdk-java/src/test/java/`
- **OTel Collector Config**: `mca-prototype/config/otel-collector-config.yaml`
- **Python SDK Workbench**: `mca-prototype/mca-sdk-nodejs/TESTING_WORKBENCH.md`
