# OTel Collector Self-Monitoring Metrics Guide

This guide documents the self-monitoring metrics exposed by the OpenTelemetry Collector on port **:8889** in Prometheus format. This is the handoff point for downstream monitoring implementation (Google-Managed Prometheus, Cloud Monitoring dashboards, Alertmanager).

## Architecture Overview

```
Collector Internal Telemetry (:8888 internal)
    ↓
Prometheus Receiver (scrapes localhost:8888)
    ↓
metrics/self pipeline (batch processor)
    ↓
Prometheus Exporter
    ↓
Port :8889 (exposed for external scraping) ← HANDOFF POINT
    ↓
[Google-Managed Prometheus (scrapes metrics) - to be implemented]
    ↓
[Cloud Monitoring Dashboards - to be implemented]
```

## Accessing Metrics

**Endpoint:** `http://otel-collector:8889/metrics`
**Format:** Prometheus exposition format (text/plain)
**Scrape Interval:** Recommended 30s
**Namespace:** `otelcol_*`

**Security Note:** The endpoint is exposed on `0.0.0.0:8889` to allow downstream Google-Managed Prometheus scraping. In production:
- Use Kubernetes NetworkPolicy to restrict access to monitoring namespace only
- Consider deploying a reverse proxy with TLS and authentication
- Ensure metrics endpoint is not exposed to the public internet
- Use VPC-internal networking for scraping in cloud environments

### Example Scrape

```bash
curl http://localhost:8889/metrics
```

## Available Metrics

### Receiver Metrics

Monitor telemetry ingestion at the receiver layer.

| Metric | Type | Description | Use Case |
|--------|------|-------------|----------|
| `otelcol_receiver_accepted_spans` | Counter | Total spans successfully accepted | Track ingestion rate |
| `otelcol_receiver_refused_spans` | Counter | Total spans refused (backpressure) | Detect overload conditions |
| `otelcol_receiver_accepted_metric_points` | Counter | Total metric points successfully accepted | Track metric ingestion |
| `otelcol_receiver_refused_metric_points` | Counter | Total metric points refused | Detect metric overload |

**Labels:**
- `receiver`: Receiver name (e.g., `otlp`)
- `transport`: Protocol (e.g., `http`, `grpc`)
- `service`: `otel-collector`
- `environment`: `prototype` (or production value)
- `job`: Prometheus scrape job name (e.g., `otel-collector`)
- `instance`: Collector instance identifier (e.g., `otel-collector-0:8889`, `otel-collector-1:8889`)

**Multi-Instance Deployments:** In HA deployments with multiple collector replicas, the `instance` label is critical for distinguishing metrics from different collectors. Use PromQL aggregations like `sum by (job)` for cluster-wide metrics or `by (instance)` for per-instance analysis.

### Processor Metrics

Monitor batch processing behavior.

| Metric | Type | Description | Use Case |
|--------|------|-------------|----------|
| `otelcol_processor_batch_batch_send_size` | Histogram | Distribution of batch sizes sent | Tune batch size config |
| `otelcol_processor_batch_batch_size_trigger_send` | Counter | Batches sent due to size trigger | Verify batch behavior |
| `otelcol_processor_batch_timeout_trigger_send` | Counter | Batches sent due to timeout trigger | Balance latency vs throughput |

**Labels:**
- `processor`: Processor name (e.g., `batch`)

### Exporter Metrics

Monitor telemetry export success/failure.

| Metric | Type | Description | Use Case |
|--------|------|-------------|----------|
| `otelcol_exporter_sent_spans` | Counter | Total spans successfully exported | Track export success rate |
| `otelcol_exporter_send_failed_spans` | Counter | Total spans failed to export | Detect export failures |
| `otelcol_exporter_sent_metric_points` | Counter | Total metric points exported | Track metric export |
| `otelcol_exporter_send_failed_metric_points` | Counter | Total metric points failed | Detect metric export issues |

**Labels:**
- `exporter`: Exporter name (e.g., `debug`, `file/metrics`, `prometheus`)

### Queue Metrics

Monitor internal queue depth and capacity for exporters with sending queues configured.

| Metric | Type | Description | Use Case |
|--------|------|-------------|----------|
| `otelcol_exporter_queue_size` | Gauge | Current items in queue | Detect queue buildup |
| `otelcol_exporter_queue_capacity` | Gauge | Maximum queue capacity | Calculate utilization |

**Note:** Queue metrics apply only to exporters with explicit queueing configured (e.g., OTLP exporters with `sending_queue` enabled). The Prometheus exporter and file exporters used in this configuration do not use sending queues, so these metrics may not be present for all exporters. They are included here for completeness when monitoring application telemetry pipelines with queued exporters.

**Calculated Metric:**
- Queue Utilization: `(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) * 100`

### Resource Metrics

Monitor collector process health.

| Metric | Type | Description | Use Case |
|--------|------|-------------|----------|
| `process_cpu_seconds_total` | Counter | Total CPU time consumed | Calculate CPU % usage |
| `process_resident_memory_bytes` | Gauge | Memory usage (RSS) | Monitor memory consumption |
| `go_goroutines` | Gauge | Number of goroutines | Detect goroutine leaks |
| `go_memstats_alloc_bytes` | Gauge | Allocated heap bytes | Monitor memory allocation |

## Recommended Dashboard Layout

### Dashboard: Collector Health Overview

**Row 1: Request Rate**
- **Panel 1:** OTLP Requests/sec
  - Query: `rate(otelcol_receiver_accepted_spans{receiver="otlp"}[5m])`
  - Type: Graph (time series)
  - Y-axis: spans/sec

- **Panel 2:** Export Success Rate
  - Query: `rate(otelcol_exporter_sent_spans[5m])`
  - Type: Graph (time series)
  - Y-axis: spans/sec

- **Panel 3:** Error Rate
  - Query: `rate(otelcol_exporter_send_failed_spans[5m])`
  - Type: Graph (time series)
  - Y-axis: spans/sec
  - Threshold: Red if > 5% of sent rate

**Row 2: Queue Health**
- **Panel 1:** Queue Depth
  - Query: `otelcol_exporter_queue_size`
  - Type: Graph (time series)
  - Y-axis: items

- **Panel 2:** Queue Capacity Utilization
  - Query: `(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) * 100`
  - Type: Gauge
  - Unit: %
  - Thresholds: Yellow > 80%, Red > 95%

- **Panel 3:** Batch Trigger Distribution
  - Query 1: `rate(otelcol_processor_batch_batch_size_trigger_send[5m])`
  - Query 2: `rate(otelcol_processor_batch_timeout_trigger_send[5m])`
  - Type: Stacked area graph
  - Legend: Size trigger vs Timeout trigger

**Row 3: Resource Usage**
- **Panel 1:** CPU Usage
  - Query: `rate(process_cpu_seconds_total[5m]) * 100`
  - Type: Graph (time series)
  - Unit: %
  - Threshold: Yellow > 70%, Red > 90%

- **Panel 2:** Memory Usage
  - Query: `process_resident_memory_bytes / 1024 / 1024`
  - Type: Graph (time series)
  - Unit: MiB
  - Threshold: Red > 400 MiB (configured limit: 512 MiB)

- **Panel 3:** Goroutines
  - Query: `go_goroutines`
  - Type: Graph (time series)
  - Threshold: Yellow > 1000 (may indicate leak)

## Recommended Alert Thresholds

### Critical Alerts (Page On-Call)

**1. High Error Rate**
- Condition: `(rate(otelcol_exporter_send_failed_spans[5m]) / rate(otelcol_exporter_sent_spans[5m])) > 0.10`
- Severity: Critical
- Description: Export error rate > 10%
- Action: Check exporter connectivity, backend health

**2. Queue Near Full**
- Condition: `(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) > 0.95`
- Severity: Critical
- Description: Queue utilization > 95%
- Action: Scale up collector, investigate slow exporters

**3. Memory Exhaustion**
- Condition: `process_resident_memory_bytes > (512 * 1024 * 1024 * 0.90)`
- Severity: Critical
- Description: Memory usage > 90% of configured limit (512 MiB)
- Action: Check for memory leaks, increase limit

### Warning Alerts (Monitor)

**4. Elevated Error Rate**
- Condition: `(rate(otelcol_exporter_send_failed_spans[5m]) / rate(otelcol_exporter_sent_spans[5m])) > 0.05`
- Severity: Warning
- Description: Export error rate > 5%
- Action: Investigate intermittent failures

**5. Queue Buildup**
- Condition: `(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) > 0.80`
- Severity: Warning
- Description: Queue utilization > 80%
- Action: Monitor for continued growth

**6. High CPU Usage**
- Condition: `rate(process_cpu_seconds_total[5m]) > 0.80`
- Severity: Warning
- Description: CPU usage > 80%
- Action: Consider scaling horizontally

**7. Backpressure Detected**
- Condition: `rate(otelcol_receiver_refused_spans[5m]) > 0`
- Severity: Warning
- Description: Receiver refusing spans due to backpressure
- Action: Scale collector or reduce ingestion rate

## Example Prometheus Queries

### Ingestion Rate (spans/sec)
```promql
rate(otelcol_receiver_accepted_spans{receiver="otlp"}[5m])
```

### Export Success Percentage
```promql
(
  rate(otelcol_exporter_sent_spans[5m]) /
  (rate(otelcol_exporter_sent_spans[5m]) + rate(otelcol_exporter_send_failed_spans[5m]))
) * 100
```

### Average Batch Size
```promql
histogram_quantile(0.95, rate(otelcol_processor_batch_batch_send_size_bucket[5m]))
```

### Queue Utilization Percentage
```promql
(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) * 100
```

### CPU Usage Percentage
```promql
rate(process_cpu_seconds_total[5m]) * 100
```

### Memory Usage (MiB)
```promql
process_resident_memory_bytes / 1024 / 1024
```

## Integration with Google-Managed Prometheus

### GMP PodMonitoring Configuration

For Google-Managed Prometheus, use PodMonitoring (recommended) or a compatible scrape configuration:

```yaml
scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 30s
    static_configs:
      - targets: ['otel-collector:8889']
        labels:
          cluster: 'production'
          datacenter: 'us-central1'
```

### Alternative: Kubernetes Service Monitor (for non-GMP environments)

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: otel-collector-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otel-collector
  endpoints:
    - port: prometheus-metrics
      interval: 30s
      path: /metrics
```

## Troubleshooting

### Metrics Not Appearing

**Symptom:** No metrics visible at :8889/metrics

**Checks:**
1. Verify collector is running: `docker ps | grep otel-collector`
2. Check port exposure: `netstat -an | grep 8889`
3. Test endpoint: `curl http://localhost:8889/metrics`
4. Review collector logs: `docker logs mca-otel-collector`
5. Verify config: Check `service.telemetry.metrics.level: detailed`

### High Refused Spans

**Symptom:** `otelcol_receiver_refused_spans` incrementing

**Root Causes:**
- Exporter bottleneck (slow backend)
- Queue full (increase queue size)
- Memory pressure (increase memory limit)
- Insufficient CPU (scale horizontally)

**Actions:**
1. Check exporter metrics for failures
2. Review queue utilization
3. Monitor resource usage
4. Scale collector instances

### Export Failures

**Symptom:** `otelcol_exporter_send_failed_spans` incrementing

**Root Causes:**
- Backend unavailable (network, service down)
- Authentication failure (invalid credentials)
- Rate limiting (backend throttling)
- Timeout (slow backend response)

**Actions:**
1. Check backend health
2. Verify network connectivity
3. Review exporter configuration
4. Check backend logs for errors

## Multi-Instance Label Strategy

When running multiple collector instances (e.g., in Kubernetes with replicas > 1), Prometheus automatically adds labels to distinguish metrics:

**Critical Labels for Multi-Instance:**
- `job`: The scrape job name from Prometheus config (e.g., `otel-collector`)
- `instance`: Unique collector instance (e.g., `10.0.1.5:8889`, `otel-collector-0:8889`)

**Example PromQL Queries:**

```promql
# Cluster-wide ingestion rate (sum across all instances)
sum by (job) (rate(otelcol_receiver_accepted_spans[5m]))

# Per-instance ingestion rate (identify hot spots)
rate(otelcol_receiver_accepted_spans[5m])

# Instances with high error rates
topk(5, rate(otelcol_exporter_send_failed_spans[5m]) by (instance))

# Total cluster memory usage
sum by (job) (process_resident_memory_bytes)

# Instance with highest memory usage
topk(1, process_resident_memory_bytes)
```

**Dashboard Considerations:**
- Use `instance` variable dropdown to filter by specific collector
- Show cluster-wide aggregates alongside per-instance breakdowns
- Alert on per-instance thresholds (not just cluster aggregates)

## Production Considerations

### Scaling Guidelines

**Vertical Scaling (Single Instance):**
- CPU: 2-4 cores recommended
- Memory: 512 MiB - 2 GiB (adjust based on queue size)
- Disk: Minimal (no persistent storage)

**Horizontal Scaling (Multiple Instances):**
- Load balance OTLP traffic across instances
- Each instance self-monitors independently
- Aggregate metrics in Prometheus
- Use consistent labels (environment, cluster) for aggregation

### High Availability

- Deploy multiple collector instances
- Use health checks for load balancer routing
- Monitor each instance separately
- Alert if any instance fails health check

### Security

- Restrict :8889 access to monitoring network only
- Use NetworkPolicy in Kubernetes
- Consider enabling TLS for metrics endpoint
- Add authentication if exposing externally

## Next Steps (Downstream Implementation)

This guide provides the specification for downstream monitoring. The following steps are **outside the scope of this story** and should be implemented by the storage/visualization team:

1. Configure Google-Managed Prometheus to scrape `http://otel-collector:8889/metrics`
2. Create Cloud Monitoring dashboards using recommended layout
3. Configure Alertmanager with recommended alert thresholds for GMP
4. Set up PagerDuty/email integration for critical alerts
5. Configure BigQuery for long-term metric storage
6. Create runbooks for common alert scenarios

## References

- [OpenTelemetry Collector Monitoring](https://opentelemetry.io/docs/collector/monitoring/)
- [Prometheus Exposition Format](https://prometheus.io/docs/instrumenting/exposition_formats/)
- [Cloud Monitoring Dashboards](https://cloud.google.com/monitoring/dashboards)
- [Google-Managed Prometheus](https://cloud.google.com/stackdriver/docs/managed-prometheus)
