# Monitoring

The Application SDK exposes Prometheus metrics covering workflow / activity
execution, HTTP server traffic, custom application metrics, and Temporal
SDK internals — all through a single endpoint per pod. Scrape configuration
differs by deployment topology (combined-mode pods are scraped directly;
worker-only pods push to a Pushgateway).

## Architecture

The SDK funnels every metric source into the in-process
`prometheus_client.REGISTRY` and exposes that registry to operators by
one of two transports, depending on pod role:

| Pod role | Transport | Endpoint |
|---|---|---|
| **Combined** | Direct scrape | `http://<pod>:8000/metrics` (FastAPI, including proxied Temporal Rust-core metrics when enabled) |
| **Handler** | Direct scrape | `http://<pod>:8000/metrics` (FastAPI in-process metrics; Temporal Rust-core proxy disabled by default) |
| **Worker** (split deployment) | Push | Pushgateway at `ATLAN_PROMETHEUS_PUSHGATEWAY_URL` |

The Temporal SDK's Rust-core Prometheus endpoint (`127.0.0.1:9464`) is
bound **loopback only** and is not externally reachable. The FastAPI
`/metrics` handler proxies it in-process for combined pods; handler-only
pods skip that proxy by default because no local worker/runtime is started
with the handler process. The worker's `TemporalCoreCollector` reads it
locally and includes the families in each Pushgateway push. So scrape
configurations always target the FastAPI port (or the Pushgateway), never
`:9464` directly.

### What's in the metric body

Every emitted series carries one inlined resource-attribute label —
`app_name` — so the most common operator query (filter by connector)
works without a `target_info` join. Per-release attributes
(`app_version`, `app_release_id`, `app_sdk_version`,
`app_release_channel`) and the rest of the OTel `Resource` travel via
the `target_info` gauge (one row per pod) and are recovered at query
time with `* on(instance) group_left(...) target_info`. See
[Metrics Standards](../standards/metrics.md) for the cardinality rules
that bound the rest of the label set.

The FastAPI `/metrics` endpoint (or a worker push) merges:

- **Custom metrics** from `record_metric()` calls and direct OTel meter
  use (`application_sdk.observability.metrics`)
- **HTTP server instrumentation** (FastAPIInstrumentor) using stable
  OTel HTTP semantic conventions: `http.request.method`, `http.route`,
  `server.address`, `network.protocol.version`,
  `http.response.status_code`. The metric is named
  `http_server_request_duration_seconds` (seconds, not milliseconds).
  Stable conventions are enabled via `OTEL_SEMCONV_STABILITY_OPT_IN=http`
  — set in the chart by default for v3+ apps.
- **Temporal SDK** families from `MetricsInterceptor` (counters and
  histograms emitted by activity/workflow execution) plus, when enabled
  and reachable, the Rust-core families proxied from `127.0.0.1:9464`
  (`temporal_request_*`, `temporal_workflow_task_*`,
  `temporal_sticky_cache_*`, `temporal_num_pollers`, etc.)
- **`prometheus_client` defaults** (`process_*`, `python_gc_*`,
  `python_info`)

### `ATLAN_ENABLE_TEMPORAL_CORE_METRICS`

This env var **does not gate the FastAPI `/metrics` route** — that
endpoint is always registered. The flag controls whether worker/combined
mode creates a Temporal Runtime that binds `127.0.0.1:9464`:

| Value | Effect |
|---|---|
| `true` (default) | Rust core binds 9464 in worker/combined mode; combined FastAPI proxy + worker `TemporalCoreCollector` can read it |
| `false` | Rust core builds an explicit `Runtime` with log forwarding but no Prometheus listener; FastAPI `/metrics` still serves SDK + HTTP + python defaults but lacks the `temporal_*` Rust-core families |

`run_dev_combined()` proactively sets it to `false` in local dev so a
hot-reload-restarted process doesn't fail to bind 9464 (which the
previous process is still holding in `TIME_WAIT`).

### Pushgateway for worker pods

Worker-only pods (split deployment) don't run a FastAPI server, so
Prometheus has no scrape target. Instead, they push the local registry
to a Pushgateway every `ATLAN_PROMETHEUS_PUSHGATEWAY_INTERVAL_SECONDS`
(default 30 s) plus a final push on shutdown. Configuration:

```bash
ATLAN_PROMETHEUS_PUSHGATEWAY_URL=http://prometheus-pushgateway.monitoring.svc.cluster.local:9091
ATLAN_PROMETHEUS_PUSHGATEWAY_INTERVAL_SECONDS=30
ATLAN_PROMETHEUS_PUSHGATEWAY_DELETE_ON_SHUTDOWN=true            # default
ATLAN_PROMETHEUS_PUSHGATEWAY_SWEEP_STALE_ON_START=true          # default
ATLAN_PROMETHEUS_PUSHGATEWAY_SWEEP_STALENESS_SECONDS=300        # default
ATLAN_PROMETHEUS_PUSHGATEWAY_HTTP_TIMEOUT_SECONDS=10            # default
ATLAN_PROMETHEUS_PUSHGATEWAY_SHUTDOWN_DELETE_DELAY_SECONDS=35   # default
```

Hardening behaviors that ship by default: graceful exits DELETE the
group from the gateway (no leak); each new worker sweeps stale
predecessor groups left by OOM/eviction; every gateway HTTP call is
bounded to 10 s; the final push is held for 35 s before DELETE so
Prometheus has a full scrape window. See [ADR-0012](../adr/0012-observability-consolidation.md)
for the design rationale.

### Available Temporal SDK metrics

A non-exhaustive subset of what flows through the Rust-core path:

| Metric | Description |
|--------|-------------|
| `temporal_activity_execution_latency` | Activity execution duration (histogram) |
| `temporal_activity_schedule_to_start_latency` | Time from activity schedule to start (histogram) |
| `temporal_workflow_completed_total` | Total completed workflows (counter) |
| `temporal_workflow_endtoend_latency` | End-to-end workflow duration (histogram) |
| `temporal_request_latency` | gRPC request latency to Temporal server (histogram) |
| `temporal_long_request_total` | gRPC long-poll requests (counter) |
| `temporal_worker_task_slots_available` | Available worker task slots (gauge) |
| `temporal_worker_task_slots_used` | In-use worker task slots (gauge) |
| `temporal_sticky_cache_hit_total` | Sticky cache hits (counter) |
| `temporal_sticky_cache_size` | Current sticky cache size (gauge) |

The full list is defined by the [Temporal Python SDK](https://docs.temporal.io/references/sdk-metrics).

### Prometheus scrape configuration (combined / handler pods)

In Kubernetes the `atlan-app` chart provisions a ServiceMonitor when
`metrics.enabled: true` is set in the app's values. The scrape uses the
Service's named port `http`:

```yaml
spec:
  endpoints:
    - port: http        # main Service declares 8000 as `name: http`
      path: /metrics
      interval: 60s     # configurable via metrics.interval
```

For raw Prometheus configurations, target the handler port directly:

```yaml
scrape_configs:
  - job_name: <app>-handler
    static_configs:
      - targets:
          - <handler-host>:8000
    metrics_path: /metrics
```

### Recommended Alerts

| Alert | Condition | Severity |
|-------|-----------|----------|
| High activity failure rate | `rate(temporal_activity_execution_failed_total[5m]) > 0.05` | Warning |
| Worker slots exhausted | `temporal_worker_task_slots_available == 0` | Critical |
| Elevated workflow latency | `histogram_quantile(0.99, temporal_workflow_endtoend_latency_bucket) > 300` | Warning |
| Temporal server errors | `rate(temporal_long_request_failure_total[5m]) > 0` | Warning |

---

## Distributed Tracing (OTLP)

Enable OpenTelemetry trace export to send spans to your cluster's OTLP collector:

```bash
ENABLE_OTLP_LOGS=true                              # export log records via OTLP
OTEL_EXPORTER_OTLP_ENDPOINT=http://$(K8S_NODE_IP):4317  # node-local collector
ATLAN_ENABLE_OTLP_TRACES=true                      # export trace spans
# Metrics use the Prometheus path (FastAPI /metrics + Pushgateway).
# The previous OTLP metric exporter was removed in PR #1573.
```

In Kubernetes, set `OTEL_EXPORTER_OTLP_ENDPOINT` using the Downward API:

```yaml
env:
  - name: K8S_NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(K8S_NODE_IP):4317"
```

---

## Structured Logging and `self.logger`

`self.logger` is available in both `run()` and `@task` methods. It is automatically bound with:

| Field | Value |
|-------|-------|
| `app_name` | The running entry-point's app name (see note below) |
| `workflow_run_id` | Temporal workflow run ID (canonical name; `run_id` is a backwards-compat alias) |
| `correlation_id` | Platform-level correlation identifier |
| `source` | Which layer emitted the line — `sdk`, `dependency`, or the app's own name (see [Log provenance](#log-provenance-the-source-field)) |

These fields appear on every log entry without any manual binding — use `self.logger` directly.

> **`app_name` is per-entry-point (CNCT-93).** It is resolved from the workflow's
> own input args, which the contract toolkit stamps with each DAG node's own
> `app_name`. For a single-entry-point connector this is just its kebab-case name
> (e.g. `mysql`). For a **multi-entry-point bundle** it is the *entry-point's*
> name — `powerbi-crawler`, `powerbi-miner` — **not** the connector-level bundle
> name, so each entry-point's logs are queryable on their own. This is the value
> the run UI queries a node's logs by. When the workflow input carries no
> `app_name` (an older, not-yet-regenerated app), it falls back to the process-wide
> `ATLAN_APPLICATION_NAME` env — i.e. prior behaviour. The per-entry-point value
> is a **log-only** field: metrics (both the `temporal_*` lifecycle families and
> `record_metric()` custom metrics) and the trace-level `app.name` (OTel
> Resource) deliberately stay **connector-level**, so for a bundle a metric's or
> trace's `app_name` may differ from the `app_name` on its correlated log lines
> — by design. See [Metrics Standards](../standards/metrics.md).

```python
class MyConnector(App):
    @task
    async def fetch_data(self, input: FetchInput) -> FetchOutput:
        self.logger.info("fetching page=%d", page_num)
        # Emits: {"level":"INFO","msg":"fetching page=3","app_name":"my-connector","workflow_run_id":"...","correlation_id":"..."}
```

Use **%-style** message bodies (`"fetching page=%d", page_num`) rather than keyword arguments. See [Logging Standards](../standards/logging.md) and [ADR-0011](../adr/0011-logging-level-guidelines.md).

### Structured attributes and the OTLP allowlist

Structured kwargs on a log call (e.g. `logger.info("event", outcome="clean", assets_total=42)`) are
not forwarded to OTLP wholesale. Only keys on an SDK-side **allowlist**
(`_KNOWN_EXTRA_KEYS` in `application_sdk/observability/logger_adaptor.py`) become `LogAttributes` on
the exported record; unlisted keys are dropped. This keeps the exported attribute set intentional —
adding a new queryable field is a deliberate one-line change next to the emitter. Certain dotted
prefixes (`atlan.`, `temporal.`, `failure.`, `exception.`, `otel.`, `tenant.`, `workflow_run.`) pass
through without being individually listed.

### Log provenance: the `source` field

Every record carries a low-cardinality `source` attribute answering *which layer emitted this line*.
It is stamped centrally — by both `AtlanLoggerAdapter` and the stdlib-logging bridge — so app authors
cannot get it wrong, and it is allowlisted for OTLP, so it is queryable alongside `app_name`:

| `source` | Records it covers |
|----------|-------------------|
| `sdk` | Anything logged from inside `application_sdk` — framework internals and the interceptor lifecycle lines |
| `dependency` | Known third-party loggers (`httpx`, `daft_io`, `temporalio`, …) and the forwarded `daprd` sidecar lines |
| *the app label* | Everything else — by default the app's own `APPLICATION_NAME` (e.g. `mysql`), so a reader sees *which* app spoke |

`source` answers a different question from `app_name`: `app_name` says *which app / entry-point* the
record came from (per-entry-point for a bundle — see the note above), `source` says *which layer
within it*. That makes `source = sdk` the filter for "is this the framework or the connector's own
code" during triage.

The app label is overridable with the `ATLAN_LOG_SOURCE` environment variable — the Automation
Engine sets `ATLAN_LOG_SOURCE=ae` so its orchestration lines are attributable to the engine rather
than to whichever app it happens to be running. Apps should leave it unset. See
[Common Utilities → Logging Configuration](common.md#configuration).

An explicit `source=` kwarg on an individual log call still wins over the derived value.

### Log↔trace correlation (`trace_id` / `span_id`)

OTLP log records carry real trace context: `trace_id` and `span_id` are populated from the span
active in the emitting thread, so a log line joins to its span in the trace UI. Previously both were
hard-coded to zero, which made log↔trace correlation impossible.

When no span is active — which includes the case where trace export is off
(`ATLAN_ENABLE_OTLP_TRACES` unset, the current production default) — both fall back to zero and the
record is exported untraced, exactly as before. Enabling traces is therefore what turns this
correlation on.

`correlation_id` remains the business-level join key and is **not** replaced by trace context: it
spans an entire platform-level operation across process and workflow boundaries, whereas a
`trace_id` covers one trace. See [Correlation IDs](#correlation-ids).

### Lifecycle log lines

`LogInterceptor` emits four lifecycle lines per run, with OTel semantic-convention attributes
attached. The **event token is the exact message prefix** — that is the stable, greppable contract
for dashboards and alerts:

| Token | Level | Message body |
|-------|-------|--------------|
| `workflow.started` | INFO | `workflow.started <WorkflowType>` |
| `workflow.ended` | INFO / WARNING / ERROR | `workflow.ended <WorkflowType> OK (<ms>ms)`, `… BLOCKED (preflight gate)`, or `… FAILED (<code>): <message> — at <file>:<line> in <fn>` |
| `activity.started` | INFO | `activity.started <ActivityType>` |
| `activity.ended` | INFO / WARNING / ERROR | same three shapes as `workflow.ended` |

The body after the token is a **human-readable summary, not a contract** — it names the subject and,
on failure, folds in the first line of the exception message (truncated) and the root-cause frame, so
a line stays diagnosable in renderers that drop structured attributes. Match on the token prefix and
the structured attributes, never on the body text.

!!! note "Token stability"

    The task-level tokens (`activity.started` / `activity.ended`) and workflow-level tokens
    (`workflow.started` / `workflow.ended`) are a stable contract — dashboard queries, saved log
    searches, and alert rules may match on these literal prefixes. The bodies after the token are
    human-readable summaries and may change; match on the token prefix and the structured
    attributes, not the body text.

### Asset-validation outcome event

`App.upload()`'s warn-only asset validation (see [Apps → Asset-Validation Outcome](apps.md#asset-validation-outcome))
emits a structured event named `"Transformed-asset validation outcome"` on **every validated
upload**, from inside the `upload` activity — so the Temporal context (`workflow_run_id`, `app_name`)
is auto-stamped and each row joins to the workflow outcome by run id in ClickHouse. The following
attributes are allowlisted and reach OTLP:

| Attribute | Meaning |
|-----------|---------|
| `outcome` | `"clean"` or `"flagged"` |
| `assets_total` | records seen in the batch |
| `assets_passed` | records that passed per-asset `.validate()` |
| `assets_invalid` | per-asset validation failures |
| `assets_orphaned` | referential-integrity (orphan) failures |
| `assets_undeserializable` | records that could not be decoded |
| `asset_validation_matrix` | compact JSON array of per-failure detail (bounded rows per axis), `JSONExtract`-able |

Emitting `outcome="clean"` too gives a denominator, so a dashboard can rank connectors by
flag-rate rather than only seeing failures. Uploads with nothing to validate (validation disabled, or
a non-`transformed/` path) emit no event.

### Replay suppression

`self.logger` suppresses log output during Temporal workflow replay by default, matching the behaviour of Temporal's native `workflow.logger`. This prevents duplicate bare lines (missing `workflow_id`/`run_id`) that would otherwise appear in Grafana when a worker replays history after a sticky-cache eviction.

If you need replay logs — for example when using `temporalio.worker.Replayer` locally to inspect workflow history — re-enable them in two ways:

- **Per-instance** (in-code): `self.logger.log_during_replay = True`
- **Globally** (env flag): `ENABLE_WORKFLOW_REPLAY_LOGS=true`

---

## Memory pressure and OOM diagnostics

OOM kills produce no application-level log — the kernel sends SIGKILL and the
process vanishes silently. The SDK surfaces four log signals so operators have
a clear evidence trail and a short time-to-diagnosis.

### Log surfaces (grep patterns)

| # | When | Level | Where | Grep |
|---|------|-------|-------|------|
| 1 | Worker/combined/handler startup | `INFO` | pod log (first lines) | `"Process memory at start"` |
| 2 | Every 20 s during an active task, once RSS ≥ 80 % of limit | `WARNING` | pod log | `"Memory pressure on task"` |
| 3 | Immediately after pod restart, if `exitCode == 137` | `CRITICAL` | pod log (first lines of new pod) | `"exit code 137 = SIGKILL"` |
| 4 | When Temporal re-dispatches an activity after worker loss | `WARNING` | workflow log | `"re-dispatched after worker eviction"` |

Signal 1 establishes a baseline (RSS at startup, limit, %) so you can see
where memory stood when the pod was last healthy. Signal 2 fires on the rising
edge and re-arms after the ratio drops below 75 %, giving pre-kill leading
indicators in the killed pod's log. Signal 3 fires in the **replacement** pod's
entrypoint immediately on restart — before any Temporal heartbeat timeout — so
the first thing you see in `kubectl logs` is the exit code, along with the
diagnostic commands to run. Signal 4 names OOM kill (pod exit 137) explicitly
alongside KEDA scale-down, spot preemption, and rolling deploys.

### Required Kubernetes configuration

Signal 2 and signal 1's percentage require `K8S_POD_MEMORY_LIMIT` to be
injected via the Downward API:

```yaml
env:
  - name: K8S_POD_MEMORY_LIMIT
    valueFrom:
      resourceFieldRef:
        resource: limits.memory
        divisor: "1"          # raw bytes; parse_pod_memory_limit() also accepts Ki/Mi/Gi suffixes
```

When this env var is absent the memory-pressure warning is silently disabled
(no false positives in local dev / non-Kubernetes environments).

### Diagnostic runbook (OOM kill)

1. **Check exit code in the new pod** (`kubectl logs <pod>` — look for the exit-137 CRITICAL line at the top, or the process memory at start line showing a high baseline).
2. **Confirm OOM kill** — `kubectl describe pod <pod>` → `lastState.terminated.reason: OOMKilled`.
3. **Check cluster events** — `kubectl get events -n <namespace> --field-selector=reason=OOMKilling`.
4. **Review memory trajectory** — search the killed pod's log for `Memory pressure on task` lines to see how fast RSS climbed before the kill.
5. **Check eviction retries** — search the workflow log for `re-dispatched after worker eviction` to understand how many attempts Temporal made.

---

## Correlation IDs

The `correlation_id` propagates automatically across:

- The handler (set by the incoming request from Heracles)
- All `@task` log entries (bound by the framework)
- Cross-app correlation (propagated via `correlation_id` when one App's output triggers another via Automation Engine DAG)

In Grafana/Loki, filter by `correlation_id` to see the full trace of a multi-phase connector run across all apps.

Access the current correlation ID programmatically:

```python
from application_sdk.observability import get_correlation_context

ctx = get_correlation_context()
cid = ctx.correlation_id if ctx else None  # str (empty when unset) or None when ctx is None
```

---

## Observability Store Sink

By default, logs, metrics, and traces are also written to gzip-compressed NDJSON files (`.json.gz`) in the object store under `artifacts/apps/{app_name}/{deployment_name}/observability/`. This enables historical querying even when the live pipelines (OTLP for logs/traces, Prometheus scrape / Pushgateway push for metrics) are unavailable.

Control with:

```bash
ATLAN_ENABLE_OBSERVABILITY_STORE_SINK=true   # default: true
```

Retention and batching:

```bash
ATLAN_LOG_RETENTION_DAYS=30
ATLAN_LOG_BATCH_SIZE=100
ATLAN_LOG_FLUSH_INTERVAL_SECONDS=10
```

See [Configuration](../configuration.md#logging) for all observability variables.

---

## Forwarding daprd Sidecar Logs

The `daprd` sidecar's own logs go straight to the container's stdout/stderr and don't enter the SDK observability pipeline on their own.

- **On atlan-infra** (`ENABLE_ATLAN_UPLOAD=false`) this is fine: a node-level filelog collector scrapes every container's stdout — daprd included — and ships it to the central log backend.
- **In SDR mode** (`ENABLE_ATLAN_UPLOAD=true`, i.e. customer infra) there is **no** such node-level collector, so daprd's logs would be invisible to Atlan — only present in the customer's own pod logs.

So in SDR mode the SDK **automatically** forwards daprd's logs through its own pipeline (the same path as the app's logs, so they land in the lakehouse `app_logs` table with `app_name` / `is_sdr` populated), while still echoing them to the container's own logs. No configuration is required — it is gated on `ENABLE_ATLAN_UPLOAD`. Under the hood the container entrypoint runs daprd under `application_sdk.observability.dapr_log_forwarder`, which streams each daprd log line into a `dapr.runtime` logger and forwards `SIGTERM` so graceful shutdown is unaffected.

> **Note (`kubectl logs` format in SDR mode):** In SDR mode daprd's stdout/stderr is piped into the forwarder, so `kubectl logs` shows the SDK's re-emitted format (with level, `app_name`, timestamp) rather than daprd's raw `--log-as-json` lines. The content is the same; only the format differs.

daprd is chatty, so control the volume at the source with its log level:

```bash
DAPR_LOG_LEVEL=warn   # forward warn + error (and above); raise to error to drop the rest
```

`DAPR_LOG_LEVEL` is a minimum-severity floor — `warn` captures both warnings **and** errors. It's the recommended knob for controlling daprd log volume reaching the lakehouse.
