Metadata-Version: 2.5
Name: polars-telemetry
Version: 0.3.0
Summary: OpenTelemetry instrumentation for Polars query execution
Project-URL: Homepage, https://github.com/jan-krueger/polars-telemetry
Project-URL: Issues, https://github.com/jan-krueger/polars-telemetry/issues
Project-URL: Changelog, https://github.com/jan-krueger/polars-telemetry/blob/main/CHANGELOG.md
Author-email: Jan Krueger <git@krueger-jan.de>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: observability,opentelemetry,polars,telemetry,tracing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: msgpack>=1.0
Requires-Dist: opentelemetry-api>=1.20
Requires-Dist: polars>=1.44.1
Provides-Extra: datadog
Requires-Dist: datadog>=0.50; extra == 'datadog'
Provides-Extra: otlp
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.20; extra == 'otlp'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otlp'
Description-Content-Type: text/markdown

# polars-telemetry

OpenTelemetry instrumentation for [Polars](https://pola.rs) query execution.

One span per query carrying the plan, and per-node counters as metrics, to any
OTLP collector — or a profile file you open in your browser, with no collector
at all.

[![Both plans, per-node counters and diagnostics for one query in the profile viewer](https://raw.githubusercontent.com/jan-krueger/polars-telemetry/main/docs/assets/viewer.png)](https://jan-krueger.github.io/polars-telemetry/viewer/)

> [!IMPORTANT]
> **Unaffiliated with Polars and Polars Cloud.** This package attaches to an
> interface polars exposes for its own cloud product. That interface is
> internal and carries no deprecation guarantee, so it can change or disappear
> in any polars release.
>
> Supported polars: **1.44.1 – 1.44.x**. On anything else the package emits
> less — or declines to install — with a warning; it will not break your queries.

## Install

```bash
pip install polars-telemetry          # API only; bring your own OTel SDK
pip install 'polars-telemetry[otlp]'  # with SDK and OTLP exporter
pip install 'polars-telemetry[datadog]'  # for the DogStatsD exporter
```

Python 3.10+.

## Use

```python
import polars_telemetry

polars_telemetry.install()
```

`install()` enables polars' query monitoring, which sets the engine affinity to
`"streaming"` and therefore changes how your queries execute — so it never
happens on import. `uninstall()` turns monitoring off but cannot restore the
previous affinity; polars exposes no way to read it back.

## What you get

A `polars.collect` span per query, on whatever trace context was active:

- the plan — scan sources, pushed-down predicates, join types and keys,
  group-by keys
- `polars.cpu_ms`, `polars.parallelism`, result rows
- the hottest node and its share of total CPU
- diagnostics — parallel efficiency, filter selectivity, join amplification,
  projection efficiency, morsel skew, predicate pushdown, row-group skipping
- the file, line and function that ran the query, as OpenTelemetry's
  `code.*` attributes

Per-node counters — rows, morsels, polls, work-stealing, poll latency, state
updates, IO time and bytes — as 15 metric instruments dimensioned by node kind.

Every name is listed in the
[attribute reference](https://jan-krueger.github.io/polars-telemetry/reference/spans-and-metrics/).

## Where it goes

| Exporter | Sends | To |
| --- | --- | --- |
| `OTelExporter`, the default | a span and per-node metrics | your OpenTelemetry SDK |
| `DogStatsdExporter` | the same metrics, with tags | the Datadog Agent, or Telegraf into InfluxDB |
| `FileExporter` | a profile per query: both plans, every counter | a `.jsonl` file for the viewer |
| `ConsoleExporter` | a short summary | standard error |

```python
from datadog import DogStatsd
from polars_telemetry.export.dogstatsd import DogStatsdExporter

statsd = DogStatsd(disable_buffering=False, disable_background_sender=False)
polars_telemetry.install(exporter=DogStatsdExporter(statsd))
```

`exporter` takes a list, so several can run at once. Each has a
[page in the docs](https://jan-krueger.github.io/polars-telemetry/exporters/),
with its options and what it costs.

## Profiles without a collector

```python
from polars_telemetry.export.file import FileExporter

polars_telemetry.install(exporter=FileExporter("profiles/session.jsonl"))
```

One self-contained JSON document per query: both plans with every node
property, all 19 per-node counters, the diagnostics, and a fingerprint of the
plan shape.

Drop the file on the
[profile viewer](https://jan-krueger.github.io/polars-telemetry/viewer/) to read
both plans, per-node counters, and a diff between two runs of the same shape.
It runs entirely in your browser; nothing is uploaded. To try it without a
workload of your own, download a TPC-H session from [`examples/`](examples/):
the 22 queries at scale factor 1 or 10, three runs each.

## Label what runs

```python
with polars_telemetry.label("revenue_by_region"):
    report.collect()
```

The label is on the span and in the profile; nested labels join with `/`.

## Profile a block of code

```python
from polars_telemetry import profile

with profile() as session:
    report = build_report()

session.slowest.call_site  # where the slow one was run
session.write("report.jsonl")  # open in the viewer
```

Installs instrumentation only if nothing was installed. With an application
already instrumented it collects alongside the existing exporter.

## Configure

```python
from polars_telemetry import Config

polars_telemetry.install(Config(node_metrics=False))
```

| Option | Default | Effect |
| --- | --- | --- |
| `node_metrics` | `True` | Read per-node counters once at query end |
| `include_plan` | `False` | Attach the full plan to the span as JSON |
| `call_site` | `True` | Record the file, line and function that ran the query |
| `redaction` | `None` | What to mask before exporters see a query; `Redaction()` masks literal values |
| `redact_literals` | `False` | Deprecated: use `redaction=Redaction()` |
| `resource_attributes` | `{}` | Deprecated: never applied; set them on your OpenTelemetry provider |

## Your data

Spans carry plan detail: scan paths, column names, join keys and **literal
predicate values** — `col("email") == "..."` arrives verbatim, because knowing
which predicate was slow is usually the point.

- `Config(redaction=Redaction())` masks literal values: text, numbers, dates
  and times. `Redaction(paths=True, call_site=True, labels=True)` masks more.
- `redacted(exporter, ...)` gives one exporter its own setting, so a shared
  backend can get a masked copy while a local file keeps full detail.
- Literals are never used as metric attributes, at any setting.
- Attributes that can carry user data are listed in
  `polars_telemetry.export.semconv.CARRIES_USER_DATA`.

## Polars Cloud

If `polars-cloud` is installed, its observer is wrapped and forwarded to rather
than replaced. Both work at once.

## Links

- [Documentation](https://jan-krueger.github.io/polars-telemetry/)
- [Contributing](https://github.com/jan-krueger/polars-telemetry/blob/main/CONTRIBUTING.md) — development, testing, releasing
- [Changelog](https://github.com/jan-krueger/polars-telemetry/blob/main/CHANGELOG.md)

## License

Apache-2.0. See [LICENSE](https://github.com/jan-krueger/polars-telemetry/blob/main/LICENSE) and [NOTICE](https://github.com/jan-krueger/polars-telemetry/blob/main/NOTICE).
