Metadata-Version: 2.5
Name: 021-python-observability
Version: 0.3.0
Summary: OpenTelemetry metrics, traces and safe JSON logs for FastAPI services
Project-URL: Repository, https://github.com/from021-io/observability-python
Project-URL: Changelog, https://github.com/from021-io/observability-python/blob/main/CHANGELOG.md
Project-URL: Contract, https://github.com/from021-io/observability-python/blob/main/CONTRACT.md
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: asgi,fastapi,observability,opentelemetry,otlp,structlog
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: fastapi<1,>=0.116
Requires-Dist: httpx<1,>=0.28
Requires-Dist: opentelemetry-api<2,>=1.44
Requires-Dist: opentelemetry-exporter-otlp-proto-http<2,>=1.44
Requires-Dist: opentelemetry-sdk<2,>=1.44
Requires-Dist: sqlalchemy[asyncio]<3,>=2.0
Requires-Dist: structlog<27,>=25.1
Provides-Extra: prometheus
Requires-Dist: opentelemetry-exporter-prometheus<1,>=0.65b0; extra == 'prometheus'
Provides-Extra: vercel
Requires-Dist: vercel<1,>=0.5.8; extra == 'vercel'
Description-Content-Type: text/markdown

# observability-python

Repository: `from021-io/observability-python`. Python distribution: `021-python-observability`; import: `observability_021`.
Tags up to `v0.2.0` use the previous distribution name, `021-observability`.

Common API observability for Python 3.12–3.14: FastAPI request metrics and traces,
JSON stdout logs, explicit HTTPX/SQLAlchemy instrumentation, process signals,
health/readiness and domain operation metrics. Uses OpenTelemetry over HTTP/protobuf.

The package is under development. Consumer migrations are being prepared; live
Vercel/Grafana verification follows deployment. No paid registry is required to build it.

## Versions

Every merge to `main` updates a release PR, `chore(release): X.Y.Z`, built by
release-please from the PR titles (Conventional Commits). Merging it bumps the version,
writes [CHANGELOG.md](CHANGELOG.md) and tags `vX.Y.Z`. Consumers depend on a tag:

```toml
[tool.uv.sources]
021-python-observability = { git = "https://github.com/from021-io/observability-python.git", tag = "v0.3.0" } # x-release-please-version
```

Once releases are published to PyPI, pin the exact version instead; no Git credential is
needed to install it:

```sh
uv add "021-python-observability==0.3.0" # x-release-please-version
```

## Publishing to PyPI

`.github/workflows/release.yml` publishes a release with PyPI Trusted Publishing (OIDC, no
API token). It stays off until a maintainer sets the repository variable `PYPI_PUBLISH` to
`true`; then every release tag is built, checked with `twine check --strict`, smoke-tested
and uploaded through the `pypi` environment. A manual run of the workflow publishes an
existing tag, to TestPyPI (`testpypi` environment) by default. The build refuses a tag
without a `LICENSE` file or whose `pyproject.toml` version differs from the tag.

## Install locally

```sh
uv sync --all-extras
uv build
# In a consumer's environment, use the actual wheel path:
python -m pip install /path/to/021_python_observability-0.2.0-py3-none-any.whl
```

The `vercel` extra supplies the Vercel Python SDK. Install that extra in Vercel consumers;
normal Python servers do not need it. Selecting Git, a package index or a wheel URL later
does not change this integration API.

## Configure the application once

```python
from fastapi import FastAPI
from observability_021 import setup_observability

app = FastAPI()  # Existing routers, exception handlers, auth and lifespan remain here.
observability = setup_observability(app, service_name="021-api")

@app.get("/items/{item_id}")
async def get_item(item_id: str):
    return {"id": item_id}
```

Call setup while constructing the app, before serving requests. The same app object is
returned to the server: Vercel still sees a FastAPI application. Setup returns the shared
observability object, also available at `app.state.observability`. Repeated setup on the
same app returns that object without stacking providers, middleware or log handlers.

The integration wraps FastAPI's fully built ASGI middleware stack, including its error
handler. It observes the response actually sent, including early auth/rate-limit rejections
and transformed errors. It does not decorate individual handlers. Existing lifespan
startup/state/shutdown is preserved; providers flush and stop after application teardown.

## Where telemetry goes

Example deployment variables (URLs and credentials below are placeholders):

```dotenv
OTEL_SERVICE_NAME=021-api
OTEL_DEPLOYMENT_ENVIRONMENT=staging
OTEL_SERVICE_VERSION=0.1.0
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20REPLACE_WITH_CREDENTIAL
```

Use the **ingestion endpoint** of the OTLP gateway/collector, not the Grafana dashboard URL.
With that base, traces go to `/otlp/v1/traces` and metrics to `/otlp/v1/metrics`.
The header value follows the OTLP comma-separated `key=value` format with percent encoding.
Put real credentials in deployment secrets, never in the source repository or dependency URL.

Both exporters serialize binary protobuf and default to
`Content-Type: application/x-protobuf`. Custom headers can override that header;
keep its default value. A different Content-Type does not change the encoding.
Use custom headers for gateway authentication and verify receipt of both signals
against the configured gateway during rollout.

Separate destinations are supported and take precedence over the shared base:

```dotenv
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://metrics.example.com/v1/metrics
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://traces.example.com/v1/traces
OTEL_EXPORTER_OTLP_METRICS_HEADERS=Authorization=Bearer%20METRICS_CREDENTIAL
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer%20TRACES_CREDENTIAL
```

Signal-specific endpoints are used **exactly as provided**, including their paths.
Omit all endpoints for local development without remote export. Logs remain JSON on stdout
in every environment, so Vercel collects them; the package does not export logs over OTLP.

| Setting | Default / meaning |
| --- | --- |
| `OTEL_SERVICE_NAME` | Required unless supplied as `service_name=` |
| `OTEL_DEPLOYMENT_ENVIRONMENT` | Falls back to `VERCEL_ENV`, then `local`; set `staging` explicitly for staging deployments |
| `OTEL_SERVICE_VERSION` | Falls back to `VERCEL_GIT_COMMIT_SHA`, then `unknown` |
| `OTEL_EXPORTER_OTLP_TIMEOUT` | 2 seconds; Python exporter timeout and asynchronous flush wait budget |
| `OTEL_METRIC_EXPORT_INTERVAL` | 60000 milliseconds in persistent processes |
| `OBS_TRACE_SAMPLE_RATIO` | 1.0; parent-based sampling, metrics are not sampled |
| `LOG_LEVEL` | INFO |
| `VERCEL=1` | Enables the Python SDK `wait_until` continuation |

A service instance gets a unique `service.instance.id`. Code can alternatively pass an
explicit `Settings` object; that object does not implicitly load environment variables.

In persistent processes, SDK batching/periodic export and normal shutdown deliver signals.
On Vercel, the request span closes before a post-response `wait_until` flush is registered.
Flushes run off the event loop; failures produce sanitized diagnostics and do not change
business responses. The async wait budget does not kill Python worker threads: the HTTP
exporter also has its own transport timeout. Forced process termination can still lose
telemetry. Live delivery on Vercel remains a deployment validation, not a unit-test claim.

## Dependencies and business operations

HTTPX clients opt in at construction, using a fixed dependency name. Use the sync equivalent
`observability.httpx_transport` for a synchronous `httpx.Client`.

```python
import httpx

client = httpx.AsyncClient(
    transport=observability.async_httpx_transport(dependency="company-brain"),
    base_url=brain_url,
)
# Close the client using the service's existing lifespan.
observability.instrument_sqlalchemy(engine, dependency="database")
```

Pass an existing HTTPX transport through `transport=` if it carries custom TLS, proxy,
retry or test behavior. Explicit per-host mounts also need their own observed transport.
SQLAlchemy supports both Engine and AsyncEngine. Neither adapter records URLs, hostnames,
SQL statements, SQL parameters, exception messages or response bodies. Client spans inject
W3C trace context; HTTP requests also propagate the validated request ID.

Readiness checks remain owned by each service:

```python
from sqlalchemy import text

async def database_ready():
    async with engine.connect() as connection:
        await connection.execute(text("SELECT 1"))

observability.add_health_routes(app, readiness={"database": database_ready})
```

This explicitly registers `/health` (process alive) and `/ready` (all declared checks
succeed). Checks must be async and cancellation-cooperative; False means unavailable,
None/other successful returns mean available. A failure or timeout returns sanitized 503.
Existing routes are never silently replaced. Migrate monitors and auth exemptions first.
With no declared checks, readiness represents the process only.
The JSON contract is `status: ok|unavailable` plus `checks`, not `status: ready`;
see the exact [probe contract](CONTRACT.md#probes-and-correlation).

Generic domain instrumentation is explicit and works around synchronous code or `await`:

```python
with observability.operation("brain.ingest"):
    await ingestion_service.run()

observability.increment("app.documents.count", operation="ingest", status="ok")
observability.observe("app.ingest.duration", 12.5, operation="ingest", status="ok")
```

Use static metric/operation/dependency names from code, never IDs or user-provided text.
Operation/result remain separate dimensions. `operation()` reports `ok`, `error` or `cancelled`; explicit helpers also preserve static domain outcomes.

## Logging

`get_logger(__name__)` provides structlog logging. Use static event names and approved
structured fields (`operation`, `op`, `action`, `dependency`, `status`, `reason`, `error_type`). Do not
interpolate messages or pass business payloads into approved fields. Unknown extra fields
and raw exception objects/messages are not serialized. Foreign stdlib messages are replaced
with a safe event and logger name; exporter failures get `telemetry.export_failed`.

Logs also retain UUID correlation fields: `brain_id`, `job_id`, `file_id`,
`file_version_id`, `entity_id`, `event_id` and `correlation_id`. Pass a UUID object or
UUID string; valid values are normalized and included in JSON and searchable `message`.
Invalid values and unknown fields are omitted. These fields are **never metric labels**.

Use `logger.exception("ingestion.failed", brain_id=brain_id, job_id=job_id)` inside an
exception handler, or `logger.error(..., exc_info=error)` for a captured exception.
Stdlib `logging.exception` / `logging.error(..., exc_info=True, extra={...})` and
Uvicorn errors use the same formatter. The actual exception supplies `error_type` and
an `exception` array, ordered from the current error through its cause/context chain:

```json
{"type":"ValueError","frames":[{"file":"worker.py","function":"ingest","line":42}],"frames_truncated":false}
```

Each entry contains only the type and frame locations, never exception messages,
source lines, locals or absolute directories. At most 8 chained exceptions and the
32 frames nearest each failure are retained; `exception_truncated` and each entry's
`frames_truncated` flag indicate omitted diagnostics. Explicit `raise ... from None`
suppresses the previous context. Exception-group members are not expanded; their group
type and its own frames are retained. Type/function names are capped at 128 characters
and filenames at 256. Diagnostic fields are generated from the exception, not accepted
as caller-supplied JSON. Unhandled ASGI failures include these diagnostics in the single
`request.completed` log; response codes, exception propagation and span data are unchanged.

Each HTTP request emits one `request.completed` line with service identity, a validated
UUID request ID, trace/span IDs, method, route template, status and duration. Invalid inbound
request IDs are replaced. IDs appear in logs, not metric labels. Setup owns root logging
configuration: remove the consumer's previous formatter/configuration during migration.

## Contract and checks

See [CONTRACT.md](CONTRACT.md) for signal names, units, buckets and measurement boundaries.

```sh
make check
uv build
```

Tests exercise actual ASGI responses, a local OTLP HTTP/protobuf receiver, a distributed
API-to-API call, SQLAlchemy sync/async operations, exporter failures and package installation.
No test needs Grafana credentials or calls a deployed service.

## Existing domain ports and scrape endpoints

For consumers retaining `/metrics`, install `[prometheus]`, pass `prometheus=True` at setup
and return `obs.prometheus_latest()` from the endpoint. Metrics are read from the same OTel
provider used by OTLP. Existing dashboards still need the explicit name/label migration in
CONTRACT.md; the reader does not recreate legacy instrument names.

Domain helpers accept static `attributes` from the documented allowlist. Scalar histograms
can use `unit="1"` (or `"USD"` for already-existing cost observations); duration histograms
stay in seconds. This preserves consumer domain ports without reimplementing exporters or
instrument caches.

## Grafana synchronization tool

[`tools/grafana/sync.py`](tools/grafana/sync.py) is a standalone Python 3.12–3.14
CLI using only the standard library. Consumers check it out at a release tag
(`ref: vX.Y.Z`), the same tag their runtime dependency pins. The package does
not import or run it. Services own dashboard JSON, queries, inventories, secrets
and workflow triggers; this repository owns the synchronization mechanism and tests.

Pass `--manifest /path/to/service/grafana/dashboards.json`: a nonempty JSON array
of paths relative to that manifest, for example `["overview.json", "usage/funnel.json"]`.
Paths must stay inside its directory, refer to JSON files and have unique dashboard
UIDs. No automatic file discovery or remote dashboard deletion is performed.

```sh
python3 tools/grafana/sync.py --manifest /path/to/service/grafana/dashboards.json --validate
python3 -m unittest discover -s tools/grafana -p 'test_sync.py' -v
# With the service's Actions configuration, authenticated GETs only:
python3 tools/grafana/sync.py --manifest /path/to/service/grafana/dashboards.json --dry-run
```

Publication requires `GRAFANA_URL`, `GRAFANA_SERVICE_ACCOUNT_TOKEN` and
`GRAFANA_COMMIT` (the **consumer** checkout SHA). Grafana Cloud additionally needs
`GRAFANA_NAMESPACE=stacks-<instance ID>`; OSS defaults to `default`.
`GRAFANA_FOLDER_UID` is required for creation; existing folders are preserved.
`GRAFANA_DATASOURCES` is an optional JSON mapping of variable names to datasource
UIDs, required when a new datasource variable has no saved selection.
Keep these values in consumer deployment configuration. The tool preserves saved
variable selections, refresh and time range; preflights the entire inventory before
writing; rejects redirects; sanitizes errors; and verifies every write by reading it back.
`--dry-run` reports `preflight_verified` and never checks write permission or panel queries.

The existing `make check` and package CI matrix run the common tests. Consumer
workflows validate their own manifests and fetch the same reviewed tool revision using
their existing read-only `OBSERVABILITY_GITHUB_TOKEN`, with `persist-credentials: false`.
Move the CI tool pin to a new tag through a PR after its checks pass. A tool-only
release does not require moving the runtime dependency, but keeping both on the same
tag is the default.

## Consumer rollout checklist

- Moving from `021-observability` to `021-python-observability`: rename the dependency
  (with its extras) in `[project] dependencies`, the `[tool.uv.sources]` key or the
  `requirements.txt` line, and regenerate the lock. Imports (`observability_021`) and every
  signal name stay the same.
- Merge the release PR that release-please opens (`chore(release): X.Y.Z`), then pin both
  consumers to its tag `vX.Y.Z` (`tag = "vX.Y.Z"` in `[tool.uv.sources]`, `ref: vX.Y.Z` for
  the Grafana tool); regenerate API lock/export and rerun both consumer gates.
- Update readiness JSON assertions and dependency monitors together. In Brain, old
  dependency checks moved from `/health` to `/ready`; liveness alone cannot prove readiness.
- Migrate existing metric queries, labels, duration units and alert thresholds together;
  if none exist, record that inventory explicitly before rollout.
- Replace log searches by email/raw path with the response's UUID `X-Request-ID` and
  `trace_id`. Confirm callers accept a replacement ID when they send a non-UUID value.
- Retain the `vercel` extra in Vercel consumers: `wait_until` keeps export work running
  after the response. While consuming from Git, configure the package PAT separately for
  each deployment environment; a PyPI pin needs none.

## License

Apache License 2.0. See [LICENSE](LICENSE).
