Metadata-Version: 2.5
Name: segmentstream-pipeline
Version: 0.1.0a38
Summary: Dagster and Ibis runtime integration for SegmentStream pipelines
Project-URL: Homepage, https://segmentstream.com
Author: SegmentStream
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.12
Requires-Dist: anyio<5,>=4
Requires-Dist: dagster==1.13.16
Requires-Dist: httpx<1,>=0.28
Requires-Dist: ibis-framework<13,>=12
Requires-Dist: pydantic<3,>=2.10
Provides-Extra: bigquery
Requires-Dist: ibis-framework[bigquery]<13,>=12; extra == 'bigquery'
Description-Content-Type: text/markdown

# SegmentStream pipeline SDK

Declare source tables with `s.ingestion` and warehouse SQL transformations with
`s.model`. The SDK resolves inputs, validates explicit partitioning and capabilities,
and compiles native Dagster definitions using the existing warehouse loader.

```python
from collections.abc import Iterator

import ibis.expr.types as ir
import segmentstream as s


@s.ingestion(
    tier=s.Tier.Bronze,
    partitioning=s.Partitioning.Unpartitioned,
    schema={"amount": "float64"},
)
def orders() -> Iterator[s.IngestionRow]:
    yield {"amount": 25.0}


@s.model(
    tier=s.Tier.Gold,
    partitioning=s.Partitioning.Unpartitioned,
    schema={"total_revenue": "float64"},
)
def revenue(orders: ir.Table) -> ir.Table:
    return orders.aggregate(total_revenue=orders.amount.sum())


defs = s.definitions(
    assets=[orders, revenue],
    jobs=[s.job("revenue", assets=[revenue], include_upstream=True)],
)
```

The [workspace documentation](../../docs/README.md) is organized by concept:
[connections](../../docs/connections.md), [ingestion](../../docs/ingestion.md),
[models](../../docs/models.md), [partitioning](../../docs/partitioning.md),
[jobs](../../docs/jobs.md), [execution](../../docs/execution.md), [checks](../../docs/checks.md),
[runtime values](../../docs/runtime.md), and [warehouse storage](../../docs/warehouse.md).
The CLI bundles these same pages into every starter workspace.

`s.validate(defs)` checks the graph and named job configuration without executing
assets. `s.describe(defs)` explains resolved inputs, per-asset policies, storage,
capabilities, same-day input mappings, and job selections without resolving
credentials. See the
[modular commerce starter](../../templates/workspace-app/pipeline/README.md)
for daily row ingestion, batched models, and a partitioned check.

Daily inputs may begin earlier or end later than their consumers when they use
the same timezone and day boundaries. The upstream must cover every downstream
date. The SDK supplies strict same-day mappings for inputs and ordering-only
dependencies; no native Dagster decorator or mapping option is needed. Named
jobs still select one daily calendar, with other calendars left as inputs.

## Compatibility

The SDK contains authoring, graph inspection, Ibis resources, connection HTTP
clients, and managed secret access. Managed orchestration, workers, PostgreSQL setup, and the metadata
webserver live in the internal [pipeline runtime](../pipeline-runtime/README.md).

In `0.1.0a28`, use `s.get_connection(key).get_http_client()` for provider requests.
The client forwards requests through the backend, where provider credentials and
refresh remain. The access-token and full-credential accessors have been removed.
See [connections](../../docs/connections.md) for declaring API destinations.

Workspace pipelines declare `segmentstream-pipeline[bigquery]`; our image supplies
the runtime. The old SDK `cloud-run` and `webserver` extras and
`segmentstream.runtime` module are removed in a22.

`segmentstream.dagster` retains its native exports. Native assets and high-level
declarations can coexist in `s.definitions`. Root `s.resource` and
`s.IngestionResource` remain available for legacy ingestion. Import their helpers
from `segmentstream`; the old public `segmentstream.ingestion` module is retired
so `s.ingestion` can be the decorator. [Migration examples](../../docs/native-dagster.md)
cover the mapping and IO defaults.

## Managed Google clients

The Sandbox runtime integration supplies refreshable credentials to the built-in
BigQuery warehouse and GCS IO manager. Ibis remains the user-facing warehouse API;
its internal official Google client renews credentials automatically throughout
the authorized run. Ten minutes is the lifetime of each access token, not a limit
on pipeline duration. Managed execution does not provide general ADC support.
See [runtime lifecycle and credential brokering](../../docs/sandbox-runtime.md).
Managed Dagster execution uses the backend compute provider API for every step,
including SQL assets and checks. The coordinator owns scheduling and retries; the
backend owns execution identity, scoped credentials and cleanup. See the
[compute provider contract](../../docs/provider-compute.md).

## Releases

SDK `0.1.0a38` adds compatible daily calendars with strict same-day dependency
mappings. Consumers can start later than their upstream assets; each job still
selects one daily calendar.

The GA4 query client targets `0.1.0a37`. A model constructs it with
`s.ga4_bigquery_client(key, context=context)` and submits SQL through the backend;
BigQuery writes the result directly to the model's warehouse partition. See
[GA4 BigQuery connections](../../docs/bigquery-connections.md).
Publish the SDK before releasing a CLI whose starter
pins it. PyPI versions are immutable; do not republish an existing version.

The protected `pipeline-sdk-release.yml` workflow builds a wheel and source
distribution from a `pipeline-sdk-v<version>` tag, then uses PyPI Trusted Publishing
from the `pypi` GitHub environment. No long-lived publishing token is stored in GitHub.
