Metadata-Version: 2.4
Name: overture-schema-system
Version: 0.1.1.dev0
Summary: Foundational types at the base of the Overture Maps schema system
License-Expression: MIT
Requires-Dist: pydantic>=2.13.0
Requires-Dist: shapely>=2.0.0
Requires-Dist: typing-extensions>=4.16
Maintainer: Overture Maps Schema Working Group
Requires-Python: >=3.10
Project-URL: Homepage, https://overturemaps.org
Project-URL: Source, https://github.com/OvertureMaps/schema
Project-URL: Issues, https://github.com/OvertureMaps/schema/issues
Description-Content-Type: text/markdown

# Overture Schema System

Write Pydantic models once, get validated data that serializes correctly to JSON, Parquet, and Spark. This package provides the numeric and geometric types, constraint decorators, and GeoJSON-aware base class that make Pydantic models portable across serialization targets.

## Installation

```bash
pip install overture-schema-system
```

## Feature

GeoJSON-compatible Pydantic base model. Subclasses serialize to the GeoJSON format automatically -- `geometry` and `id` at the top level, everything else under `properties` -- and validate from it:

```python
from overture.schema.system.feature import Feature
from overture.schema.system.geometric import Geometry
from overture.schema.system.numeric import float32


class Mountain(Feature):
    name: str
    max_elevation: float32


m = Mountain(
    geometry=Geometry.from_wkt("POINT(86.9252 27.9888)"),
    name="Mount Everest",
    max_elevation=8848.86,
)
```

## Numeric Types

Using `int` and `float` in a Pydantic model produces valid Python but loses information downstream -- an `int` field becomes a 64-bit integer in Parquet, Arrow, and Spark StructTypes, even when a single byte would hold every value. How wide a field needs to be is the schema author's choice, but `int` and `float` give Pydantic no way to record it. The numeric types (`uint8`, `int32`, `float32`, etc.) do: each declares a width that maps to the correct wire type in every serialization target, so a single portable declaration round-trips cleanly between Python, Parquet, Spark, and JSON Schema:

```python
from pydantic import BaseModel
from overture.schema.system.numeric import uint8, float32


class Building(BaseModel):
    height: float32 | None = None
    num_floors: uint8 | None = None
```

Integer types: `uint8`, `uint16`, `uint32`, `int8`, `int16`, `int32`, `int64`. Float types: `float32`, `float64`.

## Geometric Types

`Geometry` and `BBox` wrap Shapely and GeoJSON-compatible geometry and bounding box values so they can participate in a Pydantic model as fields, with `GeometryType` and `GeometryTypeConstraint` available to restrict a `Geometry` field to specific geometry types:

```python
from overture.schema.system.geometric import (
    Geometry,
    GeometryType,
    GeometryTypeConstraint,
)
```

Types: `Geometry`, `BBox`, `GeometryType`, `GeometryTypeConstraint`.

## String Types

Validated string types that carry their constraints into generated JSON Schemas and downstream code generation. Using `CountryCodeAlpha2` instead of `str` means Pydantic rejects `"USA"` at validation time, JSON Schema gets the right pattern, and codegen tools produce typed output:

```python
from overture.schema.system.string import CountryCodeAlpha2, LanguageTag
```

Available types: `CountryCodeAlpha2`, `RegionCode`, `LanguageTag`, `HexColor`, `JsonPointer`, `PhoneNumber`, `StrippedString`, `SnakeCaseString`, `NoWhitespaceString`, `WikidataId`.

## Field Constraints

Annotations for Pydantic fields that enforce domain rules beyond what the type alone expresses. Each constraint produces the corresponding JSON Schema keywords (e.g., `pattern`, `uniqueItems`) and is introspectable by code generation tools -- unlike Pydantic's `@field_validator`, which runs in Python only. Apply via `Annotated`:

```python
from typing import Annotated
from pydantic import BaseModel, Field
from overture.schema.system.field_constraint import (
    UniqueItemsConstraint,
    PatternConstraint,
)

OsmIdConstraint = PatternConstraint(
    pattern=r"^[nwr]\d+$",
    error_message="invalid OSM ID format: {value}. Must be n123, w123, or r123.",
)


class MyModel(BaseModel):
    osm_id: Annotated[str, OsmIdConstraint]
    tags: Annotated[list[str], UniqueItemsConstraint()] = Field(min_length=1)
```

Built-in constraints include `PatternConstraint`, `StrippedConstraint`, `UniqueItemsConstraint`, and all the string-type constraints (`CountryCodeAlpha2Constraint`, `HexColorConstraint`, etc.). All produce error messages with domain context.

## Model Constraints

Class-level decorators for cross-field validation -- relationships between fields that no single field annotation can express. Each decorator produces corresponding JSON Schema constructs (`if`/`then`, `anyOf`, etc.) and is introspectable for code generation:

```python
from pydantic import BaseModel
from overture.schema.system.model_constraint import require_any_of


@require_any_of("email", "phone")
class Contact(BaseModel):
    email: str | None = None
    phone: str | None = None
```

- `@require_any_of("a", "b", ...)` -- at least one field must be non-None
- `@require_any_true(cond1, cond2, ...)` -- at least one condition must evaluate to true
- `@radio_group("a", "b", ...)` -- at most one field may be truthy
- `@require_if("target", condition)` -- field required when condition holds
- `@forbid_if("target", condition)` -- field forbidden when condition holds
- `@min_fields_set(n, "a", "b", ...)` -- at least *n* fields must be set
- `@no_extra_fields` -- reject unrecognized fields (equivalent to `model_config = ConfigDict(extra="forbid")`)

## References

Foreign-key-style annotations that describe relationships between models. These carry no runtime enforcement but provide metadata for code generation and documentation tools:

```python
from typing import Annotated
from overture.schema.system.ref import Id, Identified, Reference, Relationship


class Park(Identified):
    pass


class ParkBench(Identified):
    park_id: Annotated[Id, Reference(Relationship.COMPOSITION, Park, role="part_of")]
```

## Discovery

Packages register models on the `overture.models` Python entry point group. Each entry maps a name to a class import path:

```toml
[project.entry-points."overture.models"]
building      = "overture.schema.buildings:Building"
building_part = "overture.schema.buildings:BuildingPart"
```

`discover_models()` walks the group, loads each entry point, and returns a dict keyed by `ModelKey`. Consumers iterate over the result without knowing which package owns any given model -- the CLI and codegen tools both run discovery to assemble their working set.

A `ModelKey` carries the entry point `name`, its `entry_point` value (`"module:Class"`), and a `frozenset[str]` of tags. [Tagging](#tagging) is how those tags get attached.

## Tagging

Tags classify discovered models. A package registers [tag providers](#providers) on `overture.tag_providers`; when `discover_models` runs, it asks every provider which tags apply to each model and attaches the resulting set to its `ModelKey`. Downstream tools read those tags -- the CLI's `--tag` filter, codegen's grouping logic, anything that reasons about a model without importing it.

```python
from overture.schema.system.discovery import (
    TagSelector,
    discover_models,
    filter_models,
)

models = discover_models()

selected = filter_models(
    models,
    TagSelector(include_any=("feature",), exclude_any=("draft",)),
)
```

### Format

Tags follow `[namespace:]predicate[=value]`:

- **Plain** -- `feature`, `overture`
- **Namespaced** -- `system:extension`
- **Key/value** -- `overture:theme=buildings`

`:` separates namespace from predicate -- one level only, no nested colons. `=` introduces a discrete value, one per tag. Predicate and namespace parts are lowercase alphanumeric (with `_`, `.`, `-`); values also accept uppercase. Matching is case-sensitive throughout.

Helpers in `overture.schema.system.discovery.tag` parse structured tags:

- `is_valid_tag(tag)` -- check whether a string matches the format
- `get_namespace(tag)` -- extract the namespace prefix, or `""` for a plain tag
- `get_values_for_key(tags, "overture:theme")` -- extract values from k/v tags with the given key

### Providers

A tag provider is a callable registered on the `overture.tag_providers` entry point group. Discovery passes it the concrete `BaseModel` subclasses for the entry point and a copy of the tags accumulated so far; tags it adds are merged into the running set after passing the reservation checks below.

```python
from collections.abc import Iterable
from pydantic import BaseModel
from overture.schema.system.discovery import ModelKey
from overture.schema.system.feature import Feature


def feature_provider(
    types: Iterable[type[BaseModel]],
    key: ModelKey,
    tags: set[str],
) -> set[str]:
    if any(issubclass(tp, Feature) for tp in types):
        tags.add("feature")
    return tags
```

```toml
[project.entry-points."overture.tag_providers"]
feature = "overture.schema.system.discovery.tag_providers:feature_provider"
```

Tags from one provider are visible to providers that run later, but execution order is unspecified -- a provider must not depend on tags added by another. Provider exceptions are caught, logged, and discarded; they do not abort discovery.

Discovery resolves the entry-point value to concrete classes before invoking providers. For class entries that yields a one-element iterable; for discriminated-union features (e.g. `Segment`, which loads as `Annotated[Union[...], Field(...)]`) it yields every arm. Providers therefore work uniformly with `issubclass` and never need to walk type expressions themselves.

### Reservation

Specific plain tags and namespaces are reserved for designated packages. For example:

| Tag or namespace | Owning package |
|---|---|
| `feature` (tag) | `overture-schema-system` |
| `system:` (namespace) | `overture-schema-system` |
| `overture` (tag) | `overture-schema-common` |
| `overture:` (namespace) | `overture-schema-common` |

When a provider attempts to set a reserved tag from an unauthorized package, discovery logs a warning and discards the tag.

### Built-in Providers

- **`feature`** (in `system`) -- adds `feature` if any concrete arm is a `Feature` subclass.
- **`overture`** (in `common`) -- adds `overture` if any concrete arm is an `OvertureFeature` subclass: the model is built on Overture's feature model, as distinct from `feature`, which says only that it is a `Feature`. Consumers that need to ask that question read the tag rather than importing `OvertureFeature`. The tag does not assert that the type belongs to the Overture schema -- a third-party `OvertureFeature` subclass receives it too, and the reservation governs who may emit the tag, not which models get it.
- **`theme`** (in `common`) -- adds `overture:theme={theme}` for each `OvertureFeature` referenced. A discriminated-union feature whose arms span multiple themes contributes one tag per distinct theme.

### Selecting Models by Tag

`filter_models(models, selector)` applies `TagSelector` predicates against each `ModelKey.tags`:

- `include_any` -- OR scope; at least one tag must match (empty: no scope filter)
- `require_all` -- AND narrowing; every tag must be present (empty: no narrowing)
- `exclude_any` -- OR-NOT subtraction; any match drops the model

An empty selector returns the input unchanged.

## Also Included

- **Optionality** -- `Omitable[T]` models JSON Schema's "may be absent but not null" semantics, which Pydantic's `T | None` conflates with nullable.
- **DocumentedEnum** -- base class for enumerations whose members carry their own docstrings, enabling code generation tools to produce documented output.
- **Metadata** -- internal key-value store used by model constraints to attach data to classes.
- **JSON Schema** -- schema generator that treats `T | None = None` as "omit when unset" rather than Pydantic's default "nullable with null default." Also handles unions of models.

## Baseline Testing

`overture.schema.system.testing` provides a pytest plugin for golden-file baseline tests of generated JSON Schemas. Implementers building feature packages use it to detect unintended schema drift.

### Helpers

- `assert_golden(actual, golden_path, *, update)` -- compare a string against a golden file. On mismatch, raises `AssertionError` with a unified diff. When `update=True`, writes `actual` to `golden_path` instead of comparing.
- `assert_json_schema_golden(model_or_union, golden_path, *, update)` -- generate the JSON Schema for a Pydantic model (or discriminated union type alias) via `overture.schema.system.json_schema` and delegate to `assert_golden`.

### Opting In

Activation is opt-in so the `--update-baselines` flag does not pollute pytest runs of packages that do not declare it. Add the plugin to your package's `pyproject.toml`:

```toml
[project.entry-points.pytest11]
overture_baselines = "overture.schema.system.testing.plugin"
```

The plugin registers:

- `--update-baselines` -- pytest CLI flag.
- `update_baselines` -- bool fixture, true when the flag is passed.

### Writing a Baseline Test

```python
from pathlib import Path

import pytest
from overture.schema.system.testing import assert_json_schema_golden

from yourpackage import Mountain  # the model you're locking down

GOLDEN = Path(__file__).parent / "mountain_baseline_schema.json"


@pytest.mark.baseline
def test_mountain_json_schema(update_baselines: bool) -> None:
    assert_json_schema_golden(Mountain, GOLDEN, update=update_baselines)
```

Convention: the golden file lives next to the test, named `<feature>_baseline_schema.json`.

Mark each baseline test with `@pytest.mark.baseline` so it can be selected or skipped via `pytest -m baseline` / `pytest -m "not baseline"`. Register the marker in your pyproject:

```toml
[tool.pytest.ini_options]
markers = [
    "baseline: golden file baseline tests",
]
```

### Updating Baselines

After an intentional schema change:

```bash
pytest -m baseline --update-baselines
```

Inspect `git diff` on the regenerated golden files to confirm the changes are intended before committing.
