Metadata-Version: 2.4
Name: osw-sanitizer
Version: 0.2.1
Summary: Python package for sanitizing OpenSidewalks dataset ZIP files
Author-email: Sujata Misra <sujatam@gaussiansolutions.com>
License-Expression: MIT
Project-URL: Documentation, https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-sanitizer/blob/main/README.md
Project-URL: GitHub, https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-sanitizer
Project-URL: Changelog, https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-sanitizer/blob/main/CHANGELOG.md
Keywords: opensidewalks,osw,geojson,sanitization,tdei
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ijson>=3.2
Requires-Dist: python-osw-validation==0.5.0
Dynamic: license-file

# OSW Sanitizer

[![Unit Tests](https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-sanitizer/actions/workflows/unit_tests.yml/badge.svg)](https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-sanitizer/actions/workflows/unit_tests.yml)
[![Coverage](https://img.shields.io/badge/coverage-%3E90%25-brightgreen)](#testing)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![Package](https://img.shields.io/badge/package-osw--sanitizer-blue)](https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-sanitizer)

`osw-sanitizer` is a Python package for sanitizing OpenSidewalks (OSW)
dataset ZIP files. It is designed to be consumed by the TDEI sanitization
service and by other Python workflows that need the same deterministic cleanup
behavior.

## What It Does

Given a dataset ZIP, the sanitizer runs these passes in order and reports every
change it made:

1. **Drops files that do not belong** — non-OSW filenames and macOS packaging
   metadata (`__MACOSX/`, `._*` resource forks, `.DS_Store`).
2. **Removes broken tags** — JSON `null` and numeric `NaN` property values.
   Look-alike strings (`"None"`, `"null"`, `"nan"`, `"n/a"`, `"na"`) and falsy
   but meaningful values (`0`, `false`, `""`) are kept.
3. **Shortens coordinates** to a configurable precision, by rounding (default)
   or truncating.
4. **Splits oversized geometry** — lines and polygon rings carrying more
   vertices than the limit are broken into consecutive parts within it.
5. **Creates missing nodes** for every `_u_id` / `_v_id` / `_w_id` that no node
   declares.
6. **Enforces unique node `_id`s** — identical repeats dropped, conflicting ones
   re-ided.
7. **Collapses duplicate nodes** sharing coordinates and tags, repointing
   references at the survivor.
8. **Verifies the graph is intact**, then **validates the result** with
   `python-osw-validation` and bundles everything into `osw_data.zip`.

## Installation

```bash
pip install osw-sanitizer
```

For local development:

```bash
python -m pip install -e .
python -m pip install pytest coverage
```

## Quick Start

```python
from osw_sanitizer import OSWSanitization, SanitizationConfig

config = SanitizationConfig(
    coordinate_precision=7,
    coordinate_rounding="round",  # or "truncate"
)

result = OSWSanitization(
    input_path="/path/to/input.zip",
    output_dir="/path/to/output",
    config=config,
).sanitize()

if result.success:
    print(result.updated_dataset_zip)   # osw_data.zip, the published bundle
    print(result.fixes_json)
else:
    print(result.message)               # includes any validator issues
```

## Service-Compatible API

`OSWSanitization.sanitize_dataset(...)` returns the same information as a
dictionary:

```python
from osw_sanitizer import OSWSanitization

result = OSWSanitization.sanitize_dataset(
    input_zip_path="/path/to/input.zip",
    output_dir="/path/to/output",
)

print(result["success"])
print(result["message"])
print(result["updated_dataset_zip"])
print(result["fixes_json"])
```

`SanitizationProcessor` is retained as an alias of `OSWSanitization`.

## Configuration

| Option | Default | Description |
|---|---:|---|
| `coordinate_precision` | `7` | Maximum decimal places retained for coordinate values. |
| `coordinate_rounding` | `"round"` | How a too-long coordinate is shortened: `"round"` to the nearest value, halves away from zero, or `"truncate"` toward zero. |
| `max_geometry_vertices` | `2000` | Lines and polygon rings carrying more vertices than this are split into parts within the limit. |
| `validate_output` | `True` | Validate the sanitized dataset and publish `osw_data.zip`. Set `False` to sanitize without judging the result against the OSW schema. |

The configuration names match the OSW formatter and validator packages where
applicable. Configuration is passed in code — the package reads no environment
variables and no `.env` file.

Each option can also be passed directly to the constructor:

```python
OSWSanitization(input_path=..., output_dir=..., coordinate_precision=6)
```

## Input Requirements

`input_path` must point to an existing `.zip` archive. The sanitizer returns an
unsuccessful `SanitizationResult` without writing any output when:

| Input | Message |
|---|---|
| Missing path | `Input dataset path is missing` |
| Path does not exist | `Input dataset not found at path: <path>` |
| Not a `.zip` filename | `Input dataset must be a .zip file: <path>` |
| `.zip` filename that is not a zip archive | `Input dataset is not a valid zip archive: <path>` |

## Supported Dataset Files

Supported filenames come from `python-osw-validation`, so the sanitizer keeps
exactly the files the OSW validator accepts. The dataset keys are
`OSW_DATASET_FILES`:

- `edges`
- `lines`
- `nodes`
- `points`
- `polygons`
- `zones`

Supported filename forms are:

- `<dataset>.geojson`
- `<dataset>.OSW.geojson`
- `*.<dataset>.geojson`
- `*.<dataset>.OSW.geojson`

Matching is case-insensitive.

### Removed Files

These are omitted from the sanitized output and recorded under `removedFiles`
in `fixes.json`:

| File | `fixType` |
|---|---|
| Non-OSW filenames, including unsupported `.geojson` names and non-geojson files | `unsupported_file_removed` |
| `__MACOSX/` entries, `._*` resource forks, `.DS_Store` | `macos_metadata_removed` |

## Coordinate Precision

Coordinates already within `coordinate_precision` are left byte for byte as
they are — never padded with trailing zeros — and are not reported in
`precisionUpdates`. Only longer fractions are shortened, either way:

| Input | `"round"` | `"truncate"` |
|---|---|---|
| `-122.123456789` | `-122.1234568` | `-122.1234567` |
| `47.12345674` | `47.1234567` | `47.1234567` |
| `47.12345675` | `47.1234568` | `47.1234567` |
| `-47.12345675` | `-47.1234568` | `-47.1234567` |

Rounding moves a point by at most half a unit of the last digit and has no
directional bias; truncation always moves toward zero, so it biases a dataset
slightly. Either way a coordinate can shift, which is why an edge endpoint can
end up marginally off its node — see [Graph Verification](#graph-verification).

## Geometry Vertex Limit

Features in `edges`, `lines`, `polygons`, and `zones` are held to
`max_geometry_vertices`, counted the way the OSW validator counts: every
LineString coordinate, and every polygon ring vertex except the repeated
closing one. A feature over the limit is split into consecutive parts, each
`<id>-part-<n>`, and logged under `splitGeometries`.

- **Lines** are cut into runs, with consecutive parts sharing the vertex they
  were cut at, so the line stays continuous and no vertex is lost. For edges,
  each cut becomes a `<id>-split-node-<n>` and the parts' `_u_id` / `_v_id` are
  rewired through it; the nodes themselves are materialized by the reference
  pass at the coordinates those ids imply.
- **Polygon rings** are cut into runs, each closed back on itself into its own
  ring, each keeping at least three vertices.
- A polygon with **interior rings** has no split that carries its holes, so it
  is left as it is and logged as `oversized_geometry_not_split`.

Parts are sized evenly rather than packed to the limit — 2500 vertices at a
limit of 2000 becomes 1251 + 1250, not 2000 + 501. Packing leaves a short tail,
and a few vertices off a smooth curve collapse into a degenerate sliver once
the coordinates are rounded.

Two things to know about splitting polygons:

- Closing each ring chunk cuts across the polygon, which reproduces the
  original area only where the ring is convex. A concave ring's parts tile it
  approximately, so treat this as a way to satisfy the vertex limit rather than
  an exact partition.
- A limit small enough that a part spans less ground than `coordinate_precision`
  resolves will still produce degenerate rings, which the validator then
  rejects as invalid geometry. This does not arise at the default limit; it
  needs a limit orders of magnitude smaller on densely sampled curves.

## Node Topology

Every `_u_id` / `_v_id` (edges) and `_w_id` (zones) must name a node. The
sanitizer treats the reference id as authoritative rather than repointing it:

- `_u_id` is the edge's first vertex, `_v_id` its last.
- The n-th `_w_id` is the n-th vertex of the zone's outer ring, with the
  repeated closing vertex dropped first.
- A reference no node declares gets a node created with that exact `_id`, at
  the coordinate the reference implies. A dataset can therefore come out with
  more nodes than it went in with.
- References that already resolve are left alone, even when the node sits away
  from the vertex.
- If the coordinate cannot be determined — a `_w_id` count that does not match
  the ring, or an empty reference id — nothing is invented and the reference is
  logged under `unresolvedReferences`.

### Duplicate Nodes

Duplicate node `_id`s are not allowed: a reference has to name exactly one
node. Within the nodes file the first feature to claim an `_id` keeps it, so
existing references stay pointed at the same node. A later repeat is dropped
when identical, and otherwise reassigned the next free `<id>-<n>`.

Nodes that share both their coordinates and their tags describe the same place,
so they are collapsed into the first of them. Every `_u_id` / `_v_id` / `_w_id`
pointing at a collapsed node is repointed at the survivor, logged as
`collapsedNodes` and `updatedReferences`. Comparison happens after rounding, so
nodes differing only below the precision limit collapse too. Nodes at the same
place with different tags are left alone.

### Graph Verification

After the fixes are applied, the sanitizer walks every reference once more and
records the outcome under `verification` in `fixes.json`:

```json
{
  "verification": {
    "nodeCount": 12,
    "referenceCount": 14,
    "graphIntact": true,
    "danglingReferences": [],
    "misplacedReferences": []
  }
}
```

- `graphIntact` is true when every reference resolves to a node that exists.
- `danglingReferences` holds references the sanitizer already reported as
  unplaceable; they are findings, not failures, and the run still succeeds.
- `misplacedReferences` holds references that resolve to a node sitting away
  from the vertex they describe. The reference is authoritative, so the node is
  never moved — but rounding can shift an endpoint off its node by up to one
  unit of the configured precision, and this is where that shows up.
- A dangling reference that was *not* reported as unplaceable means a preceding
  pass broke the graph. That fails the run rather than shipping a broken
  dataset.

## Output Artifacts

A validated run publishes `osw_data.zip` into `output_dir`, and that is what
`result.updated_dataset_zip` points at. It bundles:

1. The sanitized dataset ZIP, under the same filename as the input ZIP.
2. `fixes.json`, structured details about every applied change.
3. `validation_issues.json`, the validator's issues as `{"issues": [...]}`.

All three are also left loose in `output_dir`, so `result.fixes_json` points at
`fixes.json` on disk rather than inside the bundle. The bundle is published
whether or not the dataset validates — only `success` and `message` differ.

```python
result.updated_dataset_zip   # .../output/osw_data.zip
result.fixes_json            # .../output/fixes.json
```

With `validate_output=False` there is no bundle: nothing has vouched for the
dataset, so `updated_dataset_zip` is the sanitized dataset ZIP itself and no
`validation_issues.json` is written.

## Output Validation

Once sanitization finishes, the sanitized ZIP is handed to
`python-osw-validation`, configured from the same settings that produced it —
`coordinate_precision` and `max_geometry_vertices` are passed through, so the
output is judged by the limits it was sanitized to.

Either outcome publishes the same three artifacts; what changes is the result
and what `validation_issues.json` holds:

| Outcome | `success` | `message` | `validation_issues.json` |
|---|---|---|---|
| Validates | `True` | what was sanitized | `{"issues": []}` |
| Rejected | `False` | the validator's issues | the issues, so a caller can fix the dataset |

A rejected run reports the issues in the message as well:

```
Sanitized dataset is not a valid OSW dataset.
- edges.geojson (feature 0): "" is shorter than 1 character (at: features[0].properties._u_id)
```

`SanitizedDatasetValidationError` keeps the raw `issues` alongside the rendered
`messages`, and `format_issues(...)` renders any issue list the same way.

Note that the validator enforces that an edge endpoint sits exactly on its
node. Rounding can move an endpoint off its node — the sanitizer reports that
under `misplacedReferences` but does not repair it, so such a dataset
sanitizes cleanly and then fails validation here.

## fixes.json

Per-file entries carry only the keys that apply:

| Key | Written when |
|---|---|
| `removedTags` | a `null` or `NaN` tag was dropped |
| `precisionUpdates` | a coordinate was rounded or truncated |
| `addedNodeReferences` | a dangling reference caused a node to be created |
| `unresolvedReferences` | a reference could not be placed |
| `addedNodes` | nodes were added to the nodes file |
| `removedNodes` | a repeated `_id` on an identical node was dropped |
| `reassignedNodeIds` | a repeated `_id` on a differing node was re-ided |
| `collapsedNodes` | duplicate nodes were collapsed |
| `updatedReferences` | a reference followed a collapsed node |
| `splitGeometries` | a feature exceeded the vertex limit |

Alongside them, `removedFiles` lists dropped files and `verification` reports
the graph check.

```json
{
  "jobId": "",
  "files": [
    {
      "filename": "edges.geojson",
      "removedTags": [
        {
          "featureIndex": 0,
          "tag": "width",
          "value": null
        }
      ],
      "precisionUpdates": [
        {
          "featureIndex": 0,
          "coordinatePath": "coordinates[0]",
          "original": "-122.123456789",
          "updated": "-122.1234568",
          "precision": 7,
          "rounding": "round"
        }
      ]
    }
  ],
  "removedFiles": [],
  "verification": {
    "nodeCount": 0,
    "referenceCount": 0,
    "graphIntact": true,
    "danglingReferences": [],
    "misplacedReferences": []
  }
}
```

A removed `NaN` is logged as the string `"NaN"`, so `fixes.json` stays
parseable by strict JSON readers.

## Testing

Install the package and test dependencies:

```bash
python -m pip install -e .
python -m pip install pytest coverage
```

Run the unit tests:

```bash
python -m pytest
```

Run the unit tests with coverage enforcement:

```bash
coverage run -m pytest
coverage report --fail-under=90
```

The GitHub Actions unit test workflow writes timestamped test and coverage
logs into `test_results/` and uploads them to Azure Blob Storage using the
`AZURE_STORAGE_CONNECTION_STRING` secret.

Package metadata is defined in `pyproject.toml`. `setup.py` is retained as a
compatibility shim for legacy packaging workflows.

### Test Datasets

Sample dataset ZIPs are checked in under `tests/assets`. The six-file OSW
datasets (edges, lines, nodes, points, polygons, zones) are generated by
`tests/dataset_builder.py` and carry the OSW 0.3 `$schema`:

| Dataset | Covers | Sanitize result |
|---|---|---|
| `passed.zip` | clean dataset, no fixes applied | passes validation |
| `missing_references.zip` | dangling `_u_id` / `_v_id` / `_w_id` | passes validation |
| `precision_and_duplicates.zip` | over-long coordinates and repeated node ids | passes validation |
| `collapsible_nodes.zip` | duplicate nodes that collapse into one | passes validation |
| `rounding_modes.zip` | coordinates where `round` and `truncate` differ | passes validation |
| `null_and_nan_tags.zip` | null / NaN tags and their look-alikes | passes validation |
| `cleanup.zip` | macOS metadata and unsupported filenames | passes validation |
| `misplaced_references.zip` | references resolving away from their vertex | fails validation |
| `unresolvable_references.zip` | references the sanitizer will not guess at | fails validation |
| `failure.zip` | non-finite coordinates and property values | fails sanitization |
| `not_a_zip.geojson`, `corrupt.zip` | invalid inputs for the ZIP-only check | rejected as input |

The last four fail by design. The two that fail validation still publish
`osw_data.zip`, with the issues in `validation_issues.json`; pass
`validate_output=False` to skip the gate entirely.

Regenerate them with:

```bash
python tests/dataset_builder.py
```

Single-purpose zips, hand-maintained and not schema-valid, used to exercise
individual passes with `validate_output=False`:

- `precision_and_null_tags.zip`
- `zero_length_edge.zip`
- `unsupported_files.zip`
- `nested_dataset.zip`

## Release Pipelines

GitHub Actions includes package publishing workflows:

- `.github/workflows/deploy_to_test.yml` publishes to TestPyPI from `develop`.
- `.github/workflows/publish_to_pypi.yml` publishes to PyPI from semver tags or manual dispatch.

Both workflows build the package from `pyproject.toml` and use
`PYPI_API_TOKEN` for authentication.
