Metadata-Version: 2.4
Name: openroland-survey-core
Version: 0.1.1
Summary: MIT canonical survey model, persistence, import, and validation core.
Author-email: Nicu Tofan <nicu.tofan@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/OpenRoLand/survey-core
Project-URL: Repository, https://github.com/OpenRoLand/survey-core
Project-URL: Issues, https://github.com/OpenRoLand/survey-core/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: attrs==26.1.0
Requires-Dist: httpx==0.28.1
Requires-Dist: numpy==2.4.6
Requires-Dist: openpyxl==3.1.5
Requires-Dist: platformdirs==4.11.1
Requires-Dist: pyproj==3.7.2
Requires-Dist: rasterio==1.5.1
Requires-Dist: sqlalchemy==2.0.51
Requires-Dist: openroland-crs==0.1.5
Provides-Extra: cli
Requires-Dist: click==8.4.2; extra == "cli"
Provides-Extra: cad
Requires-Dist: ezdxf==1.4.4; extra == "cad"
Provides-Extra: dev
Requires-Dist: black==26.5.1; extra == "dev"
Requires-Dist: build==1.5.0; extra == "dev"
Requires-Dist: flake8==7.0.0; extra == "dev"
Requires-Dist: httpx==0.28.1; extra == "dev"
Requires-Dist: isort==8.0.1; extra == "dev"
Requires-Dist: mypy==2.3.0; extra == "dev"
Requires-Dist: pre-commit==4.6.1; extra == "dev"
Requires-Dist: pre-commit-hooks==6.0.0; extra == "dev"
Requires-Dist: pyproject-flake8==7.0.0; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: pytest-mock==3.15.1; extra == "dev"
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: twine==7.0.0; extra == "dev"
Requires-Dist: wheel==0.47.0; extra == "dev"
Dynamic: license-file

# openroland-survey-core

The MIT `openroland-survey-core` distribution contains the canonical survey
model, database schema, import/normalization, and validation APIs shared by
the Stonex Cube, RW5, and JXL extraction libraries.

The AGPL `siscadro-survey-server` distribution in `server/` contains FastAPI,
web orchestration, authentication, and frontend integration and depends on
this core package.

## Status

This package currently provides the base foundation layer only:

- The immutable `SurveyPointRecord` canonical record and its supporting
  types (`SourceMetadata`, `ParseIssue`, `ExtractionResult`, `ImportSummary`,
  `XlsxSummary`).
- A deterministic three-decimal millimetre coordinate identity
  (`openroland_survey.coordinates`).
- The SQLAlchemy target schema and database engine/session helpers
  (`openroland_survey.models`, `openroland_survey.database`).
- The single-source-file import orchestration service, including
  Stereo70/`openroland-crs` and `pyproj`-based CRS transformation to WGS84
  (`openroland_survey.services`).
- The common XLSX writer (`openroland_survey.xlsx`).
- Optional DXF writer (`openroland_survey.dxf`; install the `[cad]` extra).
- The `SurveyPointExtractor` protocol and entry-point discovery
  (`openroland_survey.extractors`), including `process_file_to_all` (import
  + XLSX + DXF for one source file).

The Cube, RW5, and JXL format adapters, the unified CLI (`[cli]` extra), the
FastAPI/Material UI web application (`[web]` extra), and the `survey-points`
deployment project are implemented in later development phases. Importing
`openroland_survey.cli`, `openroland_survey.web`, or `openroland_survey.jobs` today
raises an actionable error explaining that the required extra is missing or
that the feature is not implemented yet.

## Canonical model

`openroland_survey.records.SurveyPointRecord` is the one in-memory shape every
format adapter (Cube, RW5, JXL) produces. `north`, `east`, and `height` are
required `Decimal` values in metres; every other field is optional and
`None` when a source format does not supply it. Unmapped source-specific
values are preserved in `source_values` instead of being dropped.

## Coordinate identity

Stored identity is exact: two points share a key when their `north`,
`east`, and `height` values are equal after rounding to three decimal
places (one millimetre) using half-up rounding.
`openroland_survey.coordinates.coordinate_key()` implements this once, using
`Decimal` arithmetic built from the value's string representation so
binary floating-point noise never changes the identity. Non-finite values
(missing, NaN, infinity) are rejected with `NonFiniteCoordinateError`.

Import matching is wider: when linking an observation to an existing row,
a neighbor within ±1 mm on every axis (`MATCH_TOLERANCE_MM`) is reused.
Untimed library imports (`kind=imported` / `added`) may also link to a
timed measured neighbor with the same plan position and height within
`LIBRARY_HEIGHT_TOLERANCE_MM` (20 mm). Measured observations may take
over a library row's stored identity when they arrive second. Use
`openroland-survey-core merge-near-duplicates --database PATH` (dry run by
default) to collapse library clones already present in a database.

## Database schema

`openroland_survey.models` defines one SQLAlchemy `Base` with schema version
**2**. New databases are GeoPackages:

- `source_files`: one row per imported source file version, unique on
  `(path, sha256)` so a byte-identical reimport is idempotent while a
  modified file at the same path is tracked as a new version.
- `source_file_issues`: one row per parse/import diagnostic recorded
  for a source file version (`severity`, optional `record_id`,
  `message`). Created additively on existing schema version 2 databases
  by `ensure_schema`.
- `survey_points`: one row per unique millimetre coordinate, unique on
  `(north_mm, east_mm, height_mm)`. The same table is the GIS feature
  layer: column `geom` holds a GeoPackageBinary `POINT` in **EPSG:3844**
  with axis order `XY = (east, north)`. Scalar `latitude` /
  `longitude` remain ordinary columns for the Leaflet web map.
- `survey_point_sources`: the many-to-many association between points and
  source files, recorded as a full ORM class (not a bare table) so it can
  carry `occurrence_count` and the normalized source observation JSON.
- `schema_metadata`: a small key/value table recording the schema version.
- `gpkg_spatial_ref_sys`, `gpkg_contents`, `gpkg_geometry_columns`:
  standard GeoPackage catalogs registering `survey_points` as an
  EPSG:3844 features layer.

`openroland_survey.database` creates engines with SQLite WAL mode and foreign
keys enabled, creates missing tables with `Base.metadata.create_all()`
(never dropping existing tables), registers GeoPackage metadata, and
rejects an existing target whose tables are missing expected columns or
whose schema version does not match. Schema version 1 databases must be
copied with:

```text
openroland-survey-core migrate-database --source OLD.sqlite3 --destination NEW.gpkg
```

Migration builds `geom` from plan coordinates and fills missing WGS84
columns with the `openroland-crs` package (`openroland-crs.stereo70_to_etrs89`).
``openroland-crs`` is declared as a normal PyPI dependency. Editable sibling
installs remain a local development option but are not required by package
metadata or public builds.

Open the resulting `.gpkg` in QGIS and add the `survey_points` layer
(EPSG:3844). The web UI continues to plot WGS84 `latitude`/`longitude`.

## Import merge rules

`openroland_survey.services.import_source()` imports one `ExtractionResult`
in a single database transaction:

- A point's stored coordinate identity is never changed once created.
- An incoming observation links to an existing point when each of
  `north_mm`, `east_mm`, and `height_mm` differs by at most 1 mm; if
  several neighbors match, the closest by L1 millimetre distance wins
  (then lowest `id`). Otherwise a new row is inserted with the incoming
  exact key.
- Within one source file, equal-coordinate records pick a **primary**
  observation by quality: timed points beat untimed ones; non-imported
  kinds beat `imported` (and legacy `method="Imported"`); kind rank is
  `control`/`gps`/`base` > `auto` > `unknown` > `added`/`imported`.
- A `None` canonical field is filled from an incoming non-`None` value.
- Collection `kind` may be upgraded when a higher-ranked observation
  arrives. Legacy `method="Imported"` may be cleared when a better
  observation arrives; new imports leave `method` empty for imported
  Type.
- An existing non-`None` canonical field (other than the kind/Imported
  method rules above) is never overwritten with a conflicting value.
- Every incoming observation is preserved in the association's
  `observations_json`, including values that lost the conflict above.
- Records with equal coordinates within the same source file increment
  `occurrence_count` on one association row instead of creating duplicates.

Cube `.pd` extraction sets `kind` from `Point.PointType`, GPS linkage,
and `GPSCoordinate.RecordMode` (`PointType` `2` added, `3` imported;
`RecordMode` `0`/`2` gps, `1` control, `3` auto; other GPS-less rows are
`kind=imported` with empty `method`). Schema version `3` adds the
nullable `survey_points.kind` column; `ensure_schema` upgrades version
`2` GeoPackages additively and backfills from stored observations.

Latitude/longitude are taken directly from the record when present;
otherwise they are transformed from `(east, north)` using `pyproj` with
`always_xy=True` and the supplied source CRS. A record that has neither
geographic coordinates nor a usable source CRS raises
`UnmappableCoordinatesError`, so a source is never imported with points
silently missing from the map.

## GPS height sanity check (optional)

When enabled (Convert → **Sanity-check GPS heights against DEM**, or
``--height-check`` on CLI; on by default), EPSG:3844 imports compare each
primary observation's MN75 height against Copernicus DEM GLO-30 near the
point (``kind=base`` records are skipped). Stereo70 plan coordinates and
normal heights are converted through
**openroland-crs** (ROMGEO grid) to ellipsoidal height, then to EGM2008 for
comparison with the DEM.

Behaviour:

- **On by default for each import** that writes to the database; turn it
  off per job on the Convert page or with ``--no-height-check``.
- **Before import**, required Copernicus tiles and the EGM2008 geoid grid
  are downloaded into the user cache (``platformdirs``:
  ``openroland-survey-core/height_check``). If any required tile or grid cannot
  be retrieved, the import aborts with no points written.
- **After tiles are available**, points import normally. Per-point
  PASS/WARN/FAIL/INCONCLUSIVE results are recorded as source-file issues;
  height mismatches do not block import.
- Non-EPSG:3844 imports skip the check and record one warning issue.

This is a coarse sanity check against a 30 m surface model, not survey
certification.

## Frontend (Material UI + Leaflet)

`frontend/` contains the React, Material UI, and Leaflet single-page
application served by `openroland_survey.web` (see above). It is a
standalone Vite/TypeScript project; building or testing it does not
require the Python package to be installed, and installing the base,
`[cli]`, or `[web]` Python extras never runs `pnpm`.

The application has four pages, reachable from the app bar navigation:

- **Convert** exposes the CLI conversion workflows
  (`file-to-xlsx`, `file-to-database`, `file-to-all`, `directory-to-xlsx`,
  `directory-to-database`) behind source-kind/target-kind segmented
  controls that show only the fields relevant to the current combination
  (format override, output path/directory, recursive, source CRS,
  include-KeyedIn, overwrite, fail-fast). Submitting a job polls
  `GET /api/jobs/{id}` about once a second, showing counters, the current
  source path, per-file errors, and parse-issue counts until the job
  reaches a terminal status; a finished single-XLSX job gets a direct
  download button, and a finished database or **To DB + XLSX + DXF** job
  refreshes the app bar's point/source counts. The all-target mode writes
  `{stem}.xlsx` and `{stem}.dxf` beside the source (requires `[cad]`).
- **Sources** is a filtered, keyset-paginated table of imported source
  file versions (`GET /api/sources`) with a detail drawer showing full
  metadata, persisted parse issues, and that source's paged points.
  All parse issues across the database are also available via
  `GET /api/issues` and the CLI `list-issues` command.
- **Points** is a filtered, keyset-paginated table of canonical points
  (`GET /api/points`) plus an "Around a point" panel that accepts either
  an existing point ID or explicit north/east coordinates, a radius, and
  an optional height tolerance (`POST /api/points/near`), showing
  ascending-distance results. Selecting any point opens a detail drawer
  with every contributing source and its observation values.
- **Map** renders every point with WGS84 coordinates on OpenStreetMap
  tiles (`GET /api/map/points`), fed page by page into a chunk-loading
  Leaflet marker cluster. A bottom-right layers control can switch the
  base map to QGIS WMS layers discovered via same-origin
  ``GET /api/qgis/layers`` (tiles via ``GET /api/qgis/wms`` when the
  backend proxies MapProxy). Configure
  ``SISCADRO_QGIS_INTERNAL_URL`` and ``SISCADRO_MAPPROXY_INTERNAL_URL``
  on the server; the browser never needs those hosts.
  It shows load progress with cancel/reload
  controls, fits the view to the first loaded batch while preserving
  later pan/zoom, colors markers by fix/quality status, and opens the
  same point detail drawer when a marker is selected.

The shell adds accessible labels, keyboard-operable controls, a shared
error boundary, and Material UI snackbars for API outcomes.

### Frontend development

```bash
cd frontend
pnpm install
pnpm run dev
```

`pnpm run dev` proxies `/api/...` requests to `http://127.0.0.1:8000`, so
run `python -m openroland_survey.web --database ...` (or
`openroland-survey-core serve --database ...` with the `[cli]` extra) alongside
it. Only OpenStreetMap tile requests use the network; every other asset
is bundled locally.

### Frontend checks and packaging

```bash
pnpm run lint
pnpm run test
pnpm run build
```

`pnpm run build` writes to `frontend/dist/`. Copy that output into
`openroland_survey/static/` before building the Python package so the
FastAPI application's `GET /` and SPA-fallback routes have something to
serve and so `python -m build` packages the compiled assets into the
wheel (see `MANIFEST.in`'s `recursive-include openroland_survey/static *`):

```bash
make frontend-static
```

Or rebuild and start the local UI in one step (defaults:
`D:/prog/__py_libs__/openroland-survey-core/playground/production.gpkg`
on `127.0.0.1:8765`):

```bash
make serve
make serve DATABASE=path/to.gpkg PORT=8765
```

If `HOST:PORT` is already in use, `make serve` asks whether to kill the
holding process, switch to another free port, or quit. Non-interactive
helpers: `make serve SERVE_FLAGS=--kill` or
`make serve SERVE_FLAGS=--alt-port`.

Manual equivalent:

```bash
pnpm --dir frontend run build
python scripts/sync_frontend_static.py
```

Vitest/React Testing Library tests cover API error-message formatting,
Convert-page field visibility and payload building per source/target
mode, job-polling termination at a terminal status, keyset paged-table
row accumulation, "Around a point" form validation, and the map feed's
page accumulation and stale-request cancellation.

## Docker (survey app)

See ``docker/README.md``. Local stack:

```powershell
.\venv\Scripts\Activate.ps1
cd docker
copy docker-compose.local.yml.example docker-compose.local.yml
cd ..
make d-local
```

``make d-local`` builds the frontend and private/survey wheels on the host,
then starts Compose (app installs from ``docker/wheels/``).

- App: http://localhost:8000

Optional QGIS WMS base maps come from the separate
[``qgis-provider``](https://github.com/) project (``CadPlatf/qgis-provider``).
Start the provider lab first, then the survey app:

```powershell
cd D:\prog\CadPlatf\qgis-provider
make serve

cd D:\prog\__py_libs__\openroland-survey-core
copy .env.local.example .env.local   # optional
make serve
```

``make serve`` optionally loads repo-root ``.env.local`` (then ``.env``).
Missing file or vars is fine (OpenStreetMap only). Shell env still wins.
Example host defaults: MapProxy ``:9081``, config API ``:9500``.

``make d-local`` reuses the same ``.env.local`` when present
(``env_file``, optional). Compose overrides the URL values with Docker
DNS (``http://mapproxy/...``, ``http://qgis:5000``) so host ``localhost``
entries do not leak into the container. Shared flags such as
``SISCADRO_QGIS_PROXY_ENABLED`` come from the file when set.

The browser uses ``/api/qgis/layers`` and ``/api/qgis/wms`` only.
See ``docker/README.md``, ``deploy/production/README.md``, and
``docker-compose.local.yml.example``.

## Installation

```bash
python -m pip install -e .
```

Base installation depends only on `attrs`, `openpyxl`, `pyproj`, and
`sqlalchemy`. It does not install Click, FastAPI, Pydantic, or Uvicorn.

Install the optional extras once their features are implemented:

```bash
python -m pip install -e .[cli]
python -m pip install -e .[web]
python -m pip install -e .[cad]
python -m pip install -e .[cli,web,cad]
```

The `[cad]` extra installs `ezdxf` and enables `file-to-all` / Convert
**To DB + XLSX + DXF** (and `openroland_survey.dxf` in general).

## Development

Start by creating a virtual environment and installing the development
dependencies:

```bash
python -m venv venv
venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -e .[dev,cli,web,cad]
```

Or, if you have `make` available:

```bash
make init-d
```

Run the standard checks before committing:

```bash
make delint
make lint
make typecheck
make test
```
