Metadata-Version: 2.5
Name: aind-metadata-reader
Version: 0.3.0
Summary: Read historical AIND metadata without rewriting source records
Project-URL: Repository, https://github.com/AllenNeuralDynamics/aind-metadata-reader/
Project-URL: Issues, https://github.com/AllenNeuralDynamics/aind-metadata-reader/issues
Project-URL: Changelog, https://github.com/AllenNeuralDynamics/aind-metadata-reader/blob/main/CHANGELOG.md
Author: Allen Institute for Neural Dynamics
Author-email: Galen Lynch <galen@galenlynch.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Provides-Extra: validation
Requires-Dist: jsonschema>=4.26.0; extra == 'validation'
Description-Content-Type: text/markdown

# AIND Metadata Reader

Read historical AIND metadata, inspect incomplete records, and record explicit
corrections while preserving the source. Readers support selected representations
from `aind-data-schema` and `biodata-schema` without requiring either package at
runtime. Python 3.10 and newer are supported.

This library is **experimental**. Its API may evolve before version 1.0.
The examples below run from this checkout using the included historical fixtures.

## What the library does

Metadata changes in several ways: fields move, vocabulary grows, and concepts
change meaning. A processing input recorded as a file path conveys different
information from an input recorded as an asset reference. The reader exposes
those distinctions through small views of code, references, and coordinate frames.

Reading preserves the original values and unknown fields. Schema-invalid metadata
remains inspectable, and a malformed component does not hide usable siblings.
You can separately request a validation report or derive a corrected interpretation.
Neither operation silently replaces the source.

```mermaid
flowchart LR
    source["JSON file or mapping"] --> capture["Preserved snapshot"]
    capture --> views["Views, available values, and diagnostics"]
    capture --> validation["Optional validation report"]
    views --> changes["Explicit correction or inference"]
    changes --> derived["Derived snapshot and change manifest"]
```

You choose the source. The library accepts local files and caller-supplied data;
it does not query DocDB, reconcile stores, or decide which record is authoritative.
Image loading, registration, and transform application stay with their existing
libraries.

## Get started with a historical record

Prepare the checkout environment:

```sh
uv sync
```

Read the included processing example from `aind-data-schema` v1.4.0:

```python
from aind_metadata_reader import ReadStatus, read_json, read_processing

record = read_json("tests/fixtures/history/v1.4.0-processing.json")
processing = read_processing(record)

print(record.schema_version)  # 1.1.4: the core schema version, not the package version
for process in processing.processes:
    print(process.process_type, process.code_version)
```

The example contains three pipeline steps and two separately recorded analyses.
Their views retain the original ordering within each group and their actual source
pointers. The reader does not construct a dependency graph from that ordering.

With your own data, replace the fixture path with a processing file or an aggregate
such as `metadata.nd.json`. `read_processing` selects the processing core from an
aggregate; `read_acquisition` does the same for acquisition metadata.

The main objects have different jobs:

| Object | What it holds |
| --- | --- |
| `Snapshot` | The complete preserved payload, source provenance, and original bytes when supplied. |
| `Document` | An object selected within a snapshot. Children retain the same root snapshot. |
| `Processing`, `Process`, `Acquisition`, `CoordinateFrame` | Views of recognized concepts in a historical representation. |
| `ReadValue` | A requested interpretation, its availability, diagnostics, and raw source field. |
| `Interpretation` | Original and derived views, applied changes, retained rule decisions, and a manifest. |

For data fetched by your own client, use `Document(mapping, provenance=Source(...))`.
A `Source` can retain an S3 locator or a DocDB host, API version, database,
collection, and record ID. Supply known conversion evidence explicitly.

Inspect recorded content directly with `record.raw` or
`record.field("/processing_pipeline/pipeline_version")`. `raw` and field values
return copies. A field's `state` distinguishes `present`, `null`, and `absent`.
Pointers passed to a document are relative to that document; returned field pointers
identify the location in the complete snapshot.

## Check what an interpretation can establish

Selecting a process by type can produce several matches. `find()` returns them
all; `select()` reports whether exactly one matches:

```python
selected = processing.select("Image tile fusing")
if selected.status == ReadStatus.AVAILABLE:
    fusion = selected.value
    print(fusion.document.pointer)  # /processing_pipeline/data_processes/0
    print(fusion.code.version.value)  # 0.1

    inputs = fusion.inputs
    if inputs.status == ReadStatus.AVAILABLE:
        for reference in inputs.value:
            print(reference.kind, reference.location.value)
else:
    print(selected.status, selected.diagnostics)

assert processing.select("Analysis").status == ReadStatus.AMBIGUOUS
```

Code version and pipeline version are separate concepts. `process.code` exposes
independent `version`, `url`, `parameters`, and `inputs` results. Input/output
references distinguish `file`, `asset`, `asset_group`, and `unknown`; asset groups
retain their members. An asset URL does not establish a file-level input.

A `ReadValue` gives you four things:

| Attribute | Meaning |
| --- | --- |
| `status` | Whether the requested interpretation is available, and why it may be unavailable. |
| `value` | The interpreted value when available; otherwise None. |
| `field` | The actual recorded field, including its raw value, pointer, and snapshot. |
| `diagnostics` | Structured explanations with codes and source locations. |

Availability describes the requested interpretation:

| Status | What it tells you |
| --- | --- |
| `available` | The reader established a value. |
| `absent` | The source field was not recorded. |
| `null` | The source explicitly recorded JSON null. |
| `invalid` | The recorded content does not fit this requested interpretation. |
| `unsupported` | The reader has no established interpretation for this representation or meaning. |
| `ambiguous` | Several candidates exist and none has been selected. |

`invalid` describes one attempted interpretation; it does not declare the entire
metadata record invalid.

Convenience accessors such as `code_version`, `parameters`, `processes`, and
`array_axes` make common cases shorter. Use the corresponding component results
when missing information matters. An empty `processes` tuple, for example, could
mean an empty recorded collection or an unsupported representation:
`process_entries.status` distinguishes those situations.

`selected.require()`, `processing.one(type)`, and `acquisition.one_frame()` explicitly
assert a requirement and can raise. Ordinary inspection uses status-bearing results.

## Incomplete metadata stays inspectable

This illustrative record has a numeric code version and a malformed second entry:

```python
from aind_metadata_reader import Document

incomplete = Document(
    {
        "schema_version": "2.0.0",
        "data_processes": [
            {
                "process_type": "An unfamiliar processing term",
                "name": "example step",
                "code": {"version": 42, "parameters": {"threshold": 12}},
                "output_path": "/results/image.zarr",
            },
            "unfinished entry",
        ],
    }
)
inspected = read_processing(incomplete)
first, second = inspected.process_entries.value
step = first.value

assert step.code.version.status == ReadStatus.INVALID
assert step.code.version.field.value == 42
assert step.code.parameters.value == {"threshold": 12}
assert step.output_location == "/results/image.zarr"
assert second.status == ReadStatus.INVALID
assert second.field.value == "unfinished entry"
assert inspected.raw == incomplete.raw
```

Unfamiliar vocabulary remains recorded text. An untested schema version produces
an `unverified_version` diagnostic while recognized components remain readable.
Schema identity, recorded layout, and known historical boundaries select the
reader; each component then reports its own availability. An unfamiliar direction,
for example, withholds anatomical orientation while preserving the frame, its raw
axes, and other readable frames. This describes what the reader can interpret;
validation separately describes whether the record satisfies a schema.

Raw capture requires a finite JSON object. Malformed JSON, duplicate object keys,
and nonfinite numbers are parsing errors; schema-invalid content within a parsed
record is retained.

## Inspect coordinate frames separately from array dimensions

Acquisition views keep local and global coordinate systems separate. Select a
scope or a source pointer when several frames exist:

```python
from aind_metadata_reader import read_acquisition

acquisition = read_acquisition(read_json("tests/fixtures/v2.9.0-exaspim_acquisition.json"))
selected_frame = acquisition.select_frame(scope="local")
if selected_frame.status == ReadStatus.AVAILABLE:
    frame = selected_frame.value
    print(frame.name)
    print([(axis.name, axis.position, axis.dimension) for axis in frame.axes])
    print(frame.orientation.status, frame.orientation.value)
```

`Axis.position` follows source coordinate order. `Axis.dimension` is an image-array
dimension only where recorded. `array_mapping` reports whether a mapping is
established; mapping it to a particular Zarr array still requires evidence about
that array's dimensions. A frame can have a known anatomical orientation while
its array mapping remains unavailable.

Configured metadata supports stream and device selection. Legacy imaging metadata
exposes scope `legacy` and reports unavailable stream/device selectors. Recorded
image transforms remain accessible through `frame.configuration`; the reader does
not compose them or substitute a local frame for a global one.

## Request validation as a separate report

Install the optional validator in the checkout with `uv sync --extra validation`.
Supply the historical schema explicitly:

```python
from aind_metadata_reader import validate

schema = read_json("tests/fixtures/history/v1.4.0-processing-schema.json")
report = validate(
    record,
    schema,
    dialect="https://json-schema.org/draft/2020-12/schema",
)
assert report.status == "valid"
assert processing.validation.status == "not_checked"
```

`validate()` returns a separate report and leaves the record and its views unchanged.
Reports distinguish `not_checked`, `valid`, `invalid`, and `unavailable`. An unusable
schema, missing validator, or unresolved external reference means unavailable
validation. Each report identifies the input, schema, and chosen dialect. A dialect
must be supplied when the schema does not declare one.

This checks JSON Schema assertions using `jsonschema`. It does not insert defaults,
coerce values, fetch remote schemas, run custom upstream Pydantic validators, or
check format annotations. A successful report establishes conformance to those
assertions; scientific accuracy and source authority remain separate questions.
Reading has no mandatory validator or upstream-schema dependency.

## Derive an attributed correction

A correction produces a new interpretation. The following example assumes a
calibration note establishes a revised spacing:

```python
from aind_metadata_reader import Change, apply_changes

calibration_record = Document({"voxel_spacing_um": 2.0})
change = Change.propose(
    calibration_record,
    "/voxel_spacing_um",
    1.8,
    operation_id="spacing-calibration",
    rule_id="manual.voxel-spacing",
    rule_revision="1",
    implementation="calibration-note/1",
    kind="correction",
    author="experimenter",
    reason="Updated calibration measurement",
    evidence=("calibration-record-123",),
    scope="physical spacing in micrometers",
)
result = apply_changes(calibration_record, [change], applied_at="2026-11-10T12:00:00+00:00")
assert result.original.raw == {"voxel_spacing_um": 2.0}
assert result.derived.raw == {"voxel_spacing_um": 1.8}
```

The attribution arguments become the audit trail. Operation kinds distinguish
`correction`, `inference` (requires assumptions), and `processing_frame`
(the conventions of a historical computation). Physical corrections and historical
processing frames retain distinct identities even if their resulting values match.

Application checks the exact input capture, original field value, selected document
scope, and conflicting edits. It does not require unrelated metadata to validate.
Original snapshots stay intact, and no-op operations remain recorded.

Domain rules propose changes through `RuleEvaluation`, whose outcome is
`applicable`, `not_applicable`, or `unknown`. For example,
`smartspim.evaluate_pyramid_translation_correction` checks pyramid geometry and
caller-supplied association and block-averaging evidence. Reading and evaluation
never apply its proposals. Evidenced defect signatures can qualify even when the
producer version is unverified; no first-fixed fusion version is assumed.

Pass decisions to `apply_changes(..., evaluations=(evaluation,))` to retain their
inputs, reader provenance, and links to applied operations. Unknown decisions and
proposals deliberately left unapplied can also be retained. Archive supporting
snapshots along with the target snapshot. The [API guide](docs/api.md) describes
the rule and saved-record contracts in more detail.

## Save the inputs and replay recorded changes

Save the source archive and interpretation manifest with your analysis:

```python
import json
from pathlib import Path
from aind_metadata_reader import Snapshot, replay

output = Path("analysis-provenance")
output.mkdir(exist_ok=True)
(output / "source.json").write_text(json.dumps(calibration_record.snapshot.archive(), indent=2))
(output / "interpretation.json").write_text(json.dumps(result.manifest(), indent=2))

restored = Document.from_snapshot(Snapshot.from_archive(read_json(output / "source.json").raw))
again = replay(restored, read_json(output / "interpretation.json").raw)
assert again.derived.snapshot.snapshot_id == result.derived.snapshot.snapshot_id
```

Replay applies recorded replacements to matching archived inputs. It does not rerun
the latest reader or rule. Reevaluation is a separate computation using a selected
implementation and its evidence.

`content_digest` identifies parsed values; `original_digest` identifies original
bytes when retained. `snapshot_id` also incorporates source provenance, including
retrieval evidence. Derived snapshots identify their parent snapshot and
interpretation. Restore archives for exact replay: rereading a live file creates
a new capture even if its bytes are unchanged. These hashes detect mismatches;
they do not authenticate evidence.

Archives and manifests have independent format versions. New records use `/2`,
and explicit readers retain `/1` support through fixed historical fixtures.
Persisted-record loaders and replay check their contracts strictly; the records
can always be inspected separately as JSON.

## Layout compatibility and test coverage

The reader selects a known layout from the recorded schema namespace and core
type, structural markers, and historical layout boundaries. AIND acquisition v1
uses root axes with array dimensions; configured acquisition uses coordinate
frames. Legacy processing stores processes under `processing_pipeline` with
`name` and flat code fields; modern processing uses `process_type` and nested
`code`. Conflicting layout markers or declarations withhold the ambiguous
interpretation and retain the complete raw record.

The versions below identify test coverage, independently of Python package,
pipeline, and component-code versions. `Format.versions` records these exercised
core versions. An unlisted version can use the same evidenced layout, with an
`unverified_version` diagnostic tied to its original version field. This does
not certify compatibility for fields the reader does not interpret.

| Representation | Core versions exercised | Concepts exposed |
| --- | --- | --- |
| AIND legacy acquisition | 1.0.0, 1.0.3, 1.0.4 | Recorded axes and image dimensions |
| AIND configured acquisition | 2.0.0, 2.5.1, 2.5.3, 2.5.4 | Frames and configuration references |
| biodata acquisition | 3.0.0 | Frames and configuration references |
| AIND legacy processing | 1.1.0, 1.1.4 | Processes, analyses, and file inputs |
| AIND flat processing | 2.0.0, 2.2.5, 2.3.1, 2.3.2, 2.3.3 | Processes, pipeline links, and asset/group inputs |
| biodata processing | 3.0.0 | Processes, pipeline links, and asset/group inputs |

When a record omits its namespace, historical major-version conventions can
identify AIND v1/v2 or biodata v3. Otherwise the layout must identify one reader
uniquely, or the caller must supply a `format_hint`. For example, an unversioned
configured record with a recognized namespace needs no hint; the same layout
without a namespace can match both families and needs an explicit choice.
Hints cannot override a contradictory namespace, core type, or known layout
boundary. Missing or malformed versions remain recorded and produce diagnostics;
they do not prevent reading components when identity and layout are established.

Reader selection records retain the original declared version and the evidence
used to choose a layout, including any untested-version qualification. Reading
does not convert metadata or replace its declared schema version. The library's
own snapshot, interpretation, and replay formats retain strict version checks;
their saved-contract tests remain independent of current reader dispatch.

Tests include tagged historical examples/schemas, public production metadata, and
paired v1/v2 DocDB representations. Constructed cases exercise documented historical
changes such as output cardinality. V3 support is provisional against upstream
examples until production outputs are available. See [fixture provenance](tests/fixtures/README.md)
and [historical evolution evidence](tests/fixtures/history/README.md).

Parsed integers retain Python precision; fractional values use binary64. Original
bytes preserve source number spellings. Content hashing uses the documented Python
JSON convention, not RFC 8785 canonicalization.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and checks, the
[design proposal](planning/design.md) for rationale, and
[implementation status](planning/implementation-status.md) for remaining work.
The project was rendered from `aind-python-copier-template` commit
`5dc04dcba3b3c16b8ee9c4b7c1c2314c08567bfc`. Repository setup and CI are configured. Tagged releases publish source and wheel
artifacts through the release workflow. Licensed under [MIT](LICENSE).


## SmartSPIM Registration Inputs

`aind_metadata_reader.smartspim.registration_input(process)` returns an inspectable
`ReadValue` of file references for an `Image importing` process. It preserves
legacy file inputs and recognizes the observed
`aind-smartspim-ccf-registration` 0.0.37 convention that stores the input under
`output_parameters.input_location`. Each reference retains its source pointer.
Other producer versions need independent evidence before this convention applies;
an asset-level input is not converted into a file path. This helper reads
provenance and does not apply spatial corrections.


## Pyramid Translation Corrections

`aind_metadata_reader.smartspim.evaluate_pyramid_translation_correction` diagnoses the observed
missing-translation and voxel-count translation signatures in NGFF 0.4 fused
pyramids. Callers must provide evidence linking the artifact to the selected
fusion process and establishing block averaging. These requirements also apply
when the producer version is missing or unverified; no first-fixed version is
assumed. Correct physical translations yield `not_applicable`, while unrelated
geometry yields `unknown` with no proposed changes.

Evaluation proposes changes without applying them. Pass its changes and evaluation
to `apply_changes` to retain the source snapshot and an auditable interpretation.
Previously recorded rule identities remain replayable without rerunning evaluation.
Historical registration frames remain outside this physical correction's scope.
