Metadata-Version: 2.4
Name: mini-atlas-simulator
Version: 0.2.0
Summary: UTC simulation clock, result catalog, writer, and ResultReader for Mini Atlas.
License-Expression: Apache-2.0
Keywords: simulation,catalog,utc,parquet
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyarrow
Requires-Dist: duckdb
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# mini-atlas-simulator

This is the Mini Atlas simulator library (`mini_atlas_simulator`). It is the
**only installable simulator package**. The user’s Python pipeline owns
analytics, formulas, rule execution, and emitted results. The library owns the
UTC clock, catalog identity, metric/table declarations, atomic publication, and
a read-only `ResultReader`.

M1d adds installable-wheel examples: a user pipeline can publish a live run and
a second process can inspect it through `ResultReader` with no API, frontend,
Docker, Node, or repository checkout required at runtime.

Cadences are fixed elapsed UTC durations. `1d` / `P1D` is 24 hours, not a local
calendar day across daylight-saving changes.

## Install (local development)

```text
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
python -m pip install -e ".[dev]"
```

## Install (local wheel)

The package is not published to PyPI in this milestone. Build and install a
local wheel:

```text
python -m pip wheel . -w dist --no-deps
python -m venv /tmp/sim-venv
source /tmp/sim-venv/bin/activate
python -m pip install dist/mini_atlas_simulator-*.whl
```

Runtime dependencies are the storage stack (PyArrow for Parquet publication,
DuckDB for table pages). Installing `mini-atlas-simulator` does **not** pull
Dash, Flask, Flask-RESTX, FastAPI, Waitress, NiceGUI, React, Node, Docker, or
the graph packages.

Flask/API, React, Node, and Docker are not required to log or read results.
Dash is not required to log or read results. A later Dash application will
open the same store directly through `ResultReader`.

## Storage

- **SQLite** (`catalog.sqlite`) stores catalog/scalar metadata and metric
  points: projects, runs, declarations, committed steps, and coverage scalars.
- **Parquet** stores logged tabular results for a step.
- **`ResultReader`** provides read-only access to both through the registered
  catalog. Callers do not query SQLite or Parquet themselves.

## Clock

```python
from mini_atlas_simulator import SimulationClock

clock = SimulationClock(
    start="2025-01-01T00:00:00Z",
    stop="2025-01-03T00:00:00Z",
    cadence="1d",
    lookback="1d",
)

for bounds in clock:
    # bounds.step_n starts at 1
    # bounds.previous_as_of = start + (n-1)*cadence
    # bounds.as_of = start + n*cadence  (final as_of == stop)
    # bounds.window_start = as_of - lookback
    ...
```

Start and stop must be timezone-aware UTC (`Z` or `+00:00`). Naive timestamps
and non-UTC offsets are rejected. `(stop - start)` must be a positive integer
multiple of cadence.

## Catalog

```python
from mini_atlas_simulator import open_catalog

with open_catalog("./results") as catalog:
    assert catalog.result_format_version == 1
```

A fresh `{store}/catalog.sqlite` is created in WAL mode with
`result_format_version = 1`. Reopening a compatible catalog succeeds. An
unknown or unsupported format version fails closed.

## Writer

```python
from mini_atlas_simulator import create_project, start_run

create_project(store="./results", name="sample-project", description="Sample simulation")

with start_run(
    store="./results",
    project="sample-project",
    name="baseline-30d",
    schedule=clock,
    description="Baseline configuration for the weekly study",
) as run:
    run.declare_metric(
        "slice_node_count",
        label="Slice nodes",
        unit="count",
        grain="per_step",
        direction="none",
    )
    run.declare_table("subject_scores", role="scores", subject_column="subject_id")
    for bounds in clock:
        with run.step(bounds) as step:
            step.log_metrics({"definition_id": "slice_node_count", "value": 7})
            step.log_table("subject_scores", [{"subject_id": "A-99", "score": 0.91}])
```

Project and run creation do not allocate a store-wide `commit_id`. A step is
visible in the catalog only after the short publication transaction commits.
Metrics are declared by user Python; there are no built-in evaluation formulas.

Optional `start_run(..., description=...)` stores a run description in a side
table. Omit the argument to leave the description absent (`None`). An empty or
whitespace-only string is invalid and is not stored as missing. The established
`runs` table is unchanged. `ResultReader.get_run` and run-list rows return
`description` as a string or `None`.

`list_projects` includes `run_count`. `list_runs` includes `description`,
`committed_steps` (count of committed steps at the reader's `observed_cursor`),
and `scheduled_steps`. Those committed values are not silently clamped and are
not substituted from later state. Run-row summaries add declaration metadata
`declared`, `label`, `unit`, `grain`, and `display`. Missing, logged-null,
explicit zero, no-cohort, and undeclared remain distinct; the simulator does
not derive presentation strings.

`declare_metric(..., description="...")` is an optional free-text extra. The
library stores extra keys on the declaration and returns them; it does not
require or interpret `description`.

### Lineage declarations and selected-step observations

Lineage is **bounded graph metadata** for a run: a user-declared set of stages
and, for each committed step, at most one user-recorded observation per stage.
The library does not execute, schedule, reorder, infer, or time stages. It is
not a workflow engine, event log, per-subject trace, or trend series.

```python
with start_run(
    store="./results",
    project="sample-project",
    name="baseline",
    schedule=clock,
    description="Baseline configuration for the weekly study",
) as run:
    run.declare_lineage_stage(
        "ingest",
        label="Ingest inputs",
        description="Load step inputs",
        counts=["input_n"],
    )
    run.declare_lineage_stage(
        "prepare",
        label="Prepare cohort",
        parents=["ingest"],
        counts=["kept_n", "dropped_n"],
    )
    run.declare_lineage_stage(
        "evaluate",
        label="Evaluate subjects",
        parents=["prepare"],
        counts=["evaluated_n", "flagged_n"],
    )
    run.declare_lineage_stage("review", label="Optional review", parents=["evaluate"])
    run.declare_lineage_stage(
        "publish",
        label="Publish outputs",
        parents=["evaluate", "review"],
        counts=["output_n"],
    )

    for bounds in clock:
        kept_n = 0
        dropped_n = 0
        for group in (("s1", "s2", "s3"), ("s4",)):
            for subject in group:
                if subject == "s2":
                    dropped_n += 1
                else:
                    kept_n += 1
        with run.step(bounds) as step:
            step.log_lineage(
                [
                    {"stage_id": "ingest", "status": "completed", "counts": {"input_n": 8240}},
                    {
                        "stage_id": "prepare",
                        "status": "completed",
                        "counts": {"kept_n": kept_n, "dropped_n": dropped_n},
                    },
                    {
                        "stage_id": "evaluate",
                        "status": "completed",
                        "counts": {"evaluated_n": kept_n, "flagged_n": 0},
                    },
                    {"stage_id": "review", "status": "skipped"},
                    {"stage_id": "publish", "status": "completed"},
                ]
            )
```

A `stage_id` may be observed once in every different committed step; it names
one stable graph node for the run. Nested loops inside a single `run.step(...)`
must not call `log_lineage` per subject. User Python aggregates during those
loops and records each stage once before the step commits. Repeating a
`stage_id` on the same step is rejected rather than added, overwritten, or
accumulated. Lineage is not an event log and must never scale with evaluated
subjects.

Every logged observation requires `stage_id` and `status`. `status` is exactly
`completed`, `skipped`, or `failed` (`running` and `pending` are not valid).
`counts` is optional. Unknown keys, unknown stages, and unknown count names are
rejected. A missing observation is not skipped, failed, completed, or zero; a
missing declared count is not zero; explicit zero round-trips as zero.

Declarations use a short catalog transaction and are readable at zero commits
through `ResultReader.list_lineage_stages`. Observations publish in the same
transaction and `commit_id` as the rest of the committed step. Failed or
poisoned steps leave no visible observations. A frozen `through_cursor`
excludes later observations; there is no independent Lineage cursor.
`read_lineage_at_step` requires a committed `step_n` at that cursor (no nearest
step, no clamping, no series API).

Lineage lives in optional catalog tables only (no Lineage Parquet). Stores
written by 0.1.1 remain readable: a new reader returns an empty declaration
list and `description is None`. An old 0.1.1 reader ignores the new optional
tables and continues to read established run, metric, table, rule, evidence,
and decision data. `result_format_version` remains `1`.

### Rule declarations (Task 1)

User Python evaluates rules. The library stores **declarations** (Task 1),
**per-step aggregates / set-aggregates** (Task 2), and triggered-subject
evidence with receipts (Task 3). Each `logic_ref` is **claimed identity**
supplied by your pipeline; the library does not read source files, execute
rule logic, or derive aggregate counts from evidence. Only subjects that the
caller says were triggered are logged; evaluated populations belong in
user-computed aggregates, not evidence rows.

All `declare_rule` calls must finish **before the first committed step**.
Definitions are written in a short catalog transaction (no `commit_id`) and are
visible through `ResultReader.list_rule_definitions` even when the run has zero
committed steps.

```python
with start_run(store="./results", project="sample-project", name="baseline", schedule=clock) as run:
    run.declare_rule(
        "R12",
        label="Velocity",
        description="Velocity 14d",
        version="1",
        target={"kind": "entity", "type": "subject", "key": "subject_id"},
        parameters={"window": "P14D", "k": 11},
        logic_ref={"module": "pipeline.rules.velocity", "digest": "sha256:..."},
        category="evaluation",
        tags=["velocity", "monitoring"],
    )
```

`definition_fingerprint` is SHA-256 of canonical JSON over `rule_id`,
`version`, `target.kind`, `target.type`, `target.key`, `parameters`, and
`logic_ref` (not label, description, category, tags, or run-level `code_ref`).
Optional Rule Foundation catalog tables are additive at
`result_format_version = 1`.

### Rule aggregates (Task 2)

All aggregate and set-aggregate **values are calculated in user Python**. The
library stores what you log and does not reconcile counts, compute set overlap,
or inspect the evaluated population.

Call `step.log_rule_aggregates(...)` and/or `step.log_rule_set_aggregates(...)`
only inside an active `run.step(...)` context. Rows become visible **only when
that step commits** (same short publication transaction as metrics and tables).
Missing is not zero: a declared rule with no aggregate row for a step is **Not
logged**, not numeric zero.

```python
with run.step(clock.bounds_for(1)) as step:
    step.log_rule_aggregates(
        {
            "rule_id": "R12",
            "evaluated_n": 100,
            "unique_triggered_n": 4,
            "first_triggered_n": 3,
            "repeat_triggered_n": 2,
            "extras": {"note": "user-computed"},
        }
    )
    step.log_rule_set_aggregates(
        {
            "op": "intersection",
            "rule_ids": ["R12", "R13"],
            "unique_n": 2,
        }
    )
```

Set operations require every `rule_id` to be declared on the run and to share
the same `(target.kind, target.type, target.key)`. `intersection` and `union`
require at least two unique rule ids (stored in sorted order). `difference`
requires exactly two rule ids in operand order `(A, B)` meaning A minus B.

Read aggregates with `ResultReader.read_rule_aggregates_at_step`,
`read_rule_aggregate_series`, and `read_rule_set_aggregates_at_step`. All of
these require `through_cursor` and a committed `step_n` (or step range) at that
cursor. Selected-step reads keep the full `extras` object. Trend series
`fields` are the standardized scalar core measures only (`evaluated_n`,
`trigger_event_n`, `unique_triggered_n`, `first_triggered_n`,
`repeat_triggered_n`, `cumulative_distinct_n`); `extras` is not a series field.

`read_rule_aggregate_series` is cadence-neutral and bounded:

* `MAX_RULE_AGGREGATE_SERIES_STEPS` (1000) caps `to_n - from_n + 1`, which
  covers a 600-step run without assuming a 365-day calendar;
* `MAX_RULE_AGGREGATE_SERIES_COUNT` (8) caps the number of returned series;
* a request is either one rule with multiple measures or one measure with
  multiple rules, never a rules × measures matrix.

### Triggered-subject evidence (Task 3)

Rule logic stays in user Python. A hit is not a decision and is not evaluation
truth. The writer records only triggered subjects, never the evaluated
population:

```python
with run.step(clock.bounds_for(1)) as step:
    step.log_rule_aggregates({
        "rule_id": "R12",
        "evaluated_n": 1_000_000,
        "unique_triggered_n": 1,
    })
    step.log_rule_hits(
        rule_id="R12",
        rows=[{
            "target_id": "A-19",
            "evidence": {"score": 0.91, "reason": "velocity"},
        }],
    )

with run.step(clock.bounds_for(2)) as step:
    step.log_rule_hits(rule_id="R12", rows=[])

with ResultReader(store="./results") as reader:
    cursor = reader.latest_cursor()
    status = reader.read_rule_evidence_status(
        run_id, through_cursor=cursor, step_n=1, rule_id="R12"
    )
    page = reader.read_table(
        run_id,
        "rule_hits",
        through_cursor=cursor,
        step_n=1,
        limit=100,
        offset=0,
        filter=[("rule_id", "eq", "R12")],
    )
```

`status` distinguishes `not_recorded`, `logged_empty`, and `recorded`.
Evidence is visible only when the step commits. Evidence pages are bounded:
the default is 100 rows and the maximum is 1,000; callers must supply exactly
one `rule_id` equality predicate. Evidence objects are stored as opaque,
canonical UTF-8 JSON text. When callers provide mappings, Python has already
resolved any duplicate object keys, so duplicate keys that are no longer
observable cannot be detected. `filtered_empty` and `page_past_end` describe
only a page query for stored evidence; they do not replace stored status.

### First-class decisions (Task 4)

Decisions are user-supplied records, not rule-engine output. Declare one
decision stream before the first committed step. Its `target` namespace uses
the same `kind`, `type`, and normalized, case-preserving `key` rules as rule
targets. A run may use an entity or action namespace, but it must not mix
namespaces in one stream.

```python
with start_run(store="./results", project="sample-project", name="baseline", schedule=clock) as run:
    run.declare_rule(
        "R12",
        label="Velocity",
        description="Velocity 14d",
        version="1",
        target={"kind": "entity", "type": "subject", "key": "subject_id"},
        parameters={"window": "P14D"},
        logic_ref={"module": "pipeline.rules.velocity"},
    )
    run.declare_table(
        "decisions",
        role="decisions",
        target={"kind": "entity", "type": "subject", "key": "subject_id"},
    )
    with run.step(clock.bounds_for(1)) as step:
        step.log_decisions({
            "decision_id": "D-2025-01-02-A-19",
            "target_id": "A-19",
            "contributing_rules": [{
                "rule_id": "R12",
                "version": "1",
                # Supply the fingerprint recorded for the R12 declaration.
                "definition_fingerprint": "sha256:<R12-fingerprint>",
            }],
            "payload": {"action": "review", "priority": 2},
        })
```

`decision_id` and `target_id` are NFC-normalized, trimmed, nonempty,
case-preserving UTF-8 strings of at most 1,024 characters. Decision IDs are
unique within the decision stream and step; reusing one on a later step is
allowed. `contributing_rules` is optional, but when supplied it is a
nonempty, canonicalized list whose declared rule ID, version, fingerprint,
and target namespace all match the run declarations. This is claimed
provenance, not proof of a matching hit or causal contribution.

Read configuration, stored status, and bounded rows separately:

```python
with ResultReader(store="./results") as reader:
    target = reader.get_decision_target(run_id)
    status = reader.read_decision_status(
        run_id, through_cursor=reader.latest_cursor(), step_n=1
    )
    page = reader.read_table(
        run_id, "decisions", through_cursor=reader.latest_cursor(), step_n=1,
        limit=100, offset=0, sort=[("decision_id", "asc")],
    )
```

`status` distinguishes `not_recorded`, `logged_empty`, and `recorded`;
page results additionally distinguish `filtered_empty` and `page_past_end`.
Decision pages default to 100 rows and reject more than 1,000. Reads are
pinned to `through_cursor` and require an exactly committed `step_n`; they do
not clamp or select a nearest step. A zero-commit declaration is readable but
does not invent step 1. Arbitrary decision payload is opaque canonical JSON;
finite numbers are required and integers at or above `2**53` are stored as
decimal strings.

A rule hit is not automatically a decision, and a decision is not
automatically evaluation truth. The simulator records artifacts explicitly
logged by user Python; it does not execute rule or decision logic, infer
contributing rules from hits, materialize evaluated-subject populations, or
perform analytic evaluation.

## Reader

```python
from mini_atlas_simulator import ResultReader

with ResultReader(store="./results") as reader:
    cursor = reader.latest_cursor()
    run = reader.get_run(run_id)
    steps = reader.list_committed_steps(run_id, through_cursor=cursor)
    metrics = reader.read_metrics(
        run_id,
        definition_ids=["slice_node_count"],
        through_cursor=cursor,
        step_n=steps[-1].step_n,
    )
    page = reader.read_table(
        run_id,
        "subject_scores",
        through_cursor=cursor,
        step_n=steps[-1].step_n,
        limit=100,
        offset=0,
        sort=[("score", "desc")],
    )
    rules = reader.list_rule_definitions(run_id)
    agg = reader.read_rule_aggregates_at_step(
        run_id, through_cursor=cursor, step_n=steps[-1].step_n
    )
```

`list_rule_definitions` is not step-scoped and returns `[]` for catalogs that
predate Rule Foundation tables. Aggregate reads return empty/`logged=False`
entries when rule tables are absent; they still require a committed step at the
cursor when a `step_n` is requested.

`ResultReader` does not create schema, set WAL, or mutate writer files. A
missing store directory, empty directory, or missing `catalog.sqlite` raises
`CatalogError` and creates nothing. A format-v1 catalog that contains only
`catalog_info` (no writer tables) is a valid initialized empty store: discovery
methods return empty results and `latest_cursor()` is `0`. A nonempty partial
writer schema is `CatalogError` and is never repaired.

Composing reads take `through_cursor` and see only committed steps at or before
that cursor. Format v1 stores metric and stream declarations without a
declaration `commit_id`, so a definition declared after later commits may still
be visible when reading an earlier cursor; points remain hidden until their
publication `commit_id`. That is a format-v1 limitation, not a reader pin bug.

Opening a live WAL catalog read-only may create or map SQLite's
`catalog.sqlite-shm` index and may create an empty `catalog.sqlite-wal`. Those
sidecars are not schema changes; the reader does not write catalog pages or
change `journal_mode`.

Project and run name search treats `%`, `_`, and `\` in `q` as literals. Table
pages order nulls with DuckDB `NULLS LAST` for both `asc` and `desc`.
`_commit_id` is injected after the Parquet query and cannot be used as a sort
or filter column.

A missing `as_of` or metric point is missing / Not logged, never numeric zero.
Table `coverage.status` is one of `ok`, `not_recorded`, `logged_empty`,
`filtered_empty`, or `page_past_end`. `presentation_status` is derived from
stored status plus the writer heartbeat; it never authorizes GC or a second
writer.

## Examples

See [`examples/README.md`](examples/README.md). The first writer invocation
creates the example project. A later invocation reuses that project and adds a
distinctly named run. `read_results.py` needs `--run-id` when more than one
run exists.

```text
python examples/tabular_live.py --store ./results
python examples/tabular_live.py --store ./results --run-name daily-tabular-2
python examples/read_results.py --store ./results --run-id <run_id>
```

To watch commits arrive from a second terminal:

```text
python examples/tabular_live.py --store ./results --step-delay 0.5
```

```text
python examples/read_results.py --store ./results --poll-seconds 60
```

`examples/tabular_live.py` logs illustrative user metrics (coverage, reference
count, observed count). Those labels are not built-in product behavior. Tests
may pass `--steps` for a short run; the default remains the 365-day
demonstration.

## Current limitations

Rule declarations, aggregates, set aggregates, triggered-subject evidence, and
decisions exist in the simulator. Optional run descriptions, additive list
fields, and bounded Lineage graph metadata are also present. The simulator
remains independently usable without Flask, React, Node, Docker, or the
monitor. The separately installable monitor is read-only.

Still deferred: investigation/lineage UI, resume and stale-lock takeover, GC,
export, run comparison, graph integration, evaluation-helper suite, later
roadmap passes, Dash, and PyPI publication.
Windows support is provisional and gated on CI. This does not claim that
Windows is fully supported. Ubuntu CI jobs are provisional evidence for
`ubuntu-latest` only. They are not evidence that native Linux in general was
tested, that every Linux distribution was tested, or that desktop-browser
behavior was tested on Linux.

## Tests

```text
python -m pytest
```
