Metadata-Version: 2.4
Name: mini-atlas-graph-warehouse
Version: 0.1.1
Summary: DuckDB-backed merge/upsert warehouse for GraphIngestor parquet exports
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: duckdb>=1.1
Requires-Dist: pyarrow>=14
Requires-Dist: pyfiglet>=1.0
Requires-Dist: tqdm>=4.66
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Provides-Extra: igraph
Requires-Dist: igraph<2,>=1.0; extra == "igraph"
Requires-Dist: numpy>=1.26; extra == "igraph"
Dynamic: license-file

# GraphWarehouse

Local DuckDB warehouse for entity–relationship graphs: ingest GraphIngestor Parquet, query bounded subgraphs, join stored node and edge measurements, convert selected graphs to iGraph, write results back, export to Gephi, and prepare toolbox input CSVs.

It is a general-purpose storage, query, analytics, iGraph, and export library. Typical work includes neighborhood and ego analysis, shared-resource structures, temporal slices, type-scoped score policies, constrained traversal, and Gephi publication.

Analytics results are named node or edge measurements produced **outside** GraphWarehouse. They may be anomaly scores, centrality, classifications, communities, model outputs, business metrics, or other user-defined signals. GraphWarehouse stores, discovers, filters, joins, traverses, exports, and writes those results back. It does not run external scoring algorithms.

**Safety:** keep work bounded. Seeds, type allowlists, hop counts, and windows limit the **result**. They do not make warehouse scans free. Full-warehouse iGraph conversion is in-memory and not out-of-core. Do not convert an unfiltered 50–100M-edge warehouse under a typical analyst process. Windows and 50–100M calibration remain environmental validation, not certified results.

## Capabilities

| Area | What you can do |
| --- | --- |
| Ingest | Merge GraphIngestor Parquet runs into one `.duckdb` with surrogate `node_id` / `edge_id` and null-aware property merge |
| Query | Type filters, bipartite views, ego N-hop, temporal windows, CSV / Arrow / Gephi / toolbox CSVs |
| Analytics | Load, list, summarize, and delete named runs; seed, filter, and constrain traversal by stored scores |
| iGraph | Convert a **bounded** `GraphSlice` (optional extra); write vectors back; export the in-memory graph to Gephi |

Analysts: start with the [cheat sheet](docs/ANALYST_CHEAT_SHEET.md), then the [workflow guide](docs/ANALYST_WORKFLOWS.md). Building a Parquet graph from SQL is GraphETL (a separate package); GraphQuery and GraphSlice stay in this package and are not part of GraphETL. Jupyter recipes: [graph query cheat sheet](docs/GRAPH_QUERY_CHEAT_SHEET.md). Technical recipes: [graph query cookbook](docs/GRAPH_QUERY_COOKBOOK.md). Lookup: [API reference](docs/API_REFERENCE.md). Guides: [analytics](docs/ANALYTICS_QUERY.md), [iGraph](docs/IGRAPH_INTEGRATION.md). Known unfinished CLI/seed behavior: [KNOWN_NOT_FROZEN.md](docs/KNOWN_NOT_FROZEN.md).

## Install

```bash
pip install mini-atlas-graph-warehouse
```

Optional in-memory graphs (`igraph>=1.0,<2` and `numpy>=1.26`):

```bash
pip install 'mini-atlas-graph-warehouse[igraph]'
```

Contributor/editable installation from a clone is a secondary development path:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -e /path/to/mini-atlas-graph-warehouse
python -m pip install -e "/path/to/mini-atlas-graph-warehouse[dev]"
python -m pip install -e "/path/to/mini-atlas-graph-warehouse[igraph]"
```

From the repository root, the development extra is:

```bash
python -m pip install -e ".[dev]"
```

The `igraph` extra is not a core dependency and is not in package `__all__`.

## Example

```python
from graph_warehouse import GraphQuery, GraphWarehouse

with GraphWarehouse("graph.duckdb") as wh:
    wh.ingest_runs(["data/run_a", "data/run_b"])
    print(wh.counts())

with GraphQuery("graph.duckdb") as gq:  # read-only; does not create a missing file
    with gq.ego(uids=["user:1"], hops=2).build() as slice_:
        print(slice_.counts())
        slice_.to_csv("out/ego")
```

`GraphWarehouse` owns ingest and analytics writes. `GraphQuery` is read-only by default. Finish and close writers before opening the same file with a differently configured connection. Prefer `with` for both the query session and each slice so temporary relations are dropped.

## Ingest

GraphIngestor `export_parquet` (`local-graph-ingestor-parquet` version `1`):

```text
data/run_a/metadata.json
data/run_a/nodes/*.parquet    # uid, type, properties, event_ts
data/run_a/edges/*.parquet    # from_id, to_id, type, properties, event_ts
graph.duckdb
```

Parquet columns stay string identities. The warehouse assigns `node_id` / `edge_id`. File stem is used as `type` when a parquet file has no `type` column. `properties` may be STRUCT, MAP, JSON, or JSON-encoded VARCHAR.

| `metadata.json` | Behavior |
| --- | --- |
| Missing, or object with no / empty `format` | Legacy glob of `nodes/*.parquet` and `edges/*.parquet` |
| `format` `local-graph-ingestor-parquet`, integer `version` `1` | Strict v1 adapter; **manifest is authoritative** |
| That format with any other version, or any other nonempty `format` | Fail the run |
| Present but not a JSON object | Fail the run |

v1 paths must be relative POSIX `nodes/<file>.parquet` or `edges/<file>.parquet`. Unlisted parquet files are skipped (`WARNING skip stray parquet`, `stray_files` in stats). Jobs run in sorted path order. Valid v1 input is assumed unique on `(uid, type)` and `(from_id, type, to_id)` inside the run; collisions with **already stored** rows still use `NOT EXISTS` and unique constraints. Legacy input keeps defensive identity `GROUP BY`. `parquet_batch_size` controls Arrow read size only.

`nodes_upserted` and `edges_upserted` count **rows read from Parquet**, not rows newly inserted or changed in DuckDB. Duplicate identities in a file, logical upserts, and `skip_if_ingested=False` reingestion of the same identities still report the Parquet row counts. They are not a net-delta of warehouse rows.

A uid cannot be ingested under two conflicting concrete node types. Conflicts raise `ValueError` and roll back the complete ingest. `_untyped` endpoint placeholders may still upgrade to one concrete type. The same uid with the same type remains idempotent. Warehouse identity is not being redesigned.

Each `ingest_run` is one transaction. Metadata, manifest, path, timestamp, or SQL failure rolls **that run** back. Prior successful runs stay.

Skip is allowed only when the stored `run_id` matches, the resolved export path is the same, and `graph_id` is compatible (equal when both sides have it; missing on both sides is compatible; present-vs-absent or two different values conflict). A successful skip is reported with `skipped: true` plus a reason. That skip is an identity match, not proof that file bytes are unchanged. Rewriting an export in place still skips unless you pass `skip_if_ingested=False` or CLI `--force`. Reusing a `run_id` for a different directory or conflicting `graph_id` raises `ValueError` and the CLI exits nonzero. `--force` / `skip_if_ingested=False` means upsert/reingest. It does not delete rows previously contributed by that run that are absent from the rewritten export; Warehouse does not have safe row-level run provenance for deletion. There is no content hash and no `export_id`.

Offset-aware ISO-8601 `event_ts` (including `Z`) is stored as naive UTC DuckDB `TIMESTAMP`. Incoming null does not wipe a stored timestamp. Naive or malformed timestamps fail the run. There is no schema migrator; re-ingest with `--force` to fill timestamps on older files.

Property merge: later non-null wins; JSON null is not SQL NULL (`json_type(value) <> 'NULL'`).

Recent query performance work did not change the persistent schema. Current surrogate-key warehouses open with `GraphQuery` with no migration. Very old pre-surrogate databases still need a rebuild from parquet.

CLI: `python -m graph_warehouse.cli` or `graph-warehouse` after install.

```bash
python -m graph_warehouse.cli ingest --db graph.duckdb --run data/run_a --run data/run_b
python -m graph_warehouse.cli ingest --db graph.duckdb --data-dir data/
python -m graph_warehouse.cli counts --db graph.duckdb
```

Tables and merge details: [package README](graph_warehouse/README.md). Method signatures: [API reference](docs/API_REFERENCE.md).

## Query, analytics, and export

Builder, slice lifecycle, windows, seeds, and exports: [query README](graph_warehouse/query/README.md) and [API reference](docs/API_REFERENCE.md). CSV/Arrow/Gephi columns: [export README](graph_warehouse/query/export/README.md).

Callers select an **exact** analytics `run_id`. Do not use `runs[0]`. Analytics observation windows (`params["_provenance"]["window"]`) are not warehouse `edges.event_ts`. Final filters and traversal constraints produce different reachable graphs. Type-scoped edge analytics can require scores on listed types while other selected types pass through.

Analytics CSV headers with supported unusual quoting (embedded `'` / `"`) are escaped in generated SQL. A CSV or analytics-directory load that matches **zero** warehouse identities does not register an empty successful algorithm run. Partial matches still register and store the matching rows. Unmatched identities in CSV loads are skipped; iGraph writeback unmatched identities remain a hard error.

`configure_logging` configures the `graph_warehouse` package logger (not the root logger) and sets `propagate=False` so records do not also go to ancestor loggers. Library code does not configure logging by itself.

```bash
python -m graph_warehouse.cli query --db graph.duckdb --out out/ego \
  --seed-uid user:1 --hops 2 --format csv
python -m graph_warehouse.cli load-results --db graph.duckdb \
  --csv edge_results.csv --entity edge --run-id relationship_scores_2026w01 --algo external_anomaly_model
```

## Tests and diagnostics

```bash
pytest -q
python -m pip install -e ".[igraph]"   # required for tests/test_igraph_slice.py
pytest -m e2e
GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1 pytest tests/test_ingestor_parquet_contract.py
GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1 pytest tests/e2e/test_ingestor_boundary.py
```

GraphIngestor is not a core or development dependency. Ordinary pytest skips those contract tests when it is missing unless `GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1`.

Opt-in (not CI, not Windows certification): [bench/README.md](bench/README.md), [scripts/analyst_windows_smoke.md](scripts/analyst_windows_smoke.md). Historical architecture notes are not the public entry point: `GRAPH_WAREHOUSE_REFACTOR_PLAN.md` and related plan files.
