Metadata-Version: 2.4
Name: nzi-pipeline
Version: 0.1.1
Summary: Sector-agnostic energy modelling pipeline for India's NZI scenario, backed by DuckDB. Local runs + S3-versioned seeds/outputs.
Author: Ritika
License: MIT
Keywords: energy,modelling,duckdb,scenarios,pier,rumi,iit-delhi,india,decarbonisation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: duckdb>=0.9
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: openpyxl>=3.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: pydantic>=2.0
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn>=0.27
Requires-Dist: httpx<0.28,>=0.27
Requires-Dist: boto3>=1.34
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: httpx<0.28,>=0.27; extra == "dev"
Requires-Dist: moto[s3]>=5.0; extra == "dev"

# NZI — Net Zero India

A sector-agnostic energy-demand modelling pipeline for India's net-zero
pathway analysis. Backed by **DuckDB**, it ingests Excel inputs, computes
PIER parameters, and reproduces RUMI-format demand outputs end-to-end —
with a web UI, snapshots, undo, tests, CI, and a one-command Docker
deployment.

---

## Highlights

- **Two sectors live, one framework** — `residential` and `transport`
  share the same CLI, schema, and step runner.
- **DuckDB-backed pipeline** — replaces brittle Excel chains with an
  auditable, queryable database (`residential.duckdb` ~3.8 GB,
  `transport.duckdb` smaller).
- **PIER + RUMI-compatible output** — `nzi_pipeline` writes PIER
  parameter CSVs, `pier_db` reimplements the RUMI demand formula and
  produces byte-comparable demand CSVs.
- **FastAPI web UI** — browse tables, edit cells, run steps with live
  log streaming, manage snapshots, and undo any edit.
- **Reproducible runs** — named DB snapshots, append-only audit log,
  staleness tracking via dependency graph.
- **Tested + CI** — 50 pytest cases (33 unit + 17 integration), golden
  fixtures regenerable on demand, GitHub Actions on every PR.
- **Containerised** — `docker compose up` builds the env, seeds both
  DBs, and serves the UI on port 8000.

---

## Quick Start

### Local (recommended for development)

```bash
# 1. Install
pip install -e ".[dev]"

# 2. Seed database from Excel workbooks (~4 min, includes 12 GB climate data)
python -m nzi_pipeline seed --sector residential
python -m nzi_pipeline seed --sector transport

# 3. Run the parameter pipeline
python -m nzi_pipeline run --sector residential
python -m nzi_pipeline run --sector transport

# 4. Compute demand (RUMI-format output)
python -m pier_db run --sector D_RES
python -m pier_db run --sector D_TRANS

# 5. (Optional) Start the web UI
python -m frontend            # http://127.0.0.1:8000/
```

### Docker

```bash
docker compose up -d --build
docker compose logs -f        # watch first-boot seed
# UI on http://<host>:8000/
```

Full EC2 deployment guide: [DEPLOY.md](DEPLOY.md).

### Legacy Excel pipeline (reference only)

```powershell
powershell -ExecutionPolicy Bypass -File setup.ps1
cd General\Residential_Sector_Data\Residential_Workflow_FromInput
python run_all.py
```

---

## Architecture

```
                  Excel Workbooks (RUMI_Input/)
                              |
                       nzi_pipeline seed
                              |
                              v
                  residential.duckdb / transport.duckdb
                       /         |         \
              nzi_pipeline    frontend     pier_db
                  run         (web UI)      run
                   |             |            |
                   v             v            v
            PIER Parameter   Edit / Run    Demand Output
              CSVs (~34)     / Snapshot    CSVs (~57)
                              / Undo
```

### Component summary

| Component | What it does |
|---|---|
| `nzi_pipeline/` | Sector-agnostic pipeline framework — seed Excel into DuckDB, compute parameters (ES_Demand from 112M climate rows, SEC, stock-flow), export PIER CSVs. |
| `pier_db/` | Reimplements RUMI's demand formula (`NC × NI × ES_Demand × UP × TSR × ELS × SEC`) reading from DuckDB instead of CSV. Output is RUMI-format. |
| `frontend/` | FastAPI single-page UI — browse, edit, run, snapshot, undo. Same code paths as the CLI, so all UI actions appear in the audit log. |
| `tests/` | pytest harness — unit + integration + fixture-based regression with golden manifests. |
| `scripts/` | Shell wrappers used by both devs and CI (`test.sh`, `seed.sh`, `run.sh`, `dev-docker.sh`). |

---

## CLI Reference

### `nzi_pipeline` — parameter pipeline

```bash
python -m nzi_pipeline seed   --sector <name>           # Excel -> DuckDB
python -m nzi_pipeline seed   --sector <name> --force   # drop + re-seed
python -m nzi_pipeline run    --sector <name>           # run all steps
python -m nzi_pipeline run    --sector <name> --force   # force rebuild
python -m nzi_pipeline steps  --sector <name>           # list steps
```

Implemented sectors: `residential`, `transport`.

### `pier_db` — demand computation

```bash
python -m pier_db run --sector D_RES
python -m pier_db run --sector D_TRANS
python -m pier_db run --sector D_RES --output ./output  # custom output dir
```

### `frontend` — web UI

```bash
python -m frontend                          # 127.0.0.1:8000
python -m frontend --reload                 # dev auto-reload
python -m frontend --port 9000 --no-browser
```

---

## Frontend Features

A single-page UI on top of `nzi_pipeline.core.*`. See
[frontend/README.md](frontend/README.md) for full endpoint reference.

| Tab | What it does |
|---|---|
| **Dashboard** | Staleness summary, recent edits, snapshots at a glance. |
| **DAG** | Visual step graph (left → right by topology); click for upstream/downstream lineage. |
| **Tables** | Paginated browser for `input` / `assumption` / `computed` / `_meta`; in-cell edits to `input` and `assumption` are logged. |
| **Run** | Plan, run-stale, or force rebuild — with live log streaming via SSE. |
| **Audit & Undo** | Append-only edit history; one click reverts any edit. |
| **Snapshots** | Create, restore, or delete named DB snapshots. |

DuckDB doesn't allow concurrent rw connections to one file, so the
frontend caches a per-pipeline connection and **releases it after each
job** so the CLI / `pier_db` can run between jobs.

---

## Repository Structure

```
NZI/
├── nzi_pipeline/                      # DuckDB pipeline framework
│   ├── core/                          #   Generic: db, config, registry, refresh, export
│   └── sectors/
│       ├── residential/               #   Residential: seed, defaults, schema, 7 steps
│       └── transport/                 #   Transport: seed, defaults, schema, steps
├── pier_db/                           # Demand computation (replaces rumi_demand)
│   ├── config/                        #   RUMI mapping CSVs (Common, Demand)
│   ├── demand.py                      #   Core formula: NC x UP x NI x TSR x ELS x SEC x ES
│   ├── parameters.py                  #   Read parameters from DuckDB
│   └── export.py                      #   Write RUMI-format output CSVs
├── frontend/                          # FastAPI web UI
│   ├── app.py                         #   REST + SSE endpoints
│   ├── __main__.py                    #   `python -m frontend` entry point
│   └── static/                        #   index.html + app.js + style.css (no build)
├── tests/                             # pytest suite (50 tests)
│   ├── unit/                          #   33 tests, in-memory DuckDB, <1s
│   ├── integration/                   #   17 tests, ~55s, fixture-driven
│   └── fixtures/{residential,transport}_min/  # 1.5 MB sliced Parquet inputs
├── scripts/                           # dev + CI shell wrappers
│   ├── test.sh                        #   lint + unit + integration
│   ├── seed.sh / run.sh               #   convenience wrappers
│   ├── build_{residential,transport}_fixture.py
│   └── build_manifest.py              #   regenerate golden output hashes
├── .github/workflows/ci.yml           # lint + unit + integration + docker-build
├── General/Residential_Sector_Data/   # Professor's reference data + legacy pipeline
├── PIER/                              # Professor's PIER scenario outputs (read-only)
├── Dockerfile, docker-compose.yml,
│   docker-entrypoint.sh, environment.yml
├── DEPLOY.md                          # EC2 deployment guide
├── aws_deployment_plan.csv            # Spot-Batch + S3 production plan
├── pyproject.toml                     # Package definition (MIT license)
├── residential.duckdb / transport.duckdb   # Generated, git-ignored
└── README.md
```

---

## Testing

```bash
bash scripts/test.sh all          # lint + unit + integration (~55s)
bash scripts/test.sh unit         # 33 tests, <1s
bash scripts/test.sh integration  # 17 tests, ~55s
```

| Layer | Coverage |
|---|---|
| **Unit** | In-memory DuckDB; covers `edit`, `snapshot`, `_quote`, pier_db static maps. |
| **Integration** | Database lifecycle, real-DB smoke (auto-skips if locked), full residential + transport pipeline e2e on tiny fixtures. |
| **Manifest check** | Golden-output regression at 1e-6 relative tolerance. |

### Regenerating fixtures / golden manifests

```bash
python scripts/build_residential_fixture.py
python scripts/build_transport_fixture.py
python scripts/build_manifest.py residential   # after intentional math change
python scripts/build_manifest.py transport
```

### CI

`.github/workflows/ci.yml` runs on every PR to `main` / `op_dev`:

1. **Lint** — `ruff` strict on `tests/`, advisory on `nzi_pipeline/` + `pier_db/`.
2. **Unit tests** — Python 3.11, pip-cached.
3. **Integration tests** — fixture-driven; real-DB tests auto-skip in CI.
4. **Docker build** — only on push to `main`; uses GHA cache.

In-flight runs on the same branch are auto-cancelled to save minutes.

---

## Validation against the professor's reference

```bash
cd General/Residential_Sector_Data/Residential_Workflow_FromInput
python validate_parameters_keyed.py    # PIER params vs professor
python validate_demand_keyed.py        # demand output vs professor
python compare_outputs.py              # pier_db output vs professor (latest)
```

**Current residential match rates:**

- Parameters: **32 / 34** files IDENTICAL.
- Demand: **37 / 56** files within 1% (after year fix); 13 files >10%
  remain in cooling + cook fuel mix (tracked in
  [`compare_outputs_report.md`](compare_outputs_report.md)).

---

## Adding a New Sector

The framework is designed so new sectors plug in cleanly:

1. Create `nzi_pipeline/sectors/<sector>/` with `config.py`,
   `defaults.py`, `schema.sql`, `seed.py`.
2. Add computation steps under `steps/`.
3. Register the sector in `nzi_pipeline/cli.py`.
4. Create `pier_db/config/Demand/<D_SECTOR>/` with mapping CSVs.
5. Add a fixture builder under `scripts/` and a golden manifest.

Transport is the live example — copy its layout.

---

## Documentation

| Document | Covers |
|---|---|
| [TABLES.md](TABLES.md) | Database schemas (`input` / `assumption` / `computed` / `_meta`) and what every table stores |
| [DEPLOY.md](DEPLOY.md) | EC2 deployment via Docker Compose |
| [aws_deployment_plan.csv](aws_deployment_plan.csv) | Production plan (Spot-Batch, S3, Step Functions) |
| [frontend/README.md](frontend/README.md) | Full UI tour and API reference |
| [General/Residential_Sector_Data/README_Residential.md](General/Residential_Sector_Data/README_Residential.md) | Pipeline overview, inputs, outputs, dependencies |
| [General/Residential_Sector_Data/CODE_STRUCTURE.md](General/Residential_Sector_Data/CODE_STRUCTURE.md) | Scripts, modules, step-by-step data flow |
| [General/Residential_Sector_Data/DB_PROPOSAL.md](General/Residential_Sector_Data/DB_PROPOSAL.md) | Why DuckDB, storage model, migration plan |
| [General/Residential_Sector_Data/DB_SCHEMA.md](General/Residential_Sector_Data/DB_SCHEMA.md) | Table and view definitions |
| [General/Residential_Sector_Data/DB_REFRESH.md](General/Residential_Sector_Data/DB_REFRESH.md) | Dependency graph and `refresh()` function |

---

## Project Conventions

- **Never touch professor folders** —
  `General/Residential_Sector_Data/Parameters/` and `PIER/Scenarios/`
  are read-only references.
- **Generate static files in the pipeline** — don't copy from the
  professor; produce them in `s06_export.py`.
- **Match RUMI semantics exactly** — `pier_db` mirrors
  [github.com/prayas-energy/Rumi](https://github.com/prayas-energy/Rumi).
- **Lint** — strict for `tests/`; existing `nzi_pipeline/` + `pier_db/`
  warnings are advisory until cleanup.

---

## Git Notes

- `residential.duckdb` / `transport.duckdb` are git-ignored — regenerate
  with `python -m nzi_pipeline seed`.
- `PIER/` is git-ignored (managed by external team).
- `RUMI_Input_code/` is git-ignored (legacy-pipeline output).
- **`index.lock` errors** — VS Code and terminal can race; run
  `rm .git/index.lock` (or `clear-git-lock.ps1` on Windows).
- **DuckDB locked** — if the frontend crashed, `Stop-Process -Id <pid>
  -Force`, then restart.

---

## License

MIT — see [pyproject.toml](pyproject.toml).
