Metadata-Version: 2.5
Name: bayesian-pv-census
Version: 0.1.0
Summary: Turn a detector's output into a census with credible intervals, then audit a register against it
Project-URL: Homepage, https://github.com/gabrielkasmi/bayesian-pv-census
Project-URL: Paper, https://doi.org/10.5281/zenodo.21534856
Author-email: Gabriel Kasmi <gabkasmi@gmail.com>
License: MIT
License-File: LICENSE
Keywords: audit,bayesian,census,measurement error,photovoltaic,registry,remote sensing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scipy>=1.9
Provides-Extra: dev
Requires-Dist: matplotlib>=3.6; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: figures
Requires-Dist: matplotlib>=3.6; extra == 'figures'
Description-Content-Type: text/markdown

# bayesian-pv-census

**Turn a detector's raw count into a census with credible intervals, then audit a register against it.**

A detector that finds objects in imagery misses some and invents others, at rates
that vary from place to place. Its raw total is therefore not a measurement, and
comparing it directly to an official register says as much about the detector as
about the register. This package closes that gap: given a validation sample, it
turns the raw total into a posterior over the true quantity, and turns that
posterior into a verdict on whatever the register reports.

Nothing here knows about photovoltaics, France, or geometry. A *unit* is anything
with a raw total and a validation sample — a department, a grid cell, a utility
service area. `raw` is whatever you chose to count: installed capacity, number of
installations, roof area.

This is the statistical core of *Nationally Consistent, Locally Incomplete: A Bayesian Remote-Sensing Audit of Rooftop Photovoltaic Registries* (Kasmi et al., 2026, pending peer review) , extracted as a
library. Every public function corresponds to a named component of the paper.

## Install

```bash
pip install bayesian-pv-census            # numpy, scipy, pandas
pip install 'bayesian-pv-census[figures]' # adds matplotlib
```

## One minute

```python
from bayesian_pv_census import UnitRecord, correct_unit, evaluate

unit = UnitRecord(
    unit_id="dept_86",
    raw=26_800,                          # what the detector found, any consistent unit
    precision_tp=116, precision_fp=4,    # 120 detections checked by hand
    recall_tp=68, recall_fn=33,          # 101 real objects located independently
    reported_value=26_820,               # what the register reports
)

result = correct_unit(unit)
print(f"{result.mean:,.0f}  99% CI {result.ci(0.99)}")
print(evaluate(result, unit.reported_value).status)   # below | within | above
```

The verdict is decided by the interval, not the gap. A large discrepancy inside a
wide interval is not a finding; a small one outside a tight interval is.

## The paper's data

```python
from bayesian_pv_census import correct_batch, load_demo

units = load_demo()                      # 93 French reporting units, kWp
correct_batch(units, prior="empirical_bayes")["corrected_mean"].sum()
# 4_033_003 kWp, the paper's national estimate
```

`load_demo()` returns the 93 continental French reporting units behind the
published audit: detected rooftop capacity below 36 kWp, the manual validation
counts (31,853 annotations), and the size-weighted rates of specification B.

**One column is not yet included.** The reference values under audit are the
French transmission system operator's grid-connection registry, and
redistributing them is not ours to grant. Until that clearance arrives the demo
supports estimation but not the audit, `load_demo()` says so, and
`has_reported_values()` reports it:

```python
from bayesian_pv_census import has_reported_values
has_reported_values()   # False in this release
```

Nothing is substituted in the meantime. A column of plausible-looking
placeholders would be indistinguishable from data at a glance, and reproducing a
published audit against invented references is worse than not reproducing it. The
four regression tests that need the column are skipped rather than weakened, so
the skip count in `pytest` is the honest signal. The national estimate above
needs no reference value and is checked in every release.

Once the column ships, the full battery reproduces the paper: 25 units
under-reported and 8 over-reported under specification A, an 18-unit hard core
and a 7-unit negative control.

Installation counts are deliberately absent for a different reason. The audit in
the paper is about capacity; shipping a count column would imply a quantity that
was never audited.

## What the correction assumes

The estimator is `raw × P/R`. Four assumptions stand behind it, and the package
can only speak to two of them.

| | Assumption | Can the package check it? |
|---|---|---|
| **H1** | Detection status is independent of object size: true positives, false positives and false negatives have the same mean size. | Partly. The gap between specifications A and B measures the violation, and B does not require H1. |
| **H2** | The detector is run over the entire unit; no sub-region is excluded. | No. Upstream of anything the package sees. |
| **H3** | The validation samples are drawn representatively from, respectively, the raw detections and the true population. | No. This is a property of how you annotated, and nothing in the counts reveals it. |
| **H4** | For a correctly detected object, its estimated size is unbiased for its true size. | Only its sensitivity, via the `scale` key of `run_battery`. |

H4 binds only when the quantity is a size. **If you correct a count of objects
rather than a capacity, H4 drops out entirely** — which is why the field is named
`raw` and not `raw_capacity`.

Two of the four are therefore assumptions you carry, not results the package
delivers. Reporting a credible interval without saying which of H2 and H3 you
believe, and why, states less than it appears to.

## Before you annotate

The interval's half-width has a closed form, so the annotation effort can be
budgeted in advance rather than discovered afterwards:

```python
from bayesian_pv_census import required_sample_size

b = required_sample_size(target_half_width=0.15, expected_precision=0.85,
                         expected_recall=0.65, level=0.99)
print(b.n_precision, b.n_recall)   # annotations needed per unit
```

## Specifications and the hard core

A verdict that only holds under one way of computing precision and recall is not
a finding. `run_battery` runs the audit under several and keeps what survives all
of them.

```python
from bayesian_pv_census import run_battery

battery = run_battery(units, prior="empirical_bayes")
battery.hard_core("below")       # flagged under-reported by every specification
battery.negative_control()       # flagged the other way, unanimously — the control group
battery.concordance_table()      # crosstab; empty off-diagonal corners mean no sign flips
```

Specification A counts annotated objects. Specification B weights them by size.
**B is implemented exactly as in the paper**, which computes the weighted rates as
point estimates and applies them as a deterministic rescaling of A's posterior —
so B's interval is A's interval, shifted. That is a known limitation of the
published method; the package reproduces it rather than improving on it, because
reproducing the paper is the point. See the docstring of `correct_unit` for what
a properly weighted posterior would require, and for why the `min_weighted_n`
fallback matters more than its name suggests.

A third axis needs no specification of its own. Rescaling every raw quantity by a
constant — a different surface-to-power coefficient, a different filtering
threshold — multiplies the posterior and both its bounds while the reported value
stays put, so it is passed generically:

```python
run_battery(units, specs={"A": {}, "B": {"weighted": True},
                          "C_low": {"scale": 5.5 / 5.0}})
```

The status is monotone in that constant and flips once, so the flipping point has
a closed form and no sweep is needed:

```python
from bayesian_pv_census import conversion_threshold
conversion_threshold(result, unit.reported_value, "below")
```

The coefficient itself stays outside the engine. `raw` arrives already converted,
and a `conversion_coefficient` argument would import photovoltaics into a core
that knows nothing about it.

## What forming the factor at a coarser scale costs

Correcting each unit and summing is not the same as pooling the units' rates and
correcting once. `1/R` is convex, so pooling always yields a smaller factor and a
smaller total. The cost is exactly zero when recall is homogeneous across the
pooled units, whatever the dispersion of precision, and second order in the
coefficient of variation of recall otherwise:

```python
from bayesian_pv_census import compare_aggregation_scales, pooling_penalty

compare_aggregation_scales(units, groups={"north": [...], "south": [...]}).totals
pooling_penalty(units)   # CV(R)^2 - rho * CV(P) * CV(R), and its two terms
```

The practical consequence is that there is no optimal grid to search for. Forming
the factor at the finest level your annotation budget supports is weakly better
in every case.

## Scope

Out of scope by design, and unlikely to change: geospatial sampling, annotation
tooling, temporal alignment between imagery and register, and detector-specific
parsers. Those are format- and project-specific; this package starts once you can
write down a raw total and a validation sample.

## Tests

```bash
pip install -e '.[dev]' && pytest
```

The suite validates the mathematics on synthetic data — the closed form against
the bootstrap, the monotonicity of the budgeting rule, the structural invariants
of the hard core, the exact vanishing of the pooling penalty under homogeneous
recall — and then checks the published French numbers against the shipped demo
data, so a regression in the engine cannot pass silently.

## Citation

See `CITATION.cff`. Licence MIT.
