Metadata-Version: 2.5
Name: jp-idwr-db
Version: 2026.10.7
Summary: Japanese IDWR infectious disease database and analytics toolkit built on Polars.
Project-URL: Homepage, https://github.com/AlFontal/jp-idwr-db
Project-URL: Repository, https://github.com/AlFontal/jp-idwr-db
Project-URL: Bug Tracker, https://github.com/AlFontal/jp-idwr-db/issues
Author: jp-idwr-db contributors
License: GPL-3.0-or-later
License-File: LICENSE
Keywords: epidemiology,infectious-disease,japan,polars,surveillance
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Requires-Dist: fastexcel>=0.10
Requires-Dist: httpx>=0.27
Requires-Dist: openpyxl>=3.1
Requires-Dist: platformdirs>=4.2
Requires-Dist: polars>=0.20
Requires-Dist: pyarrow>=14.0
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: tomli>=2.0; (python_version < '3.11') and extra == 'dev'
Provides-Extra: excel
Requires-Dist: fastexcel>=0.10; extra == 'excel'
Description-Content-Type: text/markdown

# jp-idwr-db

[![PyPI version](https://img.shields.io/pypi/v/jp-idwr-db)](https://pypi.org/project/jp-idwr-db/)
[![Python versions](https://img.shields.io/pypi/pyversions/jp-idwr-db)](https://pypi.org/project/jp-idwr-db/)
[![CI](https://img.shields.io/github/actions/workflow/status/AlFontal/jp-idwr-db/ci.yml?branch=main\&label=CI)](https://github.com/AlFontal/jp-idwr-db/actions/workflows/ci.yml)
[![License: GPL-3.0-or-later](https://img.shields.io/badge/License-GPL--3.0--or--later-blue.svg)](https://github.com/AlFontal/jp-idwr-db/blob/main/LICENSE)

`jp-idwr-db` makes Japan's IDWR infectious disease surveillance data easier to use.

It collects the official NIID/JIHS files, cleans and combines data published across different formats and archives, and releases analysis-ready weekly data as Parquet. You can use it through the Python package or read the release files directly with DuckDB, R, Arrow, Spark, or any other tool that supports Parquet.

<p align="center">
  <img src="docs/assets/jp-idwr-db-overview.png" alt="jp-idwr-db data pipeline" width="100%">
</p>

IDWR reports are published weekly. `jp-idwr-db` checks for new data every week and publishes an updated, versioned snapshot via GitHub Actions and releases.

## Install

```bash
pip install jp-idwr-db
```

## Quick start

Load the combined dataset:

```python
import jp_idwr_db as jp

df = (
    jp.load("unified", version="latest")
    .select(["date", "prefecture", "category", "disease", "count", "source"])
)

print(df)
```

<!-- BEGIN GENERATED UNIFIED SNAPSHOT -->
```text
shape: (6_201_180, 6)
┌────────────┬────────────┬──────────┬────────────────────────────────┬───────┬───────────────────────┐
│ date       ┆ prefecture ┆ category ┆ disease                        ┆ count ┆ source                │
│ ---        ┆ ---        ┆ ---      ┆ ---                            ┆ ---   ┆ ---                   │
│ date       ┆ str        ┆ str      ┆ str                            ┆ f64   ┆ str                   │
╞════════════╪════════════╪══════════╪════════════════════════════════╪═══════╪═══════════════════════╡
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ AIDS                           ┆ 0.0   ┆ Confirmed cases       │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute encephalitis (excluding… ┆ 0.0   ┆ Sentinel surveillance │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute hemorrhagic conjunctivi… ┆ 3.0   ┆ Sentinel surveillance │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute poliomyelitis            ┆ 0.0   ┆ Confirmed cases       │
│ 1999-04-05 ┆ Aichi      ┆ total    ┆ Acute viral hepatitis          ┆ 4.0   ┆ Confirmed cases       │
│ …          ┆ …          ┆ …        ┆ …                              ┆ …     ┆ …                     │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Viral hepatitis(excluding hep… ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ West Nile fever                ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Western equine encephalitis    ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Yellow fever                   ┆ 0.0   ┆ All-case reporting    │
│ 2026-09-21 ┆ Yamanashi  ┆ total    ┆ Zika virus infection           ┆ 0.0   ┆ All-case reporting    │
└────────────┴────────────┴──────────┴────────────────────────────────┴───────┴───────────────────────┘
```
<!-- END GENERATED UNIFIED SNAPSHOT -->

For most analyses, `unified` is the dataset you want.

You can also filter while loading:

```python
tb = (
    jp.get_data(
        disease="Tuberculosis",
        year=2024,
        prefecture=["Tokyo", "Osaka", "Hokkaido"],
        version="latest",
    )
    .select(["date", "prefecture", "disease", "count", "source"])
)

print(tb)
```

The exact values depend on the selected release; the example intentionally
omits a static output snapshot so it cannot drift from `version="latest"`.

Release data are downloaded on first use and cached locally. Use `version="latest"` when you want the newest published snapshot.

Cache and release resolution can be configured with `JP_IDWR_DB_CACHE_DIR`,
`JP_IDWR_DB_DATA_VERSION`, and `JP_IDWR_DB_DATA_BASE_URL`. The former
`JPINFECT_*` names remain accepted as compatibility aliases.

## Datasets

| Dataset    | Contents                                                                        |
| ---------- | ------------------------------------------------------------------------------- |
| `unified`  | Combined, deduplicated weekly surveillance data. Recommended for most analyses. |
| `bullet`   | Preliminary weekly all-case reports (`zensu`), 2024 onwards                     |
| `sentinel` | Weekly sentinel surveillance, 1999 onwards                                      |
| `sex`      | Final annual confirmed cases by sex                                             |
| `place`    | Final annual confirmed cases by place of infection                              |

IDWR publishes each year twice: preliminary weekly reports during the year, and final annual tables about 15 months later that include late reports. The unified dataset uses the final tables for every year they cover and the preliminary reports only after that, without overlapping records. Preliminary counts can be substantially lower for diseases with reporting delays.

For preliminary sentinel years, the source `teitenrui` files report year-to-date cumulative counts, so `jp-idwr-db` converts them to weekly incidence within each year, prefecture, and disease:

```text
weekly_count = cumulative_count[t] - cumulative_count[t - 1]
```

Weeks the totals do not determine are left empty and explained in `count_status`; nothing is imputed.

The `date` column represents the Monday at the start of the corresponding ISO surveillance week.

See [DATASETS.md](./docs/DATASETS.md) for detailed schema and coverage, and [DISEASES.md](./docs/DISEASES.md) for disease-by-disease temporal coverage.

## Use the data without Python

The data releases are regular Parquet files, so the Python package is optional.

Each GitHub release contains:

* `unified.parquet` and the other dataset tables
* `manifest.json` with file metadata, schemas, sizes, and checksums
* an optional `jp_idwr_db.duckdb` database with views over the Parquet files

For example, with DuckDB:

```sql
SELECT date, prefecture, disease, count
FROM read_parquet('unified.parquet')
WHERE disease = 'Tuberculosis'
  AND year = 2024
ORDER BY date, prefecture;
```

Or query the packaged DuckDB database directly:

```bash
duckdb jp_idwr_db.duckdb
```

The same Parquet files can be read from R, Arrow, Spark, Polars, pandas, or any other compatible tool.

## Data source and updates

The package builds on official infectious disease surveillance data published by the National Institute of Infectious Diseases (NIID) and the Japan Institute for Health Security (JIHS).

The source archive has changed over time:

* final annual surveillance tables (confirmed and sentinel) are published as `.xls` / `.xlsx`
* preliminary all-case reports (`zensu`) are published weekly as `.csv`
* preliminary sentinel reports (`teitenrui`) are published weekly as `.csv`

Source: [JIHS IDWR Surveillance Data Tables](https://id-info.jihs.go.jp/en/surveillance/idwr/rapid/),
accessed 2026-10-07 for release `v2026.10.7`.

The underlying IDWR surveillance data are weekly. `jp-idwr-db` refreshes its published release every week, incorporating newly available reports into a new versioned snapshot.

Created by editing the JIHS IDWR surveillance data: source files are parsed,
cleaned, normalized, and combined by this project, and preliminary sentinel
cumulative counts are converted to weekly incidence as described above.

`jp-idwr-db` is an independent project and is not an official NIID or JIHS publication. 

It is inspired by the R package by Tomonori Hoshi, [`jpinfect`](https://github.com/TomonoriHoshi/jpinfect) and adapted for Python and more generalized usage.


See the official [JIHS terms of use](https://id-info.jihs.go.jp/en/term_of_use.pdf).

## More documentation

* [Dataset schemas and coverage](./docs/DATASETS.md)
* [Disease coverage](./docs/DISEASES.md)
* [Data wrangling examples](./docs/EXAMPLES.md)
* [Contributing](./CONTRIBUTING.md)
* [Changelog](./CHANGELOG.md)

## License

GPL-3.0-or-later. See [LICENSE](./LICENSE).
