Metadata-Version: 2.4
Name: ncdb-tools
Version: 0.4.0
Summary: Tools for processing and analyzing National Cancer Database (NCDB) data
Project-URL: Homepage, https://github.com/brant01/ncdb-tools
Project-URL: Bug Tracker, https://github.com/brant01/ncdb-tools/issues
Project-URL: Source Code, https://github.com/brant01/ncdb-tools
Author: Jason Brant
License: MIT
License-File: LICENSE
Keywords: analysis,cancer,database,medical-research,ncdb,parquet
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Requires-Dist: polars>=1.0.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: tqdm>=4.67.1
Provides-Extra: config
Requires-Dist: python-dotenv>=1.0.0; extra == 'config'
Provides-Extra: dev
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: types-psutil>=5.9.0; extra == 'dev'
Requires-Dist: types-tqdm>=4.65.0; extra == 'dev'
Description-Content-Type: text/markdown

# ncdb-tools

[![PyPI version](https://img.shields.io/pypi/v/ncdb-tools.svg)](https://pypi.org/project/ncdb-tools/)
[![Python versions](https://img.shields.io/pypi/pyversions/ncdb-tools.svg)](https://pypi.org/project/ncdb-tools/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

A Python package for working with National Cancer Database (NCDB) data. Convert NCDB fixed-width data files into optimized parquet datasets, apply standard cancer-specific transformations, and query data efficiently using [Polars](https://pola.rs/).

## Features

- **Data Ingestion** -- Convert NCDB fixed-width `.dat` files to parquet format using SAS label definitions
- **Automatic Transformations** -- Age handling, site groupings, histology groupings, schema normalization
- **Efficient Querying** -- Fluent API to filter by primary site, histology, diagnosis year, and more
- **Data Dictionary** -- Auto-generate data dictionaries in CSV, JSON, and HTML with variable descriptions
- **Memory Efficient** -- Automatic memory detection, lazy evaluation, streaming support
- **Type Safe** -- Full type hints, input validation with `NCDBValidationError`

## Installation

```bash
pip install ncdb-tools
```

## Data Access

**This package does not include NCDB data.** You must obtain access through official channels:

- **NCDB Participant Sites**: Contact your institution's cancer registry
- **Research Access**: Apply through the [American College of Surgeons](https://www.facs.org/quality-programs/cancer-programs/national-cancer-database/)
- **Data Use Agreements**: Required for all NCDB data usage

## Quick Start

### Build a Dataset

```python
import ncdb_tools

result = ncdb_tools.build_parquet_dataset(
    data_dir="/path/to/ncdb/dat/files",
    generate_dictionary=True,
)
print(f"Dataset: {result['parquet_dir']}")
```

### Query Data

```python
import ncdb_tools

# Fluent filtering API
df = (ncdb_tools.load_data("/path/to/parquet/dataset")
      .filter_by_primary_site(["C509", "C500"])   # Breast
      .filter_by_histology([8500, 8140])           # Ductal, Adenocarcinoma
      .filter_by_year([2018, 2019, 2020])
      .collect())

# Explore before collecting
query = ncdb_tools.load_data("/path/to/parquet/dataset")
print(query)            # NCDBQuery(rows=2500000, cols=338, path=...)
print(query.count())    # 2500000
print(query.columns)    # ['PUF_CASE_ID', 'AGE', ...]

sample = query.sample(n=1000)
```

## Query API

### Filter Methods

All filter methods return `self` for chaining:

```python
query = (ncdb_tools.load_data(path)
         .filter_by_primary_site(["C509"])
         .filter_by_histology([8500])
         .filter_by_year([2020, 2021])
         .drop_missing_vital_status())
```

| Method | Description |
|--------|-------------|
| `filter_by_primary_site(sites)` | Filter by ICD-O-3 primary site codes |
| `filter_by_histology(codes)` | Filter by histology codes |
| `filter_by_year(years)` | Filter by diagnosis year |
| `drop_missing_vital_status()` | Remove cases with missing vital status |
| `filter_active_variables()` | Keep only variables with data in most recent year |
| `select_demographics()` | Select common demographic variables |
| `select_outcomes()` | Select common outcome variables |

### Utility Methods

```python
query.columns        # list of column names
query.count()        # row count without collecting
query.sample(n=100)  # random sample as DataFrame
query.describe()     # {"total_rows": ..., "columns": ..., "parquet_path": ...}
query.collect()      # execute and return DataFrame
query.lazy_frame     # access the underlying Polars LazyFrame
```

### Polars Integration

Any Polars LazyFrame method can be called directly on the query object:

```python
import polars as pl

result = (ncdb_tools.load_data(path)
          .filter_by_primary_site(["C509"])
          .select(["PUF_CASE_ID", "AGE", "PRIMARY_SITE", "YEAR_OF_DIAGNOSIS"])
          .filter(pl.col("AGE") > 50)
          .group_by("PRIMARY_SITE")
          .agg(pl.count())
          .collect())
```

## Standard Transformations

`build_parquet_dataset()` automatically applies:

1. **Data Type Conversion** -- Numeric columns detected and converted; clinical codes preserved as strings
2. **Age Processing** -- Original `AGE` preserved; `AGE_AS_INT` (numeric, 90 for "90+") and `AGE_IS_90_PLUS` flag added
3. **Site Groupings** -- `SITE_GROUP` derived from primary site codes (Breast, Lung, Colon, etc.)
4. **Histology Groupings** -- `HISTOLOGY_GROUP` for major cancer types (Adenocarcinoma, Squamous, Melanoma, etc.)
5. **Schema Normalization** -- Consistent data types across all year files

## Cancer-Specific Filtering

### Primary Site

```python
# ICD-O-3 topography codes
breast = query.filter_by_primary_site(["C509", "C500", "C501"])
lung   = query.filter_by_primary_site(["C340", "C341", "C349"])
colon  = query.filter_by_primary_site(["C180", "C182", "C187"])
```

### Histology

```python
# ICD-O-3 morphology codes
adenocarcinoma = query.filter_by_histology([8140, 8141, 8142])
squamous       = query.filter_by_histology([8070, 8071, 8072])
melanoma       = query.filter_by_histology([8720, 8721, 8722])
```

### Treatment Analysis

```python
import polars as pl

treatment = (ncdb_tools.load_data(path)
    .filter_by_primary_site(["C509"])
    .lazy_frame
    .group_by("RX_SUMM_SURG_PRIM_SITE")
    .agg([
        pl.count().alias("n"),
        pl.col("DX_DEIDENTIFY_DAYS").mean().alias("mean_days"),
    ])
    .sort("n", descending=True)
    .collect())
```

## Data Dictionary

Generated dictionaries include column names, data types, variable descriptions (from SAS labels), missing data counts, and summary statistics.

```python
from ncdb_tools import DataDictionaryGenerator

gen = DataDictionaryGenerator()
gen.generate_from_parquet(
    parquet_dir="/path/to/dataset",
    output_file="data_dictionary.html",
    format="html",
)
```

Available formats: **CSV**, **JSON**, **HTML** (with color coding and interactive features).

## Configuration

```bash
# Environment variables
export NCDB_DATA_DIR="/path/to/ncdb/data"
export NCDB_OUTPUT_DIR="/path/to/output"
export NCDB_MEMORY_LIMIT="8GB"
```

Or use a `.env` file (install with `pip install ncdb-tools[config]`).

## Memory & Performance

```python
# Automatic memory detection
mem = ncdb_tools.get_memory_info()
print(f"Available: {mem['available']}")

# Explore before committing to full collection
info = query.describe()
print(f"{info['total_rows']:,} rows, {info['columns']} columns")

# Sample for exploration
sample = query.sample(n=10000)
```

## Requirements

- Python >= 3.10
- NCDB participant user files (PUF) in fixed-width `.dat` format
- SAS labels file (optional, for variable descriptions in data dictionary)

## License

MIT

## Disclaimer

This package is not affiliated with or endorsed by the American College of Surgeons or the National Cancer Database. Users must obtain data through official channels and comply with all applicable data use agreements.
