Metadata-Version: 2.4
Name: cli-dataset-inspector
Version: 0.1.0
Summary: Inspect tabular datasets, report data quality issues, and generate EDA notebooks
Author: Erik
License-Expression: MIT
Project-URL: Repository, https://github.com/KishlakEnjoyer/cli-dataset-inspector
Project-URL: Issues, https://github.com/KishlakEnjoyer/cli-dataset-inspector/issues
Keywords: dataset,eda,data-quality,jupyter,cli
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.2
Requires-Dist: numpy>=1.26
Requires-Dist: typer>=0.16
Requires-Dist: rich>=13.7
Requires-Dist: nbformat>=5.10
Requires-Dist: matplotlib>=3.8
Requires-Dist: scikit-learn>=1.4
Requires-Dist: openpyxl>=3.1
Requires-Dist: xlrd>=2.0.1
Requires-Dist: pyarrow>=14
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: nbclient>=0.10; extra == "dev"
Requires-Dist: ipykernel>=6.29; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Requires-Dist: ruff>=0.11; extra == "dev"
Dynamic: license-file

# CLI Dataset Inspector

**English** | [Русский](docs/README.ru.md)

Inspect a tabular dataset, find data quality issues, and generate a self-contained
Jupyter Notebook with exploratory analysis, preprocessing and an optional baseline model.

## Installation

Requires Python 3.10 or newer. Before the first PyPI release, install from a local checkout:

```bash
python -m pip install .
```

For development:

```bash
python -m venv .venv
# Windows PowerShell: .venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
python -m pip install -e ".[dev]"
```

After publication, installation will be: `python -m pip install cli-dataset-inspector`.
All supported file readers and baseline dependencies are installed with the package.
To open notebooks, use a Jupyter-capable editor or install JupyterLab separately.

## Quick start

```bash
dataset-inspect examples/customers.csv
dataset-inspect examples/customers.csv --target churn
dataset-inspect examples/customers.csv --json
dataset-inspect examples/customers.csv --report-output reports/customers.json
dataset-inspect examples/customers.csv --jupyter --output reports/eda.ipynb
dataset-inspect examples/customers.csv --target churn --baseline --output reports/model.ipynb
```

`python -m dataset_inspector` is equivalent to `dataset-inspect`.

**The CLI never trains a model.** `--baseline` implies `--jupyter` and requires
`--target`. It adds executable preprocessing, training and evaluation cells to
the notebook. Open the notebook and run its cells to train the model.

## Input formats

| Format | Extensions | Options |
| --- | --- | --- |
| CSV / TSV | `.csv`, `.tsv` | `--sep`, `--encoding`; comma for CSV, tab for TSV |
| Excel | `.xlsx`, `.xlsm`, `.xls` | `--sheet` (name or zero-based index; default 0) |
| JSON | `.json` | Flat records or pandas-compatible columns orientation |
| JSON Lines | `.jsonl`, `.ndjson` | One flat JSON object per line |
| Parquet | `.parquet`, `.pq` | Tabular columns |

CSV/TSV and JSON also support pandas-inferred compression such as `.csv.gz`
and `.jsonl.gz`. Use `--format csv|excel|json|jsonl|parquet` to override extension detection.

```bash
dataset-inspect data.csv --sep ";" --encoding cp1251
dataset-inspect book.xlsx --sheet "Sales"
dataset-inspect book.xlsx --sheet 1
dataset-inspect book.xlsx --sheet name:0
dataset-inspect records.jsonl --json
dataset-inspect data.parquet --target price --task regression --baseline
dataset-inspect data.txt --format csv
```

`--sheet name:0` selects a sheet literally named `0`; `--sheet 0` selects the
first sheet. Input paths are local files. Nested JSON/Parquet values must be
flattened first. Column names are normalized to strings and must remain unique.

## Reports and warnings

The console and JSON report include:

- Rows, columns, memory usage in bytes, duplicate count and percentage.
- Per-column dtype, filled/missing counts, missing percentage, unique count and percentage.
- Numeric min, max, mean, median, standard deviation and outlier count using the 1.5 × IQR rule.
- Category counts, most frequent values and the top three values.
- Optional target summary, detected task and class distribution or regression statistics.
- Data quality warnings.

Warnings cover empty datasets/columns, missing values at or above 30%, constant
columns, possible IDs (at least 95% unique and more than 20 distinct values),
mixed or suspicious numeric/text values, infinite numeric values, duplicate rows,
and class imbalance (smallest/largest class count below 0.25).

These are heuristics, not proof of a problem. Numeric summaries exclude infinity.
Undefined numeric statistics are `null` in JSON and `n/a` in the console.
Outliers are reported in numeric summaries, not automatically removed.

### JSON output

`--json` writes one strict JSON document to stdout with no tables or status text.
Status/error messages use stderr. `--report-output PATH` saves the same report
as UTF-8 JSON and can be combined with either console or JSON output.

```bash
dataset-inspect examples/customers.csv --json --target churn --report-output reports/result.json
```

The report has `schema_version: "1.0"` and these top-level keys:
`shape`, `duplicates`, `missing`, `memory`, `columns_summary`, `numeric_summary`,
`categorical_summary`, `target`, `warnings`. `target` is null when unspecified.
It contains no trained model or metrics; training only happens in the notebook.

## Notebook and baseline

Generated notebooks include data loading, general statistics, missing values,
duplicates, a generation-time warning snapshot, histograms, category frequencies,
correlations, and optional target analysis.

The load cell preserves the format, separator, encoding and sheet settings.
The notebook reads the original file through an editable absolute `DATASET_PATH`;
it does not embed the dataset and runs without the `dataset_inspector` package.
It needs pandas, numpy, matplotlib, scikit-learn, IPython/Jupyter and the relevant reader library.

Without `--task`, nonnumeric/bool targets and numeric targets with at most
20 unique nonmissing values are classified; other numeric targets use regression.
Use `--task classification` or `--task regression` to override this heuristic
consistently in the report and notebook. Regression requires a numeric target.

Baseline behavior:

- Exclude missing/infinite targets; treat infinite numeric features as missing.
- Numeric features: median imputation and standard scaling.
- Categorical features: mode imputation and one-hot encoding that tolerates unseen values.
- Dates and other nonnumeric features are treated as categories; review them for your domain.
- All learned transformations fit on training data only.
- 80/20 train/test split, seed 42; classification uses stratification.
- LogisticRegression: accuracy, macro F1 and a precision/recall/F1 classification report.
- LinearRegression: MAE, RMSE and R².
- If rows/classes/features cannot support a valid split, the notebook skips training and explains why.

Change the split with `--test-size 0.3 --random-state 7`, or edit the configuration
cell. After execution, `baseline_result` contains status and metrics (or a skip
reason), and `model` is the fitted scikit-learn pipeline when training succeeds.
The CLI itself does not execute notebooks or save trained model files.

Plots and correlation matrices are limited to the first 20 relevant columns.
Edit `MAX_PLOTS` in the notebook to change this. Inspection loads the full dataset
into memory. Very large datasets and high-cardinality categoricals can be expensive.
Random holdout evaluation assumes independent rows: review duplicates, identifiers,
time/group structure and leakage before interpreting metrics.

## Output paths and errors

- `--output PATH` / `-op PATH`: notebook `.ipynb` file or directory; needs `--jupyter` or `--baseline`.
- Default notebook path: `<dataset_stem>_analysis.ipynb` in the current directory.
- `--report-output PATH`: exact JSON report file path.
- Existing outputs require `--overwrite`; an input dataset can never be an output.
- Output parents are created as needed; each file is written atomically.
- There is no transaction across multiple output files: a later write failure can
  leave an earlier completed artifact in place.
- Exit codes: 0 success, 1 input/read/write failure, 2 invalid CLI options.
- Header-only tables can be inspected. Zero-byte files and tables with no columns are rejected.
- `--help` and `--version` / `-v` work without a dataset path.

## Python API

```python
from dataset_inspector.inspector import Inspector
from dataset_inspector.notebook import create_notebook

report = Inspector("data.csv", target="label").inspect()
create_notebook(
    "data.csv", target="label", data=report,
    baseline=True, output="analysis.ipynb",
)
```

The API returns ordinary JSON-compatible Python values. Reader settings are keyword
arguments to `Inspector` and passed as `load_options={...}` to `create_notebook`.

## Development and release

```bash
python -m pytest
python -m ruff check src tests
python -m ruff format --check src tests
python -m build
python -m twine check dist/*
```

Tests exercise real input formats, error handling, output protection and generated
notebook execution, including a real Jupyter kernel. The CI configuration tests
Python 3.10–3.14 on Linux and Python 3.14 on Windows, and builds/checks the distributions.
See [the release checklist](docs/RELEASING.md) before publishing.

MIT license.
