Metadata-Version: 2.4
Name: battery-data-steward
Version: 0.4.0
Summary: Evidence-aware battery data inspection, quality assurance, and preprocessing.
Author: Battery Data Steward contributors
License-Expression: MIT
Project-URL: Repository, https://github.com/shiyunliu-battery/Battery-Data-Steward
Project-URL: Issues, https://github.com/shiyunliu-battery/Battery-Data-Steward/issues
Project-URL: Changelog, https://github.com/shiyunliu-battery/Battery-Data-Steward/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/shiyunliu-battery/Battery-Data-Steward#readme
Keywords: battery,cycler,data-conversion,data-quality,preprocessing,provenance,eis,neware,arbin
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: polars>=0.20
Requires-Dist: pandas>=2.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: xlrd>=2.0
Requires-Dist: reportlab>=4.0
Provides-Extra: yaml
Requires-Dist: PyYAML>=6.0; extra == "yaml"
Provides-Extra: matlab
Requires-Dist: scipy>=1.10; extra == "matlab"
Requires-Dist: h5py>=3.8; extra == "matlab"
Provides-Extra: mpr
Requires-Dist: galvani>=0.5; extra == "mpr"
Provides-Extra: report
Requires-Dist: reportlab>=4.0; extra == "report"
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-cov>=5.0; extra == "test"
Requires-Dist: PyYAML>=6.0; extra == "test"
Requires-Dist: scipy>=1.10; extra == "test"
Requires-Dist: h5py>=3.8; extra == "test"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: PyYAML>=6.0; extra == "dev"
Requires-Dist: scipy>=1.10; extra == "dev"
Requires-Dist: h5py>=3.8; extra == "dev"
Requires-Dist: galvani>=0.5; extra == "dev"
Requires-Dist: reportlab>=4.0; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Dynamic: license-file

# Battery Data Steward

**Inspect. Validate. Trace. Reuse.**

Evidence-aware battery data preprocessing and quality assurance.

[![Version 0.4.0](https://img.shields.io/badge/version-0.4.0-blue.svg)](CHANGELOG.md)
[![Python >=3.10](https://img.shields.io/badge/Python-%3E%3D3.10-blue.svg)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Package](https://img.shields.io/badge/package-battery--data--steward-blue.svg)](pyproject.toml)

Battery Data Steward (BDS) is an open-source Python toolkit that helps researchers
inspect raw battery data, identify quality risks, track preprocessing decisions,
and prepare reviewable datasets for downstream analysis.

BDS detects supported formats, explains column and unit mappings, identifies
time-axis and current-sign risks, reports unsupported fields, and converts
supported time-series and EIS data to CSV or Parquet with review reports.

```bash
bds inspect raw_export.csv
```

Inspection is local and non-mutating. It does not modify the source file,
insert missing samples, or silently change current values.

The package is vendor-neutral and independent. It is not certified by any cycler
vendor or standards body. Adapter support describes behavior implemented and
validated by this project; users should verify representative exports from their
own cycler software before using the package in automated production workflows.

## Installation

Python 3.10 or newer is required. The renamed distribution is
`battery-data-steward`, starting with the planned 0.4.0 release. The PyPI command
below becomes available after that release is published. Until then, install
this checkout with `python -m pip install .`.

When migrating an environment that has the old distribution installed, first run
`python -m pip uninstall battery-data-standard`. Both distributions provide the
same import paths and CLI, so they should not be installed together.

```bash
pip install --upgrade battery-data-steward
```

Optional extras are available for additional input formats:

```bash
pip install "battery-data-steward[yaml]"
pip install "battery-data-steward[matlab]"
pip install "battery-data-steward[mpr]"
```

The distribution installs the `bds` command. Use the public short import;
the historical module path remains compatible:

```python
import bds

# Existing code remains supported.
import battery_data_standard as bds
```

## Features

The package can:

- inspect unfamiliar battery cycler files before conversion;
- detect supported cycler and data formats;
- explain column, unit, and current-sign mappings;
- report missing fields, sampling gaps, and unmapped columns;
- provide suggested next steps when a file cannot be processed cleanly;
- convert supported time-series and EIS data to CSV or Parquet;
- generate JSON, HTML, Excel, and PDF reports;
- process directories, zip archives, and tar archives locally;
- export staging data for PyProBE, PyBaMM, cellpy, BEEP, DuckDB, and
  Polars workflows;
- use optional column-mapping profiles for lab-specific headers.

The package does not upload source data to an external service. It reads local
files and writes local outputs.

## Quick Start

Inspect the installed version:

```bash
bds --version
```

Inspect an unfamiliar file:

```bash
bds inspect raw_export.csv
```

Preview mappings for a downstream target:

```bash
bds inspect raw_export.csv --target pybamm --format json
bds inspect raw_export.csv --target pyprobe
```

Write review reports:

```bash
bds inspect raw_export.csv --output inspect.json --output inspect.html
```

Detect only the likely cycler format:

```bash
bds detect raw_export.csv
```

Convert a time-series file:

```bash
bds convert raw_export.csv normalized.bds.csv --cycler auto --report auto
```

`--report auto` writes `normalized.bds.report.json` and
`normalized.bds.report.pdf` next to the converted CSV. Add
`--report-format html` or `--report-format xlsx` when those review formats are
needed.

Export directly to a downstream staging format:

```bash
bds convert raw_export.csv pybamm_drive_cycle.csv --target pybamm
bds convert raw_export.csv pyprobe_staging.parquet --target pyprobe --format parquet
```

Validate a converted file:

```bash
bds validate normalized.bds.csv
```

Convert a directory or archive and write a JSONL manifest:

```bash
bds batch raw_exports normalized_exports --recursive --manifest manifest.jsonl
bds batch raw_exports.zip normalized_exports --manifest manifest.jsonl
```

Audit a raw folder before committing to conversion:

```bash
bds audit raw_exports --recursive --json audit.json --html audit.html
```

The audit report scores each raw data file and highlights conversion failures,
missing required fields, unit conversions, time-axis repairs, current-sign
evidence, duplicate timestamps, non-monotonic time, suspicious flat
voltage/current, and cycle/step anomalies. Optional-column coverage is reported
separately so small but valid fixtures are not treated as low-quality data just
because they omit fields such as temperature or energy.

Inspect runtime adapter metadata and the pinned schema:

```bash
bds formats
bds inspect-schema
```

`bds formats` reports each adapter's support tier and evidence tier so users can
tell whether support is backed by public fixtures or unit tests.

## Python API

Read a supported time-series export into a Polars dataframe:

```python
import bds

df = bds.read("raw_export.csv", cycler="auto")
```

Show the user-facing export column names defined in the export template:

```python
import bds

export_df = bds.to_export_frame(df)
print(export_df.columns)
```

### Preserve Raw Current Sign And Repair Time Axis

For real experimental datasets, it is often useful to preserve the current sign
exactly as recorded by the source file, allow documented time-axis repairs, and
record any regular-sampling gaps:

```python
df = bds.read(
    path,
    cycler="auto",
    current_sign="preserve",
    repair_policy="repair",
    time_sampling_policy="warn",
)
```

Use `current_sign="preserve"` when downstream analysis should keep the raw
charge/discharge sign convention from the instrument. Use
`repair_policy="repair"` when the pipeline accepts documented normalizations
such as shifting elapsed test time to start at zero or sorting non-monotonic time
values.

When a fixed sampling interval is detected, BDS reports missing time points by
default (`time_sampling_policy="warn"`) without inserting rows. Explicitly set
`time_sampling_policy="repair"` to insert samples with
`time_sampling_interpolation="linear"`. Repairs skip gaps across known cycle or
step boundaries, unknown segment identifiers, and step-time resets. If segment
fields are absent, BDS cannot establish protocol boundaries: review the protocol
before enabling interpolation. Set `time_sampling_interval_s` when the cadence
is known.

Use an explicit cycler when the source format is known:

```python
df = bds.read("arbin_export.csv", cycler="arbin")
```

Convert a file and keep the conversion report:

```python
report = bds.convert(
    "raw_export.csv",
    "normalized.bds.csv",
    cycler="auto",
    report_path="auto",
)
```

This writes the converted CSV plus JSON and PDF reports. HTML and Excel reports
can be added with `report_formats=("html", "xlsx")`.

Write a downstream staging table by selecting an export target:

```python
bds.convert("raw_export.csv", "pybamm_drive_cycle.csv", target="pybamm")
bds.convert("raw_export.csv", "cellpy_staging.csv", target="cellpy")
```

Read data and report information in memory:

```python
df, report = bds.read_with_report("raw_export.csv", cycler="auto", strict=False)
```

Create step-level or cycle-level summaries from a normalized dataframe:

```python
steps = bds.summarize_steps(df)
cycles = bds.summarize_cycles(df, capacity_reset="cycle")
```

## Internal Normalized Model and Export Targets

The converter uses an internal BDS time-series table as an implementation
contract for parsing and export. It is not presented as a universal
scientific-data quality guarantee. Every successful default export contains
three required fields:

| Field | Unit | Description |
| --- | --- | --- |
| `Test Time (s)` | s | Elapsed time from the start of the test. |
| `Voltage (V)` | V | Measured cell or channel voltage. |
| `Current (A)` | A | Measured current. |

Additional fields, such as cycle number, step number, capacity, energy,
temperature, power, and internal resistance, are included when they are available
in the source file.

The default BDS CSV and Parquet exports use user-facing labels with units in
parentheses. Lower-level adapter data may use internal labels; prefer
`bds convert` or `to_export_frame(..., target="bds")` for public handoff.

## Supported Format Families

The package includes adapters for NEWARE, Arbin, Maccor, BioLogic, Repower, PEC,
Novonix, BaSyTec, LANDT, and generic tabular exports. Generic readers support
delimited text, Excel, MATLAB, and Parquet inputs where the file contains or can
be mapped to time, voltage, and current columns.

BioLogic `.mpt` text exports are supported by default. Binary BioLogic `.mpr`
files are supported through the optional `mpr` extra, which installs the
`galvani` backend.

See [docs/supported-formats.md](docs/supported-formats.md) for adapter scope and
support-tier definitions.

## EIS Data

EIS files use a separate standardized table from row-wise time-series data.
Use EIS-specific commands or API functions for known impedance files:

```bash
bds detect-kind impedance.csv
bds convert-eis impedance.csv normalized.eis.csv
```

```python
eis = bds.read_eis("impedance.csv")
report = bds.convert_eis("impedance.csv", "normalized.eis.csv")
```

EIS conversion skips entirely empty rows and exact repeated header rows, records
cleanup notices, and rejects nonempty rows with invalid numeric data or
non-positive frequency. Errors identify a parsed table row, not a physical file
line. Explicit `deg` or `rad` phase labels determine the unit. Unitless `Zphz`,
`Zphase`, and `Phase` headers assume degrees with a warning; use
`read_eis(path, phase_unit="rad")` or `bds convert-eis ... --phase-unit rad` to
override that assumption. Conversion reports retain these notices.

`read()` and `convert()` are time-series entry points. `batch` and
`batch_convert()` can route mixed directories and archives that contain
time-series files, EIS files, including Gamry `.DTA` ZCURVE files, and
unsupported helper files.

## Column-Mapping Profiles

Profiles map lab-specific column names to canonical column names. JSON profiles
are supported by the base installation. YAML profiles require the `yaml` extra.

```json
{
  "columns": {
    "test_time": "time_seconds",
    "voltage": "cell_voltage",
    "current": "cell_current"
  }
}
```

Use a profile from the CLI:

```bash
bds convert lab_export.csv normalized.bds.csv --cycler generic --profile profile.json
```

## Intake Ambiguities and Accumulator Summaries

Dotted dates such as `01.02.2023` are parsed day-first. Slash-separated dates
retain month-first parsing with a warning; set `"date_order": "dayfirst"` or
`"monthfirst"` in a mapping profile. Profiles also accept an explicit `encoding`
when automatic legacy encoding detection is uncertain. Ambiguous strings such
as `"1,000"` are rejected rather than guessing whether a comma is a decimal or
thousands separator. Set profile `"decimal_separator": ","` for decimal commas
or `"decimal_separator": "."` for decimal points and comma thousands separators.

Headerless wide files require explicit column mapping. Multiple Arbin channel
sheets require an explicit `sheet` or `batch --excel-sheets all`; independent
channels are never silently concatenated. Sheet-level batch failures retain
successful records and processing continues unless `fail_fast=True`.

Undirected capacity/energy values are preserved as `source_capacity_ah` or
`source_energy_wh`. Directional fields are derived only from recognizable
charge/discharge status. Reports carry `accumulator_reset_scope`, defaulting to
`unknown`; a profile can declare the scope after protocol review.

`summarize_cycles(df)` warns when reset scope is unknown and labels maxima as
recorded maxima, not transferred totals. Use `capacity_reset="cycle"` for
cycle-reset accumulators or `capacity_reset="step"` to sum maxima from consecutive
step segments. These modes require complete, zero-based accumulator coverage;
truncated measurements and test-wide accumulators need a separate baseline-aware
calculation. BDS does not invent cycle identifiers or integrate missing capacity.

## Current Sign Convention

The default current convention for reading and BDS exports is charge-positive
and discharge-negative. `convert(..., target="pybamm")` and batch PyBaMM exports
instead default to discharge-positive. An explicit `current_sign` overrides the
target default. Source power values are preserved; if current normalization
changes their corresponding current values, reports warn that the source power
sign convention requires review.

Use `--current-sign preserve` to retain the source sign convention, or
`--current-sign discharge-positive` when a downstream workflow requires
discharge-positive current. When a source file contains a recognizable
charge/discharge status column, adapters use it to normalize current sign more
explicitly.

BDS can record a warning-only adjacent-point current-sign sanity check in
conversion metadata and audit records when `current_sign` is charge-positive or
discharge-positive. Use `--current-sign-check adjacent` to enable this O(n)
heuristic for files that need extra sign review.

## Validation and Reports

Every conversion returns a machine-readable report with schema version, row
count, columns, validation status, warnings, provenance, adapter metadata, repair
operations, and time-sampling findings. With `report_path="auto"`, BDS writes
JSON and PDF reports by default; HTML and Excel reports are optional formats.

Strict validation is enabled by default. Repairable issues are reported with the
default `repair_policy="warn"`. Use `repair_policy="repair"` or
`--repair-policy repair` only when the pipeline explicitly accepts the documented
normalizations.

## Research Scope and Roadmap

Today, BDS provides data inspection, table validation, diagnostics, conversion,
and reports containing available provenance, parameters, warnings, and recorded
repair operations. These outputs help researchers review data and processing;
a successful conversion or validation does not establish that a dataset can
support SOH, RUL, EIS fitting, or a specific scientific conclusion.

The planned **Battery Data Quality & Readiness Package** would add task-specific
readiness criteria, explicit unresolved evidence, and review decisions. These
readiness gates and a full transformation evidence workflow are future work,
not current capabilities. See [project direction](docs/project-direction.md).

BDS focuses on input quality and preprocessing review. Battery-Feature-Lab
focuses on feature extraction and diagnostics; Battery Context focuses on
evidence packages and scientific sufficiency. These are complementary project
boundaries, not bundled integrations or guarantees of scientific validity.

## Documentation

Public documentation is in the `docs` directory:

- [Package migration and publishing](docs/releasing.md)
- [Project direction and roadmap](docs/project-direction.md)
- [Python API reference](docs/api-reference.md)
- [Supported formats](docs/supported-formats.md)
- [Export template](docs/export-template.md)
- [Export targets](docs/export-template.md#export-targets)
- [Schema compatibility](docs/schema-compatibility.md)
- [Diagnostics and audit reports](docs/diagnostics.md)
- [Current sign convention](docs/current-sign.md)
- [Step and cycle semantics](docs/step-cycle-semantics.md)
- [Warning and issue codes](docs/warning-codes.md)
- [Fixture contribution guide](docs/fixture-contribution.md)
- [Ecosystem integrations](docs/integrations.md)

## License and Attribution

This project is distributed under the MIT License. See [NOTICE.md](NOTICE.md)
for the project independence notice.
