Metadata-Version: 2.4
Name: bclconvert
Version: 0.1.0
Summary: Illumina BCL-Convert SampleSheet v2 generation/parsing and 10x barcode-kit loading
Author-email: Caleb Kibet <ckibet@iavi.org>
License: MIT
Keywords: bioinformatics,illumina,bcl-convert,samplesheet,10x
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Requires-Dist: polars>=1.0
Provides-Extra: excel
Requires-Dist: pandas>=1.5; extra == "excel"
Requires-Dist: openpyxl>=3.0; extra == "excel"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pytest-mock>=3.10; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0; extra == "dev"
Requires-Dist: psutil>=5.9; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Dynamic: license-file

# BCL Convert Pipeline

A toolkit for Illumina BCL-Convert sample-sheet generation, demultiplexing,
and 10x single-cell Excel-to-samplesheet transforms.

## The unified `bclconvert` tool

The canonical, standalone tool lives in **`src/bclconvert/`** — a single
installable package with a `bclconvert` CLI that unifies sample-sheet
generation, demultiplexing, and the single-cell Excel transform. It is
polars-based and was consolidated from `bcl2airr` plus the legacy scripts in
this repo. `bcl2airr` depends on it via thin re-export shims (see
[`docs/bcl2airr-shim-migration.md`](docs/bcl2airr-shim-migration.md)).

```bash
pip install -e .            # core (polars + click)
pip install -e '.[excel]'   # + pandas/openpyxl for the `transform` command
```

## CLI reference

All commands support a global `-v/--verbose` flag (must come before the
subcommand, e.g. `bclconvert -v make-sheet ...`) for debug logging.

### `make-sheet` — generate a SampleSheet v2

```bash
# All-barcodes mode: emit every barcode in the chosen kit(s) — no sample list
bclconvert make-sheet --run-dir RUN/ --barcode-type dual --kit all -o SampleSheet.csv

# Known-samples mode: look up i7/i5 for a Lane,Sample_ID,Index list
bclconvert make-sheet --run-dir RUN/ --samples samples.csv \
    --barcode-type dual --kit TT,TN -o SampleSheet.csv

# Custom (non-bundled) index file, forward-strand i5
bclconvert make-sheet --barcode-type dual --index-file my_index.csv \
    --i5-workflow A --runinfo RUN/RunInfo.xml -o SampleSheet.csv
```

| Option | Description |
|---|---|
| `--barcode-type {dual,single}` | Required. Barcode kit type used for the run. |
| `--kit CODES` | Bundled kit code(s), comma-separated (`TT,TN`) or `all`. Defaults to `all` unless `--index-file` is given. |
| `--index-file PATH` | Custom Illumina-format index CSV (repeatable). Used in addition to `--kit`. |
| `--i5-workflow {A,B}` | i5 orientation. Default `B` (reverse complement — correct for NextSeq 550/1k/2k and NovaSeq 6000). `A` is forward strand. |
| `--samples PATH` | Known-samples CSV (`Lane,Sample_ID,Index`, or `Sample_ID,Index`). Omit for all-barcodes mode. `Lane` may be an integer or `*` (bcl-convert's native "all lanes" wildcard), and is passed through unchanged. |
| `--run-dir PATH` | Run folder containing `RunInfo.xml` and/or `RunParameters.xml`. |
| `--runinfo PATH` | Explicit `RunInfo.xml` path (overrides `--run-dir`). |
| `--runparameters PATH` | Explicit `RunParameters.xml` path (overrides `--run-dir`). |
| `--run-name NAME` | Override the `RunName` in `[Header]`. |
| `-o, --output PATH` | Required. Output `SampleSheet.csv` path. |

### `run` — execute `bcl-convert`

```bash
bclconvert run --bcl-dir RUN/ --sample-sheet SampleSheet.csv --output-dir out/ --threads 8
```
`--dry-run` prints the `bcl-convert` command without executing it.

### `pipeline` — `make-sheet` + `run` in one step

```bash
bclconvert pipeline --bcl-dir RUN/ --barcode-type dual --kit all --output-dir out/
```
Takes the same barcode/kit/i5 options as `make-sheet`, plus `--sample-sheet-name`
(default `SampleSheet.csv`), `--threads`, and `--dry-run`.

### `transform` — single-cell Excel → sample CSV

Needs the `[excel]` extra (`pip install -e '.[excel]'`).

```bash
bclconvert transform -e plate.xlsx -f /path/to/fastq -o samples.csv
```

### `validate` — check a SampleSheet v2

```bash
bclconvert validate SampleSheet.csv
```
Exits non-zero and prints `INVALID: <path>` if required sections or data rows
are missing; prints `OK: <path>` otherwise.

## File formats

### Custom index CSV (`--index-file`)

Dual-index kits use the Illumina 10x format:

```csv
index_name,index(i7),index2_workflow_a(i5),index2_workflow_b(i5)
SI-TT-A1,GTAACATGCG,AGTGTTACCT,AGGTAACACT
```

### Known-samples CSV (`--samples`)

```csv
Lane,Sample_ID,Index
1,Sample1,SI-TT-A1
*,Sample2,SI-TT-A2
```
`Sample_ID`/`Index` (case-insensitive) are matched by header name; `Lane` is
optional. Headerless files fall back to positional `[Lane,] Sample_ID, Index`
order. `Index` must match an `index_name` from the loaded kit(s)/index file(s).

### Output SampleSheet v2 structure

```
[Header]
FileFormatVersion,2
RunName,<run_id>

[Reads]
Read1Cycles,<n>
Read2Cycles,<n>

[BCLConvert_Settings]
CreateFastqForIndexReads,1
MinimumTrimmedReadLength,8
FastqCompressionFormat,gzip
MaskShortReads,8

[BCLConvert_Data]
Lane,Sample_ID,Index,Index2   # Lane column omitted in all-barcodes mode
...
```

## Python API

Everything the CLI does is backed by a public library API exported from
`bclconvert/__init__.py` — not just internal glue for `cli.py`. `pip install
-e .` gets you both.

```python
from pathlib import Path
import bclconvert

# Load barcodes for bundled kit(s) (polars DataFrame: index_name, i7, i5)
barcodes = bclconvert.load_index("dual", kits=["TT", "TN"])

# Resolve run metadata from RunInfo.xml/RunParameters.xml
meta = bclconvert.resolve_run_metadata(run_dir="RUN/")

# Build a SampleSheetV2 in-memory (all-barcodes or known-samples mode)
sheet = bclconvert.build_samplesheet(barcodes, run_meta=meta, samples="samples.csv")
sheet.sample_count      # int
sheet.to_csv()          # str, the full SampleSheet.csv content
sheet.write("out.csv")  # str or Path

# Parse an existing sample sheet back into the same model.
# parse_samplesheet_v2 reads from disk given a Path; a str is parsed as literal
# CSV text. Passing a path as a str raises ValueError rather than failing quietly.
parsed = bclconvert.parse_samplesheet_v2(Path("SampleSheet.csv"))
parsed = bclconvert.parse_samplesheet_v2(sheet.to_csv())  # str = CSV text
bclconvert.validate_sample_sheet("SampleSheet.csv")  # bool; str or Path both fine here

# Run bcl-convert itself (thin subprocess wrapper, returns exit code)
bclconvert.run_bcl_convert(bcl_dir="RUN/", sample_sheet="SampleSheet.csv", threads=8)
```

Exported surface (`bclconvert.__all__`): kit constants (`AVAILABLE_KITS`,
`KIT_CODES`, `DUAL_INDEX_KITS`, `SINGLE_INDEX_KITS`, `NORMALIZED_COLUMNS`),
`load_barcode_kit`/`load_index`/`load_all_dual_index_barcodes`/`resolve_kit_names`,
`RunMetadata`/`parse_runinfo`/`parse_runparameters`/`resolve_run_metadata`,
`build_samplesheet`, `SampleSheetV2`/`parse_samplesheet_v2`/`validate_sample_sheet`,
`barcodes_to_samples`/`samples_to_barcodes`, `run_bcl_convert`. This is the
same surface `bcl2airr` consumes via thin re-export shims (see
[`docs/bcl2airr-shim-migration.md`](docs/bcl2airr-shim-migration.md)), so
it's designed for cross-package use, not CLI-only.

## Legacy scripts — verification baseline

The original pandas scripts, retained for cross-checking the unified tool's
output during the verification window, now live under
[`src/bclconvert/legacy/`](src/bclconvert/legacy/README.md) — see that
directory's README for usage.

## Repository structure

```
bcl-convert/
├── src/bclconvert/             # Canonical, unified package + CLI
│   ├── cli.py                  #   bclconvert CLI: make-sheet/run/pipeline/transform/validate
│   ├── builder.py              #   build SampleSheetV2 (all-barcodes + known-samples)
│   ├── runinfo.py              #   RunInfo.xml / RunParameters.xml metadata parsing
│   ├── runner.py                #   bcl-convert execution wrapper
│   ├── transform.py            #   single-cell Excel → sample CSV (HTO→CMO)
│   ├── samplesheet.py          #   SampleSheetV2 model + v2 parser/writer
│   ├── legacy/                 #   Retained pandas scripts (verification baseline; see legacy/README.md)
│   └── barcodes/
│       ├── kits.py             #   10x barcode-kit loaders + kit selection (polars)
│       └── data/*.csv          #   5 bundled 10x kit CSVs
├── pyproject.toml              # Packaging; entry point: bclconvert = cli:main
├── docs/
│   └── bcl2airr-shim-migration.md  # Plan to repoint bcl2airr at bclconvert
├── tests/                      # unit/integration/performance test suites
├── Index/                      # Legacy index files (subset, duplicate of barcodes/data)
└── README.md                   # This documentation
```

## Requirements

- Python >= 3.9
- `click`, `polars` (core)
- `pandas`, `openpyxl` (only for `transform`, via the `[excel]` extra)
- `bcl-convert` (Illumina, for the `run`/`pipeline` commands)

## Troubleshooting

1. **Missing bcl-convert**: ensure the Illumina `bcl-convert` binary is
   installed and on `PATH` — `bclconvert run`/`pipeline` shell out to it.
2. **`FileNotFoundError` on kit loading**: check `--kit` codes match bundled
   kits (`TT`, `TN`, `TS` dual; `N`, `T` single) or that `--index-file` points
   at a valid Illumina-format CSV.
3. **`... index name(s) not found in the loaded barcode kit(s)`**: the
   `--samples` CSV references an `Index` name not present in the kit(s)/index
   file(s) you loaded — add the missing kit via `--kit`.
4. **Dry run**: add `--dry-run` to `run`/`pipeline` to print the `bcl-convert`
   command without executing it.
