Metadata-Version: 2.4
Name: demultiplexer3
Version: 1.0.0
Summary: A python command line interface to demultiplex illumina reads by inline tags.
Home-page: https://github.com/DominikBuchner/demultiplexer3
Author: Dominik Buchner
Author-email: dominik.buchner524@gmail.com
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: biopython>=1.84
Requires-Dist: pandas>=2.2.3
Requires-Dist: openpyxl>=3.1.1
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: duckdb>=0.10.0
Requires-Dist: psutil>=5.7.3
Requires-Dist: tqdm>=4.66.4
Requires-Dist: isal>=1.0.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# demultiplexer3

A Python CLI tool for demultiplexing paired-end Illumina sequencing reads using inline tags. Successor to [demultiplexer2](https://github.com/DominikBuchner/demultiplexer2).

## Key improvements over demultiplexer2

- **Output-oriented parallelism** â€” workers write individual samples as soon as parsing is done, instead of blocking on the largest input file.
- **Cross-product tag validation** â€” forward and reverse tags are checked jointly (concatenated Levenshtein distance), eliminating the infinite-loop risk of separate checks.
- **Configurable minimum distance** (`--min_dist`) and optional **tag trimming** (`--trim`).
- **DAG-based scheduler** using `concurrent.futures.ProcessPoolExecutor` with dynamic job submission.
- **Parquet intermediate format** with sample identity classified once at parse time, for fast columnar filtering downstream.

## Installation

```bash
pip install demultiplexer3
```

### Dependencies

- biopython
- pandas
- openpyxl
- pyarrow
- duckdb
- psutil
- tqdm
- isal (faster gzip via Intel ISA-L; falls back to stdlib `gzip` if unavailable)

## Usage

### 1. Create a primerset

```bash
demultiplexer3 create_primerset --name <name> --n_primers <n>
```

Creates an Excel file with three sheets (`general_information`, `forward_tags`, `reverse_tags`) for the user to fill in primer sequences and tag names/sequences. IUPAC ambiguity codes are supported. Saved as `<name>_primerset.xlsx` in the current working directory (same as the tagging scheme below).

### 2. Create a tagging scheme

```bash
demultiplexer3 create_tagging_scheme --name <name> --data_dir <dir> --primerset_path <path>
```

Scans `data_dir` for `.fastq.gz` file pairs, loads the primerset, and interactively asks which tag combinations were used. Produces a tagging scheme Excel file where sample names are filled in by the user.

### 3. Demultiplex

```bash
demultiplexer3 demultiplex --primerset_path <path> --tagging_scheme_path <path> --output_dir <dir> [--workers <n>] [--min_dist <d>] [--trim]
```

| Parameter | Description | Default |
|---|---|---|
| `--primerset_path` | Path to the primerset Excel file | required |
| `--tagging_scheme_path` | Path to the tagging scheme Excel file | required |
| `--output_dir` | Directory for output FASTQ files | required |
| `--workers` | Number of parallel workers | physical cores âˆ’ 2 |
| `--min_dist` | Minimum Levenshtein distance between tag combinations | 2 |
| `--trim` | Remove original tag sequences from the start of each read | off |

## How it works

1. **Tag validation & extension** â€” Tags are equalized to the same length, IUPAC codes are expanded, and the cross-product of all (fwd+rev) combinations is checked for minimum Levenshtein distance. If the distance is too low, tags are extended base-by-base from the primer sequence.

2. **Parse phase** â€” Each input FASTQ pair is read with BioPython's `FastqGeneralIterator`. The first N bases of each read are extracted as the tag and classified against that input's known tag combinations right away, storing the result as an integer sample index. All data is written to an lz4-compressed Parquet file (a purely transient intermediate â€” compression here favors speed over ratio).

3. **Write phase** â€” For each sample, DuckDB filters the Parquet file on that integer index and writes matching reads in batches to gzipped FASTQ output files.

4. **Cleanup** â€” Parquet files are deleted once all downstream write jobs for them complete. Unmatched reads are aggregated into `unmatched_logfile.xlsx`, keeping only the highest-count combinations per input file (the long tail of near-random spacer variants carries no diagnostic value).

## License

MIT
