Metadata-Version: 2.2
Name: nanosimformer
Version: 1.1.1
Summary: An end-to-end Transformer-based nanopore signal simulator with basecaller guidance
Home-page: https://github.com/BioinfoSZU/NanoSimFormer
Author: Shaohui Xie, Lulu Ding, Ling Liu, Zexuan Zhu
Author-email: 2060271006@email.szu.edu.cn
License: Apache-2.0
Project-URL: Bug Reports, https://github.com/BioinfoSZU/NanoSimFormer/issues
Project-URL: Source, https://github.com/BioinfoSZU/NanoSimFormer
Keywords: nanopore sequencing signal simulation transformer deep-learning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: torch==2.9
Requires-Dist: torchtune
Requires-Dist: torchaotqdm
Requires-Dist: pysam
Requires-Dist: pod5
Requires-Dist: h5py
Requires-Dist: toml
Requires-Dist: mappy
Requires-Dist: biopython
Requires-Dist: pandas
Requires-Dist: scipy
Requires-Dist: flask
Requires-Dist: ont-bonito
Requires-Dist: tensorboard
Requires-Dist: packaging
Requires-Dist: ont-remora
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

NanoSimFormer: An end-to-end Transformer-based nanopore signal simulator with basecaller guidance
-----

NanoSimFormer is a high-fidelity nanopore sequencing signal simulator built on Transformer architectures.
It supports both ONT DNA (R10.4.1) and direct-RNA (RNA004) signal simulation, enabling users to generate synthetic POD5 files from references or existing basecalled reads.

## 🚀 Download and Installation

### System Requirements

  * **GPU:** NVIDIA GPU with CUDA compute capability \>= 8.x (e.g., Ampere, Ada, or Hopper GPUs like A100, RTX 3090, RTX 4090, H100)
  * **Driver:** NVIDIA driver version \>= 525

NanoSimFormer is compatible with Linux and has been fully tested on Ubuntu 22.04.

### Installation via Docker 

We recommend installing NanoSimFormer using the pre-built Docker image. Ensure you have Docker and the NVIDIA Container Toolkit installed by 
following this [tutorial](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).

```shell 
docker pull chobits323/nano-sim:v1.1
```

### Install from conda and pip
Alternatively, you can install NanoSimFormer in a [conda](https://www.anaconda.com/download/success) environment.

```shell
conda create -n nanosim -c conda-forge python==3.10
conda activate nanosim
pip install nanosimformer -i https://pypi.org/simple --extra-index-url https://download.pytorch.org/whl/cu126

# install flash-attn
wget https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.4.17/flash_attn-2.7.4.post1+cu126torch2.9-cp310-cp310-linux_x86_64.whl
pip install flash_attn-2.7.4.post1+cu126torch2.9-cp310-cp310-linux_x86_64.whl
```

**Note:** The Docker image already contains all required pre-trained model files. For pip/conda installations, download the model checkpoints from [Zenodo](https://doi.org/10.5281/zenodo.20364515) and unzip them to the package install path, following the instructions below:

```shell
# Find the package install path
python -c "import nano_signal_simulator; print(nano_signal_simulator.__path__[0])"
# Example output: /home/xsh/anaconda3/envs/nanosim/lib/python3.10/site-packages/nano_signal_simulator

# Download and unzip model checkpoints
wget https://zenodo.org/records/20364515/files/models.zip
unzip models.zip -d $(python -c "import nano_signal_simulator; print(nano_signal_simulator.__path__[0])")
```

-----

## ⚙️ CLI Overview

NanoSimFormer provides a unified command-line interface with four subcommands:

```text
python -m nano_signal_simulator <command> [options]

Commands:
  simulate          Simulate nanopore signals from a reference genome/transcriptome or reads
  create_dataset    Create a training dataset from aligned BAM + POD5 files
  visualize         Launch an interactive web viewer for inspecting dataset chunks
  train             Train a NanoSimFormer model
```

Use `-h` with any subcommand for detailed help:
```shell
python -m nano_signal_simulator simulate -h
python -m nano_signal_simulator create_dataset -h
python -m nano_signal_simulator visualize -h
python -m nano_signal_simulator train -h
```

-----

## 🧬 Signal Simulation (`simulate`)

### Quick Start with Docker
```shell
# Define your working directory
EXAMPLE_DIR="[WORKING_DIRECTORY_OF_EXAMPLE_DATA]"  # absolute path

# Print help
docker run --rm -it --gpus=all -v ${EXAMPLE_DIR}:${EXAMPLE_DIR} --ipc=host \
  chobits323/nano-sim:latest python -m nano_signal_simulator simulate -h
```

### DNA Sequencing Simulation Examples (R10.4.1)

#### Reference-Based Simulation
Simulate reads from a reference genome (FASTA) given a specific read number or sequencing coverage.

```shell
# Simulate 1000 reads from the chromosome 22 reference.
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 1000 --gpu 0 --preset ont_r1041_dna_5khz 

# Simulate reads with 0.1x sequencing coverage from the chromosome 22 reference.
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --coverage 0.1 --gpu 0 --preset ont_r1041_dna_5khz
```

#### Adjusting Noise and Duration parameters
Adjust the standard deviation of the amplitude noise or duration samplers to generate simulated signals with varying qualities.

```shell 
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 1000 --gpu 0 --preset ont_r1041_dna_5khz \
  --noise-stdv 1.0 --duration-stdv 0.8
```

#### Circular Reference-Based Simulation
NanoSimFormer detects genome circularity via the FASTA header (`circular=true/false`) to allow simulated reads to seamlessly wrap around the end of the sequences.

*Example FASTA format*: 
```text 
>contig_1 circular=true
ATCG...
>contig_2 circular=false
CGAA...
```

```shell  
# Simulate 1000 reads from a circular E.coli reference genome (including plasmids) 
# utilizing a custom mean read length and an exponential read length distribution.
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/ecoli.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 1000 --gpu 0 --preset ont_r1041_dna_5khz \
  --mean-read-length 4394 --length-dist-mode expon 
```

#### Read-based simulation
Generate signals from basecalled reads provided in a FASTQ file (one-by-one). 

**Note**: Read IDs in FASTQ file must be in UUID format for POD5 compatibility.

```shell 
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/example.fastq \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Read --gpu 0 --preset ont_r1041_dna_5khz
```

#### Full Pipeline (Signal simulation + Basecalling) 
Use `--basecall` option to automatically run [Dorado](https://github.com/nanoporetech/dorado) that basecalling the reads into a FASTQ file after signal simulation.
Use `--emit-bam` together with `--basecall` to output basecalled results in BAM format instead of FASTQ.

```shell 
# Simulate 1000 reads from the chromosome 22 reference. 
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 1000 --gpu 0 --preset ont_r1041_dna_5khz --basecall 

# Output basecalled results in BAM format instead of FASTQ
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 1000 --gpu 0 --preset ont_r1041_dna_5khz --basecall --emit-bam
```

#### Multi-GPU Inference
Use `--multi-gpu` to automatically distribute reads across all available GPUs, or `--gpus` to specify exact GPU IDs. 

```shell
# Use all available GPUs
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 10000 --preset ont_r1041_dna_5khz --multi-gpu

# Use specific GPUs (e.g., GPU 0 and GPU 2)
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 10000 --preset ont_r1041_dna_5khz --gpus 0,2

# Multi-GPU with basecalling
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/DNA_R10.4.1/chr22.fasta \
  --output ${EXAMPLE_DIR}/DNA_R10.4.1/output \
  --mode Reference --sample-reads 10000 --preset ont_r1041_dna_5khz --multi-gpu --basecall
```
*Note: To accelerate the simulation process, you can increase the `--batch-size` parameter (default: 64) if your GPU has sufficient memory.*

### Direct-RNA Sequencing Simulation Examples (RNA004)
#### Transcriptome Reference-Based simulation
For Direct RNA sequencing (DRS), NanoSimFormer requires a 3-column TSV profile defining `transcript_name`, `trunc_start`, and `trunc_end` to simulate realistic transcript abundances and 5'/3' truncations. 
Each row in the profile TSV represents the metadata for a single simulated read.

*Example TSV Profile*: 

| transcript_name | trunc_start | trunc_end |
|-----------------|-------------|-----------|
| ENST000001      | 10          | 1200      |

```shell 
python -m nano_signal_simulator simulate \
  --input ${EXAMPLE_DIR}/RNA004/trans_ref.fasta \
  --trans-profile ${EXAMPLE_DIR}/RNA004/trans_profile.tsv \
  --output ${EXAMPLE_DIR}/RNA004/output \
  --mode Reference --gpu 0 --preset ont_rna004_4khz --basecall
```

-----

## 📊 Training Dataset Preparation (`create_dataset`)

This command creates training data from real nanopore sequencing data (aligned BAM + POD5 files). It extracts signal–sequence aligned chunks and outputs four `.npy` files.

### Prerequisites

1. **POD5 file** — raw signal data
2. **Aligned BAM file** — Dorado basecalled reads aligned to a reference using minimap2, must be indexed (`samtools index`) and contain the move-table (by Dorado `--emit-moves` option) as well as 'MD' tag (by minimap2 `--MD` option) 
3. **K-mer model table** — official ONT k-mer level table for signal refinement (download from [ONT](https://github.com/nanoporetech/kmer_models))

### Output Format

The dataset consists of four NumPy files in the output directory:

| File                  | Shape                    | Dtype     | Description                                              |
|-----------------------|--------------------------|-----------|----------------------------------------------------------|
| `signal.npy`          | `(N, signal_chunk_len)`  | `float32` | Z-normalized signal chunks                               |
| `sequence.npy`        | `(N, max_chunk_seq_len)` | `int8`    | Encoded bases (A=1, C=2, G=3, T=4, pad=0)                |
| `sequence_length.npy` | `(N,)`                   | `int32`   | Actual (unpadded) sequence length per chunk              |
| `dwells.npy`          | `(N, max_chunk_seq_len)` | `int32`   | Per-base dwell times (samples), sum = `signal_chunk_len` |

### Chemistry-Dependent Defaults

Parameters are automatically tuned based on sequencing speed:

| Chemistry   | Sample Rate | Seq Speed | Signal Chunk | Expected Bases | Min Seq Len | Max Seq Len |
|-------------|-------------|-----------|--------------|----------------|-------------|-------------|
| `dna_r1041` | 5000 Hz     | 400 bp/s  | 5000         | 400            | 300         | 500         |
| `rna004`    | 4000 Hz     | 130 bp/s  | 4000         | 130            | 97          | 162         |

### Usage Example

```shell
# DNA R10.4.1 dataset
python -m nano_signal_simulator create_dataset \
  --bam /path/to/aligned.bam \
  --pod5 /path/to/reads.pod5 \
  --kmer_model /path/to/dna_r10.4.1_e8.2_400bps/9mer_levels_v1.txt \
  --chemistry dna_r1041 \
  --output /path/to/output_dataset

# RNA004 dataset
python -m nano_signal_simulator create_dataset \
  --bam /path/to/rna_aligned.bam \
  --pod5 /path/to/rna_reads.pod5 \
  --kmer_model /path/to/rna004/9mer_levels_v1.txt \
  --chemistry rna004 \
  --output /path/to/rna_dataset

# Filter by chromosome 
python -m nano_signal_simulator create_dataset \
  --bam /path/to/aligned.bam \
  --pod5 /path/to/reads.pod5 \
  --kmer_model /path/to/9mer_levels_v1.txt \
  --chemistry dna_r1041 \
  --output /path/to/chr22_dataset \
  --chrom_ids chr1 chr2 
```

-----

## 🔍 Dataset Visualization (`visualize`)

An interactive web-based viewer for inspecting training chunks. Designed for remote machines with SSH port forwarding.

### Usage

```shell
# On the remote machine:
python -m nano_signal_simulator visualize --data_dir /path/to/dataset --port 8765

# On your local machine, forward the port:
ssh -L 8765:localhost:8765 user@remote_ip

# Then open in your browser:
# http://localhost:8765
```

-----

## 🏋️ Training (`train`)

Train a NanoSimFormer model using basecaller-guided loss. Training uses multi-GPU Distributed Data Parallel (DDP) by default.

### Prerequisites

1. A prepared dataset from `create_dataset`
2. A pre-trained basecalling model — use `--preset` to load official ONT basecalling models (pre-installed in the Docker image; for non-Docker usage, run `bonito download --models` first), or provide a custom `--calling_model` directory

### Usage Example

```shell
# Train with ONT's basecalling model for DNA R10.4.1
python -m nano_signal_simulator train \
  --dataset /path/to/training_dataset \
  --output /path/to/training_output \
  --preset dna_r1041 \
  --batch-size 128 \
  --epochs 5 \
  --lr 0.0002 \
  --ngpus 2

# Save intermediate checkpoints every 5000 steps
python -m nano_signal_simulator train \
  --dataset /path/to/training_dataset \
  --output /path/to/training_output \
  --preset dna_r1041 \
  --epochs 5 \
  --save_every_steps 5000
```

### Training Output

```
training_output/
├── log/                 # TensorBoard logs
│   └── events.out.*
└── weights/             # Model checkpoints
    ├── epoch_1.checkpoint.pth
    ├── epoch_2.checkpoint.pth
    └── step_5000.checkpoint.pth   # (if --save_every_steps is set)
```

Monitor training:
```shell
tensorboard --logdir /path/to/training_output/log
```

-----

## 🔧 Adapting to New Sequencing Chemistry or Update 

NanoSimFormer is designed to be extensible. Here is a step-by-step guide for adapting the model to future ONT sequencing updates (e.g., new pore types or basecalling models).

### Step 1: Prepare Training Data

Prepare training data following the dataset format below (the same format produced by the `create_dataset` command):

| File                  | Shape                   | Dtype     | Description                                                         |
|-----------------------|-------------------------|-----------|---------------------------------------------------------------------|
| `signal.npy`          | `(N, signal_chunk_len)` | `float32` | Z-normalized signal chunks                                          |
| `sequence.npy`        | `(N, max_seq_len)`      | `int8`    | Encoded bases (A=1, C=2, G=3, T/U=4, pad=0)                         |
| `sequence_length.npy` | `(N,)`                  | `int32`   | Actual (unpadded) sequence length per chunk                         |
| `dwells.npy`          | `(N, max_seq_len)`      | `int32`   | Per-base dwell times in samples (sum must equal `signal_chunk_len`) |

As long as your training data follows this format, NanoSimFormer can be retrained for any new pore type or chemistry.

### Step 2: Implement a New Basecaller Class

The training pipeline uses a pluggable basecaller interface. To support a new basecaller, implement the `BaseCaller` abstract class in `train.py`:

```python
from abc import ABC, abstractmethod

class BaseCaller(ABC):
    @abstractmethod
    def load_model(self, model_directory) -> nn.Module:
        """Load the pre-trained basecalling model from a directory."""
        pass

    @abstractmethod
    def basecalling_loss(self, model, x, targets, target_lengths):
        """Compute the CTC loss for the basecaller on the given signal."""
        pass
```

**Example: Adding a new basecaller implementation**

```python
class NewBasecaller(BaseCaller):
    def __init__(self):
        super().__init__()

    def load_model(self, model_directory):
        # Load your custom model or new official model 
        config = load_config(os.path.join(model_directory, "config.toml"))
        model = NewModel(config)
        state_dict = torch.load(os.path.join(model_directory, "weights.pth"), map_location='cpu')
        model.load_state_dict(state_dict)
        return model

    def basecalling_loss(self, model, x, targets, target_lengths):
        model.eval()
        # Forward pass through the basecaller
        logits = model(x)
        # Compute CTC loss
        loss = torch.nn.functional.ctc_loss(
            logits.log_softmax(dim=-1).permute(1, 0, 2),
            targets, input_lengths, target_lengths,
        )
        return loss
```

Then, train with your new basecaller:
```shell
python -m nano_signal_simulator train \
  --dataset /path/to/dataset \
  --output /path/to/output \
  --calling_model /path/to/new_calling_model_weight_dir \
  --caller_name NewBasecaller \
  --epochs 5
```

-----

## 🙏 Acknowledgement 

Some code snippets used to build the model were adapted from [torchtune](https://github.com/meta-pytorch/torchtune) library. 
We also integrated some preprocessing code snippets from [seq2squiggle](https://github.com/ZKI-PH-ImageAnalysis/seq2squiggle) to handle read sampling at given sequencing coverage.
We use [bonito](https://github.com/nanoporetech/bonito) for loading official ONT basecalling models and [remora](https://github.com/nanoporetech/remora) for signal-to-sequence alignment during training dataset preparation.

## 📖 Citation 

Please cite our publication if you use `NanoSimFormer` in your work:

```bibtex
@article{nanosimformer,
  title={NanoSimFormer: An end-to-end Transformer-based simulator for nanopore sequencing signal data},
  author={Xie, Shaohui and Ding, Lulu and Liu, Ling and Zhu, Zexuan},
  journal={bioRxiv},
  pages={2026--01},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}
```

## ©️ Copyright

Copyright 2026 Zexuan Zhu <zhuzx@szu.edu.cn>.<br>
This project is licensed under the Apache License 2.0. See the [LICENSE](./LICENSE) file for details.
