Metadata-Version: 2.4
Name: polars-seq
Version: 0.1.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Dist: polars>=1.0
License-File: LICENSE
Summary: Polars expression plugin for translating DNA/RNA to protein, bit-for-bit compatible with BioPython's Seq.translate()
Keywords: polars,bioinformatics,dna,rna,protein,translation,codon,genetic-code,biopython
Author-email: Chris Thorpe <drchristhorpe@googlemail.com>
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/drchristhorpe/polars-seq/blob/master/CHANGELOG.md
Project-URL: Homepage, https://github.com/drchristhorpe/polars-seq
Project-URL: Issues, https://github.com/drchristhorpe/polars-seq/issues
Project-URL: Repository, https://github.com/drchristhorpe/polars-seq

# polars-seq

Translate DNA/RNA to protein **inside Polars**, at Rust speed, with **exactly** the semantics of
BioPython's `Seq.translate()`.

```python
import polars as pl
import polars_seq  # noqa: F401 -- importing registers the .seq namespace

df = pl.DataFrame({"dna": ["ATGAAATTTTAA", "ATGGGCCCCTGA"]})

df.with_columns(protein=pl.col("dna").seq.translate())
# ┌──────────────┬─────────┐
# │ dna          ┆ protein │
# ╞══════════════╪═════════╡
# │ ATGAAATTTTAA ┆ MKF*    │
# │ ATGGGCCCCTGA ┆ MGP*    │
# └──────────────┴─────────┘
```

It is a native Polars **expression plugin** (Rust, via `pyo3-polars`), not a Python UDF: it runs
inside the query engine, parallelises across threads, holds no GIL, and composes with the lazy
optimiser and the streaming engine.

---

## Install and build (uv)

This is a compiled plugin, so building it needs a **Rust toolchain**. Nothing but `polars` is
needed at runtime.

### 1. Prerequisites

**uv** — if you don't have it:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

**Rust** — `rustup` gives you `cargo` and `rustc`. The `stable` channel is pinned in
`rust-toolchain.toml`, so rustup will fetch the right one automatically:

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"     # or restart your shell
cargo --version               # should print something
```

You also need a C linker (`cc`). On Debian/Ubuntu: `sudo apt install build-essential`.

### 2. Build and install

```bash
git clone <this repo>
cd polars_seq

uv sync          # creates .venv on Python 3.14, compiles the Rust extension, installs everything
```

`uv sync` reads `.python-version` (3.14) and `pyproject.toml`, downloads the interpreter if you
don't have it, builds the crate through the `maturin` backend, and installs `polars-seq` into
`.venv` along with the dev dependencies (pytest, biopython).

The first build compiles all of `polars`'s Rust dependencies and takes a few minutes. Later
builds are incremental and take seconds.

Check it works:

```bash
uv run python -c "
import polars as pl, polars_seq
print(pl.DataFrame({'dna': ['ATGAAATTTTAA']}).with_columns(p=pl.col('dna').seq.translate()))
"
```

Run the tests (this is also the BioPython parity check):

```bash
uv run pytest -q
```

### 3. Rebuilding after you change the Rust

**This is the one thing that will bite you.** `uv sync` caches the built wheel and does **not**
notice that you edited `src/*.rs` — you will keep importing the old binary and wonder why your
change did nothing. To actually rebuild, use `maturin develop`, which compiles in place:

```bash
uv run maturin develop --uv            # debug build, fast to compile
uv run maturin develop --uv --release  # optimised; use this for benchmarking
```

Or force uv to rebuild from scratch:

```bash
uv sync --reinstall-package polars-seq --no-cache
```

Pure-Python changes under `python/polars_seq/` need no rebuild at all — the install is editable.

### 4. Regenerating the codon tables

`src/codon_tables.rs` is generated from BioPython and committed. You only need to regenerate it
if you bump BioPython or change the alphabet:

```bash
uv run python codegen/generate_tables.py   # verifies 27 tables x 4913 codons, then writes
uv run maturin develop --uv                # rebuild so the new tables are compiled in
```

### 5. Building a wheel to install elsewhere

```bash
uv build                       # -> dist/polars_seq-0.1.0-*.whl
uv pip install dist/*.whl      # on any machine with Python >= 3.10; no Rust needed
```

### Troubleshooting

**`Both VIRTUAL_ENV and CONDA_PREFIX are set`** — maturin refuses to guess which environment you
mean. If you have conda on your `PATH` (an active `base` env is enough), either `conda deactivate`
first, or just unset it for the one command:

```bash
env -u CONDA_PREFIX uv run maturin develop --uv
```

**`cargo: command not found`** — `~/.cargo/bin` isn't on your `PATH`. `source "$HOME/.cargo/env"`.

**Your Rust change had no effect** — see §3. `uv sync` served you a cached wheel; use
`maturin develop`.

**Python version** — requires ≥ 3.10; developed and tested on **3.14**. The extension is built
against the stable ABI (`abi3`), so one wheel works across versions.

---

## Usage

### `.seq.translate()`

```python
pl.col("dna").seq.translate(
    table=1,           # NCBI id, or name/alias: "Standard", "SGC0", "Vertebrate Mitochondrial"
    stop_symbol="*",
    to_stop=False,     # stop at the first in-frame stop, excluding it
    cds=False,         # validate as a complete coding sequence
    gap="-",
    on_error="raise",  # or "null" -- see below
)
```

Every argument means what it means in BioPython.

```python
df.with_columns(
    protein   = pl.col("dna").seq.translate(),
    orf       = pl.col("dna").seq.translate(to_stop=True),
    mito      = pl.col("dna").seq.translate(table=2),
    bacterial = pl.col("dna").seq.translate(table="Bacterial"),
)
```

`cds=True` validates the sequence as a complete CDS: it must start with a start codon (reported
as `M` whatever it actually encodes), have a length divisible by three, end with a stop codon
(dropped from the output), and contain no internal stop.

```python
pl.col("dna").seq.translate(cds=True)   # "TTGAAATAA" -> "MK"  (TTG is a start codon)
```

Nulls pass through as nulls. A trailing partial codon is dropped, as in BioPython.

### `.seq.reverse_complement()`

IUPAC-aware and case-preserving. Six-frame translation is then just:

```python
df.select(
    **{f"fwd{i}": pl.col("dna").str.slice(i).seq.translate() for i in range(3)},
    **{f"rev{i}": pl.col("dna").seq.reverse_complement().str.slice(i).seq.translate()
       for i in range(3)},
)
```

### `polars_seq.codon_tables()`

All 27 NCBI genetic codes as a DataFrame — ids, names, aliases, and which are dual-coding.

---

## Ambiguity codes are handled properly

Ambiguous IUPAC nucleotides resolve the way BioPython resolves them, which is more subtle than
"anything unclear becomes `X`":

| codon | → | why |
|---|---|---|
| `GGN` | `G` | all four `GGx` codons are Gly, so there is no ambiguity to report |
| `TAR` | `*` | `R`=A/G, and both `TAA` and `TAG` are stops |
| `TAN` | `X` | expands to two stops *and* two Tyr — genuinely unresolvable |
| `RAT` | **`B`** | `GAT`=Asp, `AAT`=Asn → **`B`** is the IUPAC code for "Asp or Asn" |
| `SAA` | **`Z`** | `CAA`=Gln, `GAA`=Glu → **`Z`** = "Glu or Gln" |

`RAT` → `B` is the one that catches people out. An implementation that emits `X` for every
unresolvable codon looks correct until someone runs it on ambiguous data.

---

## The X-codon quirk

While building this I ran into a genuinely surprising corner of `Seq.translate()`. It is worth
knowing about whether or not you use this library.

**`CTX` translates to `L`. `XXX` is an error.**

Both contain `X`. Here is why they differ.

BioPython keeps *two* different sets of nucleotide letters, and they disagree with each other:

1. the **expansion table**, which says what each letter can stand for. It has **17** keys, and
   `X` is one of them — it expands to `GATC`, exactly like `N`.
2. the **validity check**, which decides whether a codon it *failed* to translate is at least
   made of legal characters. That set is `_ambiguous_dna_letters | _ambiguous_rna_letters` —
   **16** letters, and `X` is **not** among them.

Translation consults the expansion table **first**, and falls back to the validity check only
when the expansion fails. And the expansion fails in exactly one situation: **when it runs into a
stop codon.** So:

- `CTX` → expands to `CTA/CTC/CTG/CTT` → all four are Leucine, no stops → resolves to **`L`**.
  The validity check is never reached and the `X` costs nothing.
- `XXX` → expands to all 64 codons → amino acids *and stops* → the expansion gives up → *now*
  the validity check runs, sees the `X`, and rejects the codon: `Codon 'XXX' is invalid`.

The precise rule, which we verified exhaustively against BioPython (817 X-bearing codons × 27
tables, zero counterexamples):

> **An `X`-bearing codon is accepted if and only if none of the codons it expands to is a stop.**
> When accepted, it means exactly what the `N` spelling means.

So `X` is a perfect synonym for `N` — right up to the moment the expansion touches a stop codon,
where `N` degrades gracefully and `X` becomes a hard error:

| expansion | `N` spelling | `X` spelling |
|---|---|---|
| one amino acid, no stops | `CTN` → `L` | `CTX` → `L` |
| several amino acids, no stops | `AAN` → `X` | `AAX` → `X` |
| amino acids **and stops** | `TAN` → `X` | `TAX` → **error** |
| everything | `NNN` → `X` | `XXX` → **error** |

Note the second row: an accepted `X` codon *can* come out as `X` — there `X` is an **amino-acid**
ambiguity code (Lys-or-Asn), not a nucleotide. What it can never be is a stop.

This is easy to get wrong in both directions: treat `X` as invalid everywhere and you break
`CTX` → `L`; treat it as a plain synonym for `N` and you wrongly accept `TAX` and `XXX`. I got it
wrong the first way, and only the differential fuzz caught it — on the sequence `CTTCTX`. Every
hand-written test I had passed. `polars-seq` now reproduces the real behaviour and re-sweeps all
817 X-bearing codons × 27 tables against BioPython on each test run.

*This is BioPython's actual, current behaviour — 1.78 and 1.87 agree — so it is a quirk to match,
not a bug to route around. It falls out of the two letter-sets having drifted apart.*

---

## Differences from BioPython

Two, both deliberate, because a DataFrame is not a single sequence.

**1. `on_error="null"`.** BioPython raises on a malformed sequence, and so do we, by default. But
in a million-row frame, one bad sequence aborting the whole query is usually not what you want:

```python
df.with_columns(protein=pl.col("dna").seq.translate(on_error="null"))
# invalid sequences become null; everything else still translates
```

The default, `on_error="raise"`, reports the row index and the offending codon.

**2. No per-row warnings.** BioPython emits a `BiopythonWarning` for a trailing partial codon.
Warning once per row from inside a parallel Rust kernel is not viable, so we are quiet about it.
The *behaviour* is identical — the partial codon is dropped either way.

Dual-coding tables (27/28/31) do still warn, once, when the expression is built — and `to_stop`
with those tables is still rejected, exactly as BioPython rejects it.

---

## Correctness

The parity claim is enforced, not asserted. `uv run pytest` runs:

- **an exhaustive codon sweep** — all 4913 codons × all 27 NCBI tables = **132,651 comparisons**
  against BioPython, including which codons it *refuses*;
- **differential fuzz** — thousands of random sequences over ten alphabets (unambiguous, IUPAC,
  RNA, gapped, lower-case, invalid-character, …) crossed with the full option space, asserting we
  produce the same protein *and* fail on exactly the same inputs;
- **golden tests** for every documented rule and trap;
- **Polars integration** — nulls, empty frames, chunked and sliced Series, lazy, streaming,
  `group_by().agg()`.

The lookup tables are **generated from BioPython** (`codegen/generate_tables.py`) by calling the
real `_translate_str` on every codon, rather than by re-implementing its resolver in Rust — which
is exactly the sort of code that produces `RAT` → `X` bugs. BioPython is a build- and test-time
dependency only; it is not needed at runtime.

---

## Performance

A Rust kernel with one array lookup per codon, no per-row allocation, parallel across chunks.

`uv run python tools/validate.py` reproduces the numbers on your own machine and data, and writes
a full row-by-row BioPython comparison to `tmp/` so you can inspect the output rather than trust
it.

---

## Layout

```
codegen/generate_tables.py   BioPython -> Rust lookup tables (self-verifying)
src/translate.rs             the kernel: framing, stops, gaps, cds rules
src/codon_tables.rs          GENERATED -- do not edit
src/expressions.rs           Polars expression entry points
python/polars_seq/           the .seq namespace and argument validation
tests/                       golden, exhaustive, differential-fuzz, integration
tools/validate.py            writes validation + benchmark artefacts to tmp/
```

## Licence

MIT.

