Metadata-Version: 2.4
Name: perovskite-tokenizer
Version: 0.1.0
Summary: Chemical-aware BPE tokenizer for perovskite formulas
Author-email: Gosh Kolotyan <goshkolotyan@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/GoshKolotyan/perovskite_tokenizer
Project-URL: Issues, https://github.com/GoshKolotyan/perovskite_tokenizer/issues
Project-URL: Repository, https://github.com/GoshKolotyan/perovskite_tokenizer
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pymatgen
Requires-Dist: tokenizers<1,>=0.20
Dynamic: license-file

# perovskite_tokenizer

A small chemical-aware BPE encoder for perovskite formulas.

## Install

```bash
pip install .
```

## Use

```python
from perovskite_tokenizer import PerovskiteTokenizer

tokenizer = PerovskiteTokenizer()

encoded = tokenizer.encode_formula("CsPbBr3")
batch = tokenizer.encode_formula(["CsPbBr3", "FAPbI3"])

token_ids = encoded.token_ids
features = encoded.output_tokens
payload = encoded.to_dict()

raw = tokenizer.encode_formula("CsPbBr3", add_special_tokens=False)
```

`encode_formula` accepts either one formula or a list of formulas and returns an
`EncodedFormula` or a list of them. `encode` is an alias with the same behavior.
The packaged model adds `[CLS]` and `[END]` by default. Unsupported spans become
`[UNK]`, empty formulas raise `ValueError`, and an empty list returns `[]`.

`output_tokens` keeps non-element tokens as vocabulary IDs. Each element symbol
is followed by its atomic number, atomic mass, electronegativity, atomic radius,
ionization energy, and average ionic radius. Before tokenization, `MA`, `FA`, and
`DMA` expand to `CH3NH3`, `CH(NH2)2`, and `(CH3)2NH2`, respectively. Missing or
non-finite numeric properties are represented by `0.0`.

To use another compatible model directory:

```python
tokenizer = PerovskiteTokenizer("path/to/model-directory")
```
