Metadata-Version: 2.4
Name: anthracite
Version: 1.4.0
Summary: Universal AI training & fine-tuning framework with smart SFT, external tokenizer support, and HF streaming.
License: Apache License 2.0
        
        Copyright (c) 2026 Nebulix Labs
        
        Licensed under the Apache License, Version 2.0 (the "License");
        you may not use this file except in compliance with the License.
        You may obtain a copy of the License at
        
            http://www.apache.org/licenses/LICENSE-2.0
        
        Unless required by applicable law or agreed to in writing, software
        distributed under the License is distributed on an "AS IS" BASIS,
        WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        See the License for the specific language governing permissions and
        limitations under the License.
        
Project-URL: Homepage, https://github.com/your-org/anthracite
Project-URL: Documentation, https://github.com/your-org/anthracite#readme
Project-URL: Bug Tracker, https://github.com/your-org/anthracite/issues
Keywords: ai,deep-learning,llm,training,fine-tuning,tokenizer,nlp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: safetensors>=0.4
Requires-Dist: tqdm>=4.65
Provides-Extra: hf
Requires-Dist: datasets>=2.14; extra == "hf"
Requires-Dist: huggingface_hub>=0.20; extra == "hf"
Requires-Dist: tokenizers>=0.15; extra == "hf"
Requires-Dist: transformers>=4.36; extra == "hf"
Provides-Extra: all
Requires-Dist: datasets>=2.14; extra == "all"
Requires-Dist: huggingface_hub>=0.20; extra == "all"
Requires-Dist: tokenizers>=0.15; extra == "all"
Requires-Dist: transformers>=4.36; extra == "all"
Dynamic: license-file

# Anthracite v1.5

**Universal AI training & fine-tuning framework** — train text generation and embedding models from scratch or fine-tune them with one Python call.

---

## Table of Contents

1. [Installation](#installation)
2. [Quick Start](#quick-start)
3. [Tokenizer Options](#tokenizer-options-new-in-v15)
4. [Supervised Fine-Tuning (SFT)](#supervised-fine-tuning-sft)
5. [Dataset Formats](#dataset-formats--smart-key-detection)
6. [HF Dataset Streaming](#hf-dataset-streaming-new-in-v15)
7. [Embedding Models](#embedding-models)
8. [Generation & Inference](#generation--inference)
9. [Fine-Tuning an Existing Model](#fine-tuning-an-existing-model)
10. [Configuration Reference](#configuration-reference)
11. [PyPI Upload Guide](#pypi-upload-guide)
12. [What Changed in v1.5](#whats-new-in-v15)

---

## Installation

```bash
pip install anthracite                    # core (PyTorch required separately)
pip install anthracite[hf]               # + HuggingFace tokenizers, datasets, hub
pip install anthracite[all]              # everything above
```

> **PyTorch** is not bundled (too large).  Install it from https://pytorch.org first.

---

## Quick Start

```python
from anthracite import train, generate

# 1. Train a 20 M parameter text model
train(
    model_name  = "MyGPT-20M",
    dataset     = "my_data.jsonl",   # local file, HF dataset id, or directory
    tokens      = 100_000_000,        # training token budget
    params      = "20M",             # target parameter count
    context_length = 512,
    tokenizer   = "bundled",         # ← use built-in GPT-2 tokenizer (new v1.5)
    device      = "auto",
    precision   = "auto",
)

# 2. Generate text
text = generate("./models/MyGPT-20M", "Once upon a time")
print(text)
```

---

## Tokenizer Options (new in v1.5)

Anthracite v1.5 gives you **full control** over the tokenizer.  Four modes:

### 1. Train from scratch (default, v1.0 behaviour)
```python
train(..., tokenizer=None)   # trains a byte-level BPE tokenizer on your dataset
```

### 2. Built-in bundled tokenizer
A GPT-2 style 50 k BPE tokenizer ships with Anthracite.  Use it to skip the tokenizer training step:
```python
train(..., tokenizer="bundled")
```

### 3. HuggingFace tokenizer by repo id
```python
# Load by HF Hub repo id
train(..., tokenizer="gpt2")
train(..., tokenizer="mistralai/Mistral-7B-v0.1")
train(..., tokenizer="meta-llama/Llama-2-7b-hf")

# Load a specific file: repo/model/filename
train(..., tokenizer="bert-base-uncased/tokenizer.json")
```
Requires: `pip install transformers` (or `tokenizers` + `huggingface_hub`)

### 4. Local tokenizer directory / file
```python
train(..., tokenizer="./my_tokenizer/")     # directory with tokenizer.json
train(..., tokenizer="./tokenizer.json")    # exact file path
```

### 5. Live tokenizer object (any library)
```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
train(..., tokenizer=tok)

# SentencePiece, tiktoken, tokenizers, etc. all work via duck-typing:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
train(..., tokenizer=enc)
```

### Standalone tokenizer loading
```python
from anthracite import load_tokenizer

tok = load_tokenizer("bundled")            # built-in
tok = load_tokenizer("gpt2")              # HF Hub
tok = load_tokenizer("./my_tokenizer/")   # local path

ids = tok.encode("Hello world!")
text = tok.decode(ids)
print(text)   # "Hello world!"
```

---

## Supervised Fine-Tuning (SFT)

SFT mode trains the model to generate the *output* given the *input*, and masks input tokens from the loss so the model doesn't just learn to repeat the prompt.

```python
from anthracite import train

train(
    model_name     = "MyChat",
    dataset        = "instructions.jsonl",
    tokens         = 50_000_000,
    params         = "20M",
    objective      = "sft",           # ← enable SFT mode
    sft_mask_input = True,            # mask prompt tokens (default True)
    sft_format     = "auto",          # auto-detect format (default)
    tokenizer      = "bundled",
)
```

#### Explicit format
```python
train(..., sft_format="alpaca")      # Alpaca instruction format
train(..., sft_format="sharegpt")    # ShareGPT multi-turn
train(..., sft_format="chatML")      # ChatML messages format
train(..., sft_format="openai")      # prompt / completion
train(..., sft_format="qa")          # question / answer
train(..., sft_format="text")        # raw text, no masking
```

---

## Dataset Formats & Smart Key Detection

Anthracite v1.5 **automatically detects** your dataset format.  You don't need to specify column names — the `SmartKeyExtractor` peeks at a sample of records, scores every known format, and picks the best match.

### Supported formats

| Format | Keys detected | Input | Output |
|---|---|---|---|
| **Alpaca** | `instruction`, `input`, `output` | instruction + input | output |
| **Alpaca-short** | `instruction`, `output` | instruction | output |
| **OpenAI** | `prompt`, `completion` | prompt | completion |
| **Prompt-Response** | `prompt`, `response` | prompt | response |
| **QA** | `question`, `answer` | question | answer |
| **Generic IO** | `input`, `output` | input | output |
| **AI-User** | `user`, `ai` | user | ai |
| **ShareGPT** | `conversations` list | full conversation | (no mask) |
| **ChatML** | `messages` list | full conversation | (no mask) |
| **Raw text** | `text`, `content`, `body` | full text | (no mask) |
| **2-column generic** | any 2 string keys | first key | second key |

### Alpaca format example
```jsonl
{"instruction": "Translate to French.", "input": "Hello world", "output": "Bonjour le monde"}
{"instruction": "Summarise this.", "input": "Long text...", "output": "Short summary."}
```

### ShareGPT / ChatML format example
```jsonl
{"conversations": [{"from": "human", "value": "Hi"}, {"from": "gpt", "value": "Hello!"}]}
{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}
```

### OpenAI / prompt-completion format
```jsonl
{"prompt": "Tell me a joke.", "completion": "Why did the chicken..."}
```

### Raw text format
```jsonl
{"text": "Once upon a time in a land far away..."}
```

---

## HF Dataset Streaming (new in v1.5)

When you pass a Hugging Face dataset id, Anthracite now **streams** the data instead of downloading the whole dataset first.  This means:

- Training starts **immediately** (no multi-GB download wait)
- Large datasets (100 GB+) fit on machines with little disk space
- Memory usage stays constant regardless of dataset size

```python
train(
    model_name  = "MyGPT",
    dataset     = "HuggingFaceFW/fineweb",   # 15 TB — streams fine!
    tokens      = 1_000_000_000,
    params      = "100M",
    hf_streaming = True,    # default in v1.5
)
```

To download fully (v1.0 behaviour):
```python
train(..., hf_streaming=False)
```

---

## Embedding Models

```python
from anthracite import train, embed, similarity

# Train a sentence embedding model
train(
    model_name  = "MyEmbedder",
    model_type  = "embedding",
    dataset     = "corpus.txt",
    tokens      = 20_000_000,
    params      = "30M",
    context_length = 256,
    objective   = "contrastive",  # or "mlm" for pre-training stage
    tokenizer   = "bundled",
)

# Use it
vecs = embed("./models/MyEmbedder", ["sentence A", "sentence B"])
score = similarity(vecs[0], vecs[1])
print(f"Similarity: {score:.3f}")
```

---

## Generation & Inference

```python
from anthracite import generate

# Simple generation
text = generate("./models/MyGPT-20M", "The quick brown fox")
print(text)

# With options
text = generate(
    "./models/MyGPT-20M",
    "Tell me about space",
    max_new_tokens = 300,
    temperature    = 0.8,
    top_p          = 0.9,
)
```

---

## Fine-Tuning an Existing Model

```python
from anthracite import finetune

finetune(
    model      = "./models/MyGPT-20M",     # base model path or HF repo id
    dataset    = "instructions.jsonl",
    tokens     = 10_000_000,
    objective  = "sft",
    tokenizer  = None,   # None = use base model's tokenizer (default)
    output_dir = "./models/MyGPT-20M-SFT",
)
```

You can also fine-tune from a HuggingFace Hub model:
```python
finetune(
    model   = "your-username/MyGPT-20M",   # HF Hub repo id
    dataset = "instructions.jsonl",
    tokens  = 5_000_000,
)
```

---

## Configuration Reference

All parameters for `train()` and `finetune()`:

| Parameter | Default | Description |
|---|---|---|
| `model_name` | `"anthracite-model"` | Output model name / directory |
| `model_type` | `"text_gen"` | `"text_gen"` or `"embedding"` |
| `dataset` | — | File path, HF dataset id, or list of strings |
| `tokens` | `10_000_000` | Training token budget (int or `"100M"`) |
| `params` | `"20M"` | Target parameters (int or `"20M"`) |
| `context_length` | `512` | Sequence length |
| `vocab_size` | `32768` | Vocabulary size (when training tokenizer from scratch) |
| `tokenizer` | `None` | `None` / `"bundled"` / HF id / path / object |
| `device` | `"auto"` | `"auto"` / `"cuda"` / `"mps"` / `"cpu"` |
| `precision` | `"auto"` | `"auto"` / `"fp32"` / `"bf16"` / `"fp16"` |
| `batch_size` | `"auto"` | Effective batch size or `"auto"` |
| `learning_rate` | `3e-4` | Peak learning rate |
| `output_dir` | `./models/<name>` | Where to save the model |
| `objective` | `"auto"` | `"auto"` / `"causal_lm"` / `"mlm"` / `"contrastive"` / `"sft"` |
| `sft_mask_input` | `True` | Mask prompt tokens from loss (SFT only) |
| `sft_format` | `"auto"` | Dataset format hint (SFT only) |
| `hf_streaming` | `True` | Stream HF datasets (don't download fully) |
| `checkpoint_interval` | `5000` | Save checkpoint every N steps |
| `keep_last_checkpoints` | `3` | Number of recent checkpoints to keep |
| `seed` | `42` | Random seed |
| `val_split` | `0.01` | Fraction of data held out for validation |
| `gradient_accumulation` | `None` | Accumulation steps (auto if `None`) |
| `weight_decay` | `0.1` | AdamW weight decay |
| `grad_clip` | `1.0` | Gradient clipping norm |
| `warmup_ratio` | `0.02` | Fraction of steps used for LR warm-up |
| `num_workers` | `0` | DataLoader worker processes |

---

## PyPI Upload Guide

Follow these steps to publish your own fork or a new release to PyPI.

### 1. Install build tools
```bash
pip install build twine
```

### 2. Build the package
```bash
cd anthracite-pkg        # the directory containing pyproject.toml
python -m build          # creates dist/anthracite_ai-1.5.0.tar.gz and .whl
```

### 3. Test on TestPyPI first (recommended)
```bash
# Upload to TestPyPI
twine upload --repository testpypi dist/*

# Install from TestPyPI to verify
pip install --index-url https://test.pypi.org/simple/ anthracite
```

### 4. Upload to PyPI
```bash
twine upload dist/*
# Enter your PyPI username and password (or API token)
```

### 5. Using an API token (safer than password)
```bash
# Create a token at https://pypi.org/manage/account/token/
twine upload dist/* -u __token__ -p pypi-<your-token-here>
```

### 6. Automate with GitHub Actions
Create `.github/workflows/publish.yml`:
```yaml
name: Publish to PyPI
on:
  release:
    types: [published]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install build twine
      - run: python -m build
      - run: twine upload dist/*
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
```

---

## What's New in v1.5

### Tokenizer overhaul
- **`tokenizer="bundled"`** — GPT-2 tokenizer ships with Anthracite, no training needed.
- **External tokenizer support** — load from HF Hub, local path, or pass any live tokenizer object (HuggingFace, tiktoken, SentencePiece, etc.).
- **Bug fix** — `encode()` and `decode()` no longer require `self` to be passed manually; the tokenizer works correctly as a standalone object.

### Smart SFT format detection (`SmartKeyExtractor`)
- Automatically detects Alpaca, ShareGPT, ChatML, OpenAI, QA, and 10+ other formats.
- Correctly identifies `input` (prompt) vs `output` (answer) fields.
- Masks input tokens from the training loss so the model learns to generate answers, not repeat prompts.

### HF Dataset streaming
- Hugging Face datasets stream by default (`hf_streaming=True`).
- Training starts immediately without downloading multi-GB datasets.
- Works with any HF dataset, including terabyte-scale ones.

### PyPI-ready packaging
- `pyproject.toml` with proper metadata, optional dependencies, and entry points.
- `pip install anthracite` and `pip install anthracite[hf]` work out of the box.

---

## License

MIT License — see `LICENSE` for details.
