Metadata-Version: 2.4
Name: scratchgpt
Version: 0.3.0
Summary: A small-scale transformer-based language model implemented from scratch in Python.
Project-URL: Homepage, https://github.com/LabStrangeLoop/scratchgpt
Project-URL: Repository, https://github.com/LabStrangeLoop/scratchgpt
Author-email: Aleksandr Yeganov <ayeganov@gmail.com>, Dario Cazzani <dariocazzani@gmail.com>
License: MIT License
        
        Copyright (c) 2025 LabStrangeLoop
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: deep-learning,gpt,language-model,pytorch,tokenizer,transformer
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: numpy>=2.3.2
Requires-Dist: ptflops>=0.7.5
Requires-Dist: pydantic-settings>=2.10.1
Requires-Dist: pydantic-yaml>=1.6.0
Requires-Dist: torch>=2.8.0
Requires-Dist: tqdm>=4.67.1
Requires-Dist: types-tqdm>=4.67.0.20250809
Provides-Extra: hf-tokenizers
Requires-Dist: huggingface-hub>=0.34.4; extra == 'hf-tokenizers'
Requires-Dist: tokenizers>=0.19.0; extra == 'hf-tokenizers'
Description-Content-Type: text/markdown

# ScratchGPT

![ScratchGPT](./assets/logo.webp)

ScratchGPT is a Python project that implements a small-scale transformer-based
language model from scratch. It provides functionality for training the model
on custom datasets and generating text based on prompts.

## Features

- Custom transformer architecture implementation
- Training on user-provided text data
- Text generation using the trained model
- Flexible tokenization using TikToken
- Command-line interfaces for training and inference

## Roadmap

- [x] Switch to uv
- [x] Make it easy to modify with a config file
- [x] Extract the loss calculation from the model
- [x] Rename main to train
- [x] Create or check tokenizer interface
- [x] Create an easy to use interface
- [ ] Make it into a package
- [ ] Apply SOTA optimizations

## Requirements

- Python 3.12+
- `uv` for dependency management

## Installation

1. Clone the repository:
   ```
   git clone https://github.com/LabStrangeLoop/scratchgpt.git
   cd scratchgpt
   ```

2. Install dependencies using uv:
   ```
   uv sync --all-groups
   ```

## Usage

### Training

To train the model on your custom dataset:

```
uv run train -t <path_to_training_data> -e <experiment_folder>
```

- `-t, --train_source`: Path to the training data file or folder
- `-e, --experiment`: Path to the folder where experiment checkpoints will be saved


### Inference

To generate text using a trained model:

```
uv run infer -e <experiment_folder> [-d <device>] [-m <max_tokens>]
```

- `-e, --experiment`: Path to the folder containing the trained model
- `-d, --device`: Device to run the model on (default: "cuda")
- `-m, --max_tokens`: Maximum number of tokens to generate (default: 512)

### Tokenization

To explore the TikToken tokenizer:

```
uv run tiktoken
```

## Project Structure

- `scratchgpt/train.py`: Main training script
- `scratchgpt/infer.py`: Inference script for text generation
- `scratchgpt/model_io.py`: Utilities for saving and loading models
- `scratchgpt/tokenizer/`: Tokenizer implementations

## Development

This project uses various development tools:

- `mypy` for static type checking
- `ruff` for formatting and standard adherence
- `pytest` for testing

Run the following commands to ensure code quality:

```
uv run ruff --fix .
uv run mypy .
uv run pytest
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

[MIT License](LICENSE)

## Authors

- Aleksandr Yeganov
- Dario Cazzani
