Metadata-Version: 2.4
Name: poetix-torch
Version: 0.3.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: torch>=2.0.0
License-File: LICENSE
Summary: Hybrid Lexicon-Based Streaming Compressor for AI Infrastructure
Author-email: Mazen Mohamed <mazenmohemed123@gmail.com>
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/mazenmohemed123-ship-it/Poetix

# 🚀 Poetix (`poetix-torch`)

> **The Engine that Democratized AI Training: Breaking the 2 Million Tokens/Sec Barrier on Consumer Hardware.**

[![PyPI version](https://badge.fury.io/py/poetix-torch.svg)](https://badge.fury.io/py/poetix-torch)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![Developer](https://img.shields.io/badge/Developer-Mazen_Mohamed-red.svg)](mailto:mazenmohemed123@gmail.com)

**Poetix** is not just a library; it's a **paradigm shift**. Designed by **Mazen Mohamed (21 years old)**, this Rust-backed engine was built to solve the ultimate bottleneck in LLM training: **Data Starvation.**

While others require massive server farms, Poetix allows you to feed your GPUs at professional speeds using nothing but a standard consumer CPU.

---

## 🏗️ Hardware & Experimental Setup

All benchmarks were executed on a **Standard Consumer Machine**:
* **CPU:** Intel Core i5 (Mid-range)
* **RAM:** Testing included scenarios with only 8GB .
* **Model:** Experiments performed on high-efficiency training models (~77M Parameters).

**The Result?** We hit **1.98 Million Tokens/Sec** on a single thread. This isn't just code; it's mechanical sympathy between Rust and Hardware.

---

## ⚡ The Performance Revolution: Rocket vs. Tank

We provide two distinct engines to suit your infrastructure needs:

### 1. The "Rocket" Mode (In-Memory)
* **Speed:** **1.98M+ tokens/sec.**
* **The Trade-off:** Loads the entire dataset into RAM. 
* **Best for:** Fast iterations on small-to-medium datasets (up to a few GBs) where speed is the only god.

### 2. The "Tank" Mode (Infinite Streaming) 🛡️
* **Speed:** **~770k+ tokens/sec.**
* **The Magic:** **ZERO RAM Pressure.** Constant memory footprint (< 10 MB).
* **Why it's a game-changer:** You can train on a **2 Terabyte** dataset using a **4GB RAM laptop**. It streams data "by the drop" from the disk, ensuring your system never freezes or crashes with `Out of Memory (OOM)` errors.

---

## 💻 CLI Command Center

| Action | Command | Use Case |
| :--- | :--- | :--- |
| **Standard Compress** | `poetix compress data.json data.ptx` | Fast, block-based compression. |
| **Infinite Stream** | `cat data.json | poetix stream-compress > data.ptx` | **The Hero Mode:** Compress TBs of data with zero RAM usage. |
| **Decompress** | `poetix decompress data.ptx data.json` | Restore original data instantly. |
| **Stream Restore** | `cat data.ptx | poetix stream-decompress > data.json` | Restore data via Unix pipes. |

---

## 🧠 Core Features

* **Zero-Copy Architecture:** Leveraging Rust's safety and speed to bypass Python's Global Interpreter Lock (GIL).
* **Lexicon-Based Hybrid Compression:** Combines Aho-Corasick pattern mining with Zstd for up to **90% storage savings**.
* **PyTorch Integration:** Direct injection of `torch.Tensor` batches into your training loop.

---

## 📦 Installation

```bash
pip install poetix-torch
