Metadata-Version: 2.4
Name: edgegemma-8gb
Version: 0.1.0
Summary: A 4-bit quantized local inference wrapper for Gemma-2B optimized for 8GB VRAM.
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# EdgeGemma-8GB: Local LLM Inference & Evaluation Pipeline

An automated testing pipeline engineered to evaluate the classification capabilities of open-source Large Language Models running entirely locally within highly constrained hardware environments (8GB VRAM).

## Overview
Deploying AI in edge computing environments requires bypassing external cloud APIs due to latency, privacy, or network connectivity constraints. This project demonstrates a robust pipeline for loading, quantizing, and evaluating `google/gemma-2b-it` on consumer-grade hardware.

## Architecture & Optimizations
* **Memory Management:** Utilizes NF4 (4-bit) quantization via `bitsandbytes` to aggressively compress the model weights, allowing a 2B parameter model to run comfortably inside 8GB of VRAM alongside the OS overhead.
* **Singleton Inference Engine:** The LLM loader is modularized to prevent double-allocation in Jupyter environments, protecting against GPU out-of-memory (OOM) crashes.
* **Deterministic Evaluation:** Inference is locked to `temperature=0` with gradient tracking disabled (`torch.no_grad()`) to ensure absolute consistency and speed during zero-shot and few-shot classification benchmarking.

## Setup
1. Clone the repository.
2. Install dependencies: `pip install -r requirements.txt`
3. Run the evaluation pipeline via `notebooks/Task1_Prompting.ipynb`. Data caching and API inference are handled entirely locally.
