Metadata-Version: 2.4
Name: optimizer-rag
Version: 0.1.0
Summary: A document compression and optimization library using Groq LLMs and LangChain
Author: Mihir Kapile
Author-email: mihirkapile@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: langchain>=0.1.16
Requires-Dist: groq
Requires-Dist: tiktoken
Requires-Dist: numpy
Requires-Dist: scikit-learn
Requires-Dist: sentence-transformers
Requires-Dist: streamlit
Requires-Dist: python-dotenv
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

Document Optimizer for LLMs

A lightweight Python library and Streamlit web application to compress large documents into the most relevant chunks for downstream use in LLM-based workflows.

It leverages semantic chunking, scoring, and selection to reduce token count while preserving essential content, making it ideal for Retrieval-Augmented Generation (RAG), summarization, and QA pipelines.

Features

Token-based document chunk compression

Semantic scoring and chunk selection with Maximal Marginal Relevance (MMR) or similar methods

Integration with LangChain and Groq LLMs

Modular Python package (optimizer) with reusable backend logic

Streamlit frontend for easy document upload and summarization

Supports sentence-transformers for embeddings and tiktoken for token counting

Folder Structure

.
â”œâ”€â”€ app.py (Streamlit frontend application)
â”œâ”€â”€ requirements.txt
â”œâ”€â”€ sample_files/
â”‚ â””â”€â”€ sample_doc.txt (Sample input document for testing)
â””â”€â”€ optimizer/ (Python module)
â”œâ”€â”€ init.py
â”œâ”€â”€ compressor.py (compress_chunk function)
â”œâ”€â”€ retriever_wrapper.py
â”œâ”€â”€ scorer.py
â”œâ”€â”€ selector.py
â””â”€â”€ token_utils.py

Getting Started

Clone the Repository

git clone https://github.com/yourname/document-optimizer.git
cd document-optimizer

Install Requirements

pip install -r requirements.txt

Run the Streamlit App

streamlit run app.py

Usage

As a Python Library

Import the main compression function and run on your document text:

from optimizer.compressor import compress_chunks

chunks = compress_chunks(
â€ƒdoc_text="...your document text...",
â€ƒtop_k=5,
â€ƒchunk_size=300,
â€ƒstride=150,
)

Development

Install Your Package Locally (Editable Mode)

From the root directory:

pip install -e .

This allows you to edit the code in optimizer/ and see changes immediately without reinstalling.

Import Shortcut

optimizer/init.py enables importing key functions directly:

from optimizer import compress_chunks

Technology Stack

Python
Streamlit
LangChain
Sentence-Transformers
Scikit-Learn
Tiktoken
Groq LLMs (optional)
