Metadata-Version: 2.4
Name: asag-rag
Version: 1.1.128
Summary: Add your description here
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
Requires-Dist: build>=1.5.0
Requires-Dist: datasets>=5.0.0
Requires-Dist: faiss-cpu>=1.14.2
Requires-Dist: google-genai>=2.8.0
Requires-Dist: kaggle>=2.2.1
Requires-Dist: langchain>=1.3.4
Requires-Dist: langchain-community>=0.4.2
Requires-Dist: langchain-huggingface>=1.2.2
Requires-Dist: langchain-tavily>=0.2.18
Requires-Dist: matplotlib>=3.11.0
Requires-Dist: nltk>=3.9.4
Requires-Dist: openai>=2.41.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas>=3.0.3
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: rank-bm25>=0.2.2
Requires-Dist: scikit-learn>=1.9.0
Requires-Dist: sentence-transformers>=5.7.0
Requires-Dist: tavily-python>=0.7.25
Requires-Dist: tiktoken>=0.13.0
Requires-Dist: transformers[torch]==4.57.6
Requires-Dist: twine>=6.2.0

# ASAG-RAG

ASAG-RAG is a research project for automated short-answer grading. It can grade a dataset with an LLM alone or enrich the grading prompt with retrieved examples using a retrieval-augmented generation (RAG) pipeline.

## Requirements

- Python 3.12 (the supported range is `>=3.12,<3.13`)
- [uv](https://docs.astral.sh/uv/)
- An API key for the selected model provider. The supplied experiment configurations use OpenAI.

## Project structure

```text
.
├── main.py                 # Command-line entry point for grading experiments
├── pyproject.toml          # Python package metadata and dependencies
├── uv.lock                 # Locked dependency versions
├── .env.example            # Template for provider credentials
├── configs/
│   └── experiments/        # Reusable JSON experiment configurations
├── dataset/
│   ├── raw/                # Source datasets
│   ├── translated/         # Translated dataset variants
│   ├── processed/          # CSV splits consumed by main.py
│   ├── embeddings/         # FAISS indexes used by retrieval pipelines
│   ├── augmented/          # LLM-augmented training data
│   └── labelled/           # Labeled grading data and outputs
├── src/rag/
│   ├── pipelines/          # No-retrieval, vector, BM25, and LinUCB pipelines
│   ├── retrievers/         # Local, external, hybrid, vector, and BM25 retrieval
│   ├── llms/               # OpenAI, Gemini, Hugging Face, and OpenRouter backends
│   ├── ingestions/         # CSV, Excel, Parquet, and dataset ingestion helpers
│   ├── config.py           # Shared RAG/LLM settings
│   └── pipeline.py         # Pipeline dispatcher
├── eval/                   # Evaluation datasets, metrics, and runner scripts
├── notebooks/              # Exploratory data preparation, training, and analysis
├── params/                 # Saved experiment parameter files
├── result/                 # Generated grading and evaluation results
└── scripts/                # Maintenance and release scripts
```

The primary input is a CSV split in a folder under `dataset/processed/`. The value passed to `--split` is its filename without `.csv` (for example, `test` for `test.csv`). A grading CSV must provide the question, reference-answer, and student-answer columns named by the corresponding CLI options. Retrieval runs also require a document embedding directory; the LinUCB pipeline additionally requires a question-embedding directory.

## How to run

### 1. Install dependencies

From the repository root:

```powershell
uv sync
```

### 2. Configure credentials

Copy the template and fill in only the keys required by your selected backend:

```powershell
Copy-Item .env.example .env
```

For the default OpenAI backend, set `OPENAI_API_KEY` in `.env`:

```dotenv
OPENAI_API_KEY=your_key_here
```

The application loads `.env` automatically. Do not commit this file.

### 3. Run an included experiment

Use a JSON configuration for reproducible runs. Command-line options override values in the JSON file.

```powershell
uv run main.py --config configs/experiments/rahutomo-no-retrieval-test.json
```

To run the supplied LinUCB retrieval experiment:

```powershell
uv run main.py --config configs/experiments/saf-translated-linucb.json
```

Results are written to the configuration's `output_folder`, typically below `result/evaluation/grading/`.

### 4. Run with explicit options

No-retrieval grading:

```powershell
uv run main.py `
  --pipeline no-retrieval `
  --dataset-path ./dataset/processed/rahutomo `
  --split test `
  --output-folder ./result/evaluation/grading/rahutomo `
  --question-column Soal `
  --reference-column KunciJawaban `
  --answer-column Jawaban `
  --min-score 0 `
  --max-score 100 `
  --backend openai `
  --model gpt-4o-mini
```

LinUCB retrieval grading:

```powershell
uv run main.py `
  --pipeline linucb `
  --retrieve-method linucb `
  --dataset-path ./dataset/processed/saf-translated `
  --embedding-path ./dataset/embeddings/saf-translated `
  --question-embedding-path ./dataset/embeddings/saf-translated-question `
  --split test `
  --output-folder ./result/evaluation/grading/saf-translated `
  --question-column question_ind `
  --reference-column reference_ind `
  --answer-column answer_ind `
  --score-column score `
  --min-score 0 `
  --max-score 1 `
  --backend openai `
  --model gpt-4o-mini
```

Useful options include `--batch-num` (number of output batches), `--start` (first batch index), `--top-k` (retrieval count), and `--temperature`. See all options with:

```powershell
uv run main.py --help
```

### Pipeline note

`main.py --help` currently lists `vector` and `bm25`, while the internal pipeline dispatcher registers them as `vector-retrieval` and `bm25-retrieval`. As a result, those two CLI values do not currently resolve to a pipeline. Use `no-retrieval` or `linucb` from the command line, or align the names in `main.py` and `src/rag/pipelines/__init__.py` before using vector or BM25 runs.

## Evaluation

The `eval/` directory contains scripts for per-label, per-classification, source, and translation-quality evaluation. Run a script through `uv` after reviewing its options, for example:

```powershell
uv run eval/run_local_eval.py --help
```
