Metadata-Version: 2.4
Name: evolo-moead
Version: 0.1.0
Summary: Zero-shot evolutionary architecture search for LoRA via MOEA/D and Gradient Projection Score.
License: Apache-2.0
Project-URL: Homepage, https://github.com/your-org/EvoLoRA-MOEAD
Project-URL: Documentation, https://github.com/your-org/EvoLoRA-MOEAD#readme
Project-URL: Issues, https://github.com/your-org/EvoLoRA-MOEAD/issues
Keywords: lora,peft,neural-architecture-search,evolutionary-algorithm,moead,large-language-models,parameter-efficient-fine-tuning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.0.0
Requires-Dist: transformers>=4.36.0
Requires-Dist: peft>=0.7.0
Requires-Dist: pymoo>=0.6.1
Requires-Dist: numpy>=1.24.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Provides-Extra: examples
Requires-Dist: datasets>=2.16.0; extra == "examples"
Requires-Dist: matplotlib>=3.7; extra == "examples"

# EvoLoRA-MOEAD

**Zero-shot evolutionary architecture search for Low-Rank Adaptation (LoRA).**
> **Acknowledgment & Secondary Creation:** This repository is an independent engineering implementation and secondary creation based on the theoretical framework proposed by Wu, Neri, and Feng (2026). All necessary original theoretical credit belongs to the authors. [DOI: 10.1142/S0129065726500255](https://doi.org/10.1142/S0129065726500255)

`evolo-moead` searches for performant LoRA adapter configurations —
rank `r`, scaling factor `alpha`, and insertion sites `target_modules` —
*before* any fine-tuning begins. The search is training-free: it ranks
candidate architectures by the **Gradient Projection Score (GPS)**, a
singular-value statistic of the calibration gradients, and explores the
trade-off surface with the **MOEA/D** multi-objective evolutionary
algorithm. The recommended configuration is the *knee point* of the
discovered Pareto front and is returned as a ready-to-use
`peft.LoraConfig`.

---

## Why

Choosing LoRA hyperparameters by trial-and-error is expensive: every
candidate normally requires a full fine-tuning run. EvoLoRA-MOEAD
replaces that loop with a single forward/backward pass per candidate
and a principled, multi-objective ranking that balances three competing
goals:

| Objective | Direction | Meaning |
|-----------|-----------|---------|
| **MeanGPS** | maximise | Average rank-`r` gradient energy across LoRA layers — a proxy for adapter capacity. |
| **StdGPS** | minimise | Inter-layer dispersion of that energy — penalises unstable, lopsided adapters. |
| **TrainableParams** | minimise | Adapter parameter budget — honours the parameter-efficiency constraint. |

Internally the optimiser minimises the vector `[-MeanGPS, StdGPS, TP]`.

---

## Installation

```bash
# From PyPI (core search + HuggingFace evaluator)
pip install evolo-moead

# From source, editable, with development and example extras
git clone https://github.com/your-org/EvoLoRA-MOEAD.git
cd EvoLoRA-MOEAD
pip install -e ".[dev,examples]"
```

**Core dependencies:** `torch>=2.0.0`, `transformers`, `peft`, `pymoo`,
`numpy`. Python 3.9+.

---

## Quick start

```python
from evolo_moead import EvoLoRATuner

# `model` is any transformers.PreTrainedModel;
# `calibration_loader` yields dict batches containing `labels`.
tuner = EvoLoRATuner(model, calibration_loader)

best_config = tuner.search_best_config(n_gen=50)   # -> peft.LoraConfig

# Feed the result straight into PEFT for the real fine-tuning run.
from peft import get_peft_model
peft_model = get_peft_model(model, best_config)
```

That is the entire surface. Every evolutionary and linear-algebra
detail is hidden behind `search_best_config`.

### Running the bundled example

```bash
# CPU-only, no model download — verifies the installation in seconds
python examples/quick_start.py --backend mock

# Real model path (requires the `examples` extra and network access)
python examples/quick_start.py --backend hf --n-gen 20
```

---

## API reference

### `EvoLoRATuner(model, calibration_dataloader=None, search_space=None, num_calibration_batches=2, task_type=None, evaluator=None)`

| Argument | Default | Description |
|----------|---------|-------------|
| `model` | — | Base model to search adapters for. |
| `calibration_dataloader` | `None` | Loader of labelled `dict` batches. Required unless `evaluator` is given. |
| `search_space` | `None` | Override of the default discrete space (see below). |
| `num_calibration_batches` | `2` | Batches drawn per candidate evaluation. |
| `task_type` | `None` | `peft` task type (e.g. `"SEQ_CLS"`, `"CAUSAL_LM"`). |
| `evaluator` | `None` | Pre-built evaluator; supply a `MockEvaluator` for CPU-only demos. |

### `search_best_config(n_gen=50, n_partitions=12, seed=42, **moead_kwargs) -> peft.LoraConfig`

Runs the full search and returns the knee-point configuration.
`n_partitions` controls the Das-Dennis reference-direction density
(for three objectives, `n_dirs = (p+1)(p+2)/2`, so `p=12` yields 91
directions). Extra keyword arguments — `n_neighbors`,
`prob_neighbor_mating` — are forwarded to the MOEA/D driver.

### Default search space

```python
{
    "ranks": [4, 8, 16, 32, 64],
    "alphas": [8, 16, 32, 64],
    "target_modules": [["q", "v"], ["q", "k", "v", "o"], "all-linear"],
}
```

Override it to match the module-naming convention of the host model.
For BERT-style models, for instance, the attention projections are
named `query` / `key` / `value`:

```python
search_space = {
    "ranks": [2, 4, 8],
    "alphas": [8, 16],
    "target_modules": [["query", "value"], ["query", "key", "value"]],
}
tuner = EvoLoRATuner(model, loader, search_space=search_space, task_type="SEQ_CLS")
```

---

## How it works

```
EvoLoRATuner.search_best_config
        │
        ▼
LoRASearchProblem  ──uses──►  BaseEvaluator
   (pymoo, 3-obj)                 ├── HuggingFaceEvaluator  (real gradients)
        │                         └── MockEvaluator         (closed-form surrogate)
        ▼
   run_moead  ──►  MOEADResult  ──►  deduplicate_front
        │                                   │
        ▼                                   ▼
  Das-Dennis ref. dirs              find_knee_point (cosine curvature)
                                            │
                                            ▼
                                      peft.LoraConfig
```

### Gradient Projection Score

For a layer gradient `G ∈ ℝ^{d_out × d_in}` with singular values
`σ₁ ≥ σ₂ ≥ …`, the rank-`r` GPS is the Frobenius norm of the optimal
rank-`r` projection:

```
GPS(G, r) = sqrt( Σ_{i=1}^{r} σ_i² )
```

Tensors of rank > 2 (convolution kernels, embeddings) are unfolded to
2D with the leading axis as the output dimension. Zero, `NaN`, and
`Inf` inputs are sanitised, and a CPU `float64` fallback guards against
non-convergent GPU SVD kernels, so a single degenerate layer never
aborts the search.

### Knee-point selection

The Pareto front is Min-Max normalised per objective (to prevent the
high-magnitude parameter axis from dominating) and sorted along the
parameter-count axis. The point whose neighbouring difference vectors
subtend the largest angle is returned as the recommended trade-off.
Degenerate fronts (fewer than three distinct points) fall back to the
minimum-parameter solution.

### Memory discipline (HuggingFaceEvaluator)

Each candidate evaluation mounts a temporary adapter with
`peft.get_peft_model`, runs a short forward/backward pass, aggregates
per-layer GPS, then unconditionally cleans up in a `finally` block:

1. `zero_grad(set_to_none=True)` on the wrapped model,
2. `peft_model.unload()` to revert every module replacement,
3. `gc.collect()`, then `torch.cuda.empty_cache()` and
   `torch.cuda.ipc_collect()` when CUDA is present.

This guarantees the host model returns to its original parameter and
memory state after every call, so the search loop runs at constant
memory.

---

## Project layout

```
EvoLoRA-MOEAD/
├── pyproject.toml
├── src/
│   └── evolo_moead/
│       ├── __init__.py         # exposes EvoLoRATuner
│       ├── api.py              # high-level orchestrator
│       ├── core/
│       │   ├── gps_metric.py   # SVD gradient projection score
│       │   └── geometry.py     # Pareto knee-point extraction
│       ├── evaluator/
│       │   ├── base.py         # evaluator ABC + MockEvaluator
│       │   └── hf_evaluator.py # HuggingFace gradient evaluator
│       └── search/
│           ├── problem.py      # pymoo multi-objective problem
│           └── optimizer.py    # MOEA/D driver
├── examples/
│   └── quick_start.py
└── README.md
```

---

## Citation

This package is an independent engineering implementation and secondary creation based on the method described in *Zero-Shot
Evolutionary Architecture Search for Low-Rank Adaptation* (Wu et al.,
2026). Cite the original work when applying this engineering implementation in research. [DOI: 10.1142/S0129065726500255](https://doi.org/10.1142/S0129065726500255)

## License

Apache-2.0.
