Metadata-Version: 2.4
Name: bicep
Version: 0.1.0
Summary: Bi-directional Cross-Attention Enrichment Preprocessor — a PEFT adapter for HuggingFace transformers.
Author-email: Victor Rowello <vfrowello@gmail.com>
License: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/victorfrowello/BiCEP
Project-URL: Repository, https://github.com/victorfrowello/BiCEP
Project-URL: Issues, https://github.com/victorfrowello/BiCEP/issues
Keywords: peft,lora,transformers,huggingface,adapter,fine-tuning,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1
Requires-Dist: transformers<6,>=5.0
Requires-Dist: peft<0.20,>=0.19.1
Requires-Dist: safetensors
Requires-Dist: accelerate
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: build; extra == "dev"
Dynamic: license-file

# BiCEP — Bi-directional Cross-Attention Enrichment Preprocessor

BiCEP is a [PEFT](https://github.com/huggingface/peft) adapter that **enriches prompt embeddings**
with a small trainable bidirectional transformer before they reach a **frozen** causal LLM. Unlike
LoRA (which modifies the LLM's weights) or prompt tuning (which prepends virtual tokens), BiCEP
transforms the input embedding sequence **in place** — same length, no tokens added or removed —
then feeds the result to the frozen model.

It behaves like a native PEFT adapter: `get_peft_model`, `save_pretrained`, `from_pretrained`,
`push_to_hub`, and `generate` all work without special-casing.

> **⚠️ Experimental.** BiCEP is an architectural hypothesis whose efficacy has not yet been
> empirically validated. The API is usable and stable, but there is no published evidence yet that
> the enricher improves downstream performance. I plan to train and release checkpoints to gather
> empirical data in the near-to-mid future — and anyone is more than welcome to do so independently
> in the meantime. Until then, treat results as unproven and expect the project to stay on `0.x`.

## Install

```bash
pip install bicep        # also needs torch>=2.1, transformers>=5.0,<6, peft>=0.19.1,<0.20
```

## Configuration — three fields

| Field | Default | Meaning |
|---|---|---|
| `base_model_name_or_path` | required | HF model id / local path of the frozen base LLM |
| `num_blocks` | `3` | Number of (bidirectional self-attention + FFN) enrichment blocks |
| `d_head` | `64` | Attention head dimension. **Must divide the base model's `hidden_size`** |

`d_model`, `n_heads`, and `vocab_size` are derived automatically from the base model — you never set them.

## Train

```python
import bicep                                  # registers the BICEP PEFT type on import
from bicep import BiCEPConfig, BiCEPDataCollator
from peft import get_peft_model
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          Trainer, TrainingArguments)

name = "meta-llama/Llama-3.1-8B"
base = AutoModelForCausalLM.from_pretrained(name)
tok  = AutoTokenizer.from_pretrained(name)

model = get_peft_model(base, BiCEPConfig(base_model_name_or_path=name))
collator = BiCEPDataCollator(tokenizer=tok)   # MANDATORY — see below

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="out",
        remove_unused_columns=False,          # REQUIRED with BiCEPDataCollator (see note)
    ),
    train_dataset=dataset,                     # see "Data formats"
    data_collator=collator,
)
trainer.train()
model.push_to_hub("you/bicep-llama3")
```

Only the adapter is trained — all base-model weights stay frozen, and `save_pretrained` /
`push_to_hub` store **only** the adapter weights.

### Publish with a model card

BiCEP is a *third-party* PEFT method, so anyone loading your checkpoint must
`import bicep` first (see below). Ship a model card that says so — `render_model_card`
produces a Hub-ready `README.md` (with `library_name: peft` front matter):

```python
from pathlib import Path
from bicep import render_model_card

save_dir = "out/bicep-llama3"
model.save_pretrained(save_dir)
card = render_model_card(model.peft_config["default"], repo_id="you/bicep-llama3")
Path(save_dir, "README.md").write_text(card, encoding="utf-8")
model.push_to_hub("you/bicep-llama3")          # README.md rides along
```

## Load & generate

```python
import bicep
from peft import PeftModel
from transformers import AutoModelForCausalLM

base  = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base, "you/bicep-llama3")
model.generate(input_ids=...)                  # no BiCEP-specific arguments
```

## The collator is mandatory — causal integrity

BiCEP's enricher is **bidirectional**, so a prompt token's enriched embedding can encode *later*
prompt tokens. To keep this from leaking into the loss, two things must hold, and
`BiCEPDataCollator` enforces both:

1. **Loss masking** — prompt token positions get `labels = -100`, so loss is computed on
   response tokens only.
2. **Prompt-only enrichment** — the collator emits a `bicep_prompt_mask` that tells the model to
   enrich prompt positions only; response/generated tokens pass through as raw embeddings.

Using `BiCEPDataCollator` is **not optional**. Skipping it (or hand-rolling labels) silently
breaks the causal structure and produces a model that cannot generalize.

### Data formats

`BiCEPDataCollator` accepts either:

```python
{"prompt": "Translate to French: Hello", "response": "Bonjour"}
{"text": "The cat sat on the mat ...", "prompt_length": 42}   # prompt_length in tokens
```

A record with no response tokens (empty response, or `prompt_length` ≥ the token count) raises
`ValueError`.

### Why `remove_unused_columns=False`?

`Trainer` drops dataset columns not in the model's `forward` signature *before* collation — which
would remove the collator's raw `prompt`/`response`/`text` inputs. Set
`TrainingArguments(remove_unused_columns=False)` so the collator receives them.

## Scope (v1)

- Single-GPU training. Multi-GPU (DataParallel/FSDP/DeepSpeed) is out of scope.
- You load and manage the base model (dtype, quantization, device); BiCEP does not.
- Composing BiCEP with another PEFT adapter simultaneously is undefined.

## Development

```bash
python -m venv .venv && . .venv/Scripts/activate    # or source .venv/bin/activate
pip install -e ".[dev]"
pytest                                               # CPU-only, uses tiny HF test models
```

## License

BiCEP is licensed under the **GNU General Public License v3.0 or later**
(GPL-3.0-or-later). See [LICENSE](LICENSE) for the full text.
