Metadata-Version: 2.4
Name: nerfstudio-gnt
Version: 0.0.30
Summary: Unnoficial Implementation for GNT (Generalizable NeRF Transformer | Varma et al) on NeRFStudio
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: gdown>=6.0.0
Requires-Dist: lpips>=0.1.4
Requires-Dist: nerfstudio

# nerfstudio-gnt

Unofficial GNT (Generalizable NeRF Transformer) integration for Nerfstudio.

## Install

```bash
uv pip install -e .
```

## Verify method registration

```bash
ns-train --help | grep -i gnt
```

## Dataset format

This integration uses Nerfstudio's native `VanillaDataManager` with `NerfstudioDataParserConfig`.
Point the dataparser to a scene root containing `transforms.json`.

---

## Test-Time Adaptation & Fine-Tuning Taxonomy

GNT supports formal test-time adaptation (TTA) strategies across Full Fine-Tuning (FFT), Partial Fine-Tuning (PFT), and Parameter-Efficient Fine-Tuning (PEFT/LoRA) via `--pipeline.model.adaptation_strategy`:

| Strategy | Flag (`--pipeline.model.adaptation_strategy`) | Trainable Parameters | Description |
|---|---|---|---|
| **Zero-Shot** | `zero-shot` | 0% (0 params) | Pure generalization without updating any weights at test time. |
| **Full Fine-Tuning (FFT)** | `full-ft` | 100% (~8.9M params) | All 2D feature extractor and neural transformer weights receive gradients. |
| **PFT (Decoder-Only)** | `pft-decoder` | ~0.67% (~60k params) | Freezes the 2D ResUNet feature network; optimizes only the GNT synthesis transformer. |
| **PFT (Backbone-Only)** | `pft-backbone` | ~99.33% (~8.8M params) | Adapts 2D feature extraction to novel image distributions while keeping rendering transformer frozen. |
| **BitFit** | `bitfit` | ~0.08% (~7k params) | Freezes all weight matrices $W$ and trains only bias vectors $b$. |
| **PEFT (LoRA)** | `peft-lora` | ~0.13% (~11.8k params) | Keeps 100% of base weights frozen; injects low-rank adapters ($W = W_0 + \frac{\alpha}{r} BA$) into target projection layers. |

---

### 1. Zero-Shot Generalization (Default)
Runs test-time evaluation without updating model weights:

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy zero-shot
```

### 2. Full Fine-Tuning (FFT)
Updates both the 2D feature network and GNT transformer rays:

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy full-ft \
  --optimizers.feature_net.optimizer.lr 5e-5 \
  --optimizers.network.optimizer.lr 5e-4
```

### 3. Partial Fine-Tuning (PFT)

#### PFT-Decoder (Freeze 2D Feature Extractor)
Keeps the ResUNet backbone frozen and adapts only the ray transformer synthesis network:

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy pft-decoder
```

#### PFT-Backbone (Freeze Neural Renderer)
Optimizes the 2D feature extractor for distinct domain shifts (e.g. histopathology, microscopy) while freezing rendering:

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy pft-backbone
```

#### BitFit (Bias-Only Fine-Tuning)
Freezes all weight matrices and updates only additive bias terms:

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy bitfit
```

---

### 4. Parameter-Efficient Fine-Tuning (PEFT with LoRA)

Injects low-rank trainable decomposition matrices $A$ and $B$ into attention projections and linear layers while keeping base weights 100% frozen:

$$W = W_0 + \frac{\alpha}{r} (B \cdot A)$$

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy peft-lora \
  --pipeline.model.lora_rank 4 \
  --pipeline.model.lora_alpha 8.0 \
  --pipeline.model.lora_target_modules '["q_proj", "v_proj", "fc"]'
```

#### Zero-Overhead Inference (Weight Merging)
Before evaluation or deployment, call `model.merge_lora_weights()` to fold the adapter update directly into the base weights ($W_0 \leftarrow W_0 + \Delta W$). This guarantees **zero FPS overhead** during rendering. To resume fine-tuning, call `model.unmerge_lora_weights()`.

```python
# In Python
model.merge_lora_weights()  # Merges delta W into base weights (zero runtime overhead)
# ... run high-FPS inference ...
model.unmerge_lora_weights()  # Restores separate adapters if continuing training
```

---

## Console Diagnostic Logs

When initializing any adaptation strategy, GNT logs precise parameter accounting to the console:

```
==============================================================================
 [Few-Shot NeRF] Adaptation & Fine-Tuning Configuration
==============================================================================
 Model:               GNTModel
 Strategy:            PEFT-LORA (peft-lora)
 Total Parameters:    8,948,772
 Trainable Parameters:11,824
 Trainable Ratio:     0.1321% (peft-lora: 0.13% trainable)
 Configuration:       PEFT-LoRA (rank=4, alpha=8.0, targets=['q_proj', 'v_proj', 'fc']): 37 linear layers adapted.
==============================================================================
```

---

## Smoke Check

```bash
python test.py /path/to/scene_root/transforms.json
```
