Metadata-Version: 2.4
Name: nerfstudio-pixelnerf
Version: 0.0.22
Summary: Unofficial Implementation of `pixelNeRF: Neural Radiance Fields from One or Few Images` [Yu et al.] for NeRFStudio
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: dotmap>=1.3.30
Requires-Dist: gdown>=6.0.0
Requires-Dist: lpips>=0.1.4
Requires-Dist: nerfstudio
Requires-Dist: pyhocon>=0.3.63

# nerfstudio-pixel-nerf

Unofficial pixelNeRF integration for Nerfstudio. A framework for training and rendering few-shot Neural Radiance Fields using image-conditioned feature extraction.

## Install

```bash
uv pip install -e .
```

## Verify method registration

```bash
ns-train --help | grep -i pixel-nerf
```

## Dataset Format
This integration uses a custom pipeline to inject source views dynamically into the RayBundle metadata. It is compatible with the standard `NerfstudioDataParserConfig`. Point the dataparser to a scene root containing a valid `transforms.json`.

---

## Test-Time Adaptation & Fine-Tuning Taxonomy

PixelNeRF supports formal test-time adaptation (TTA) strategies across Full Fine-Tuning (FFT), Partial Fine-Tuning (PFT), and Parameter-Efficient Fine-Tuning (PEFT/LoRA) via `--pipeline.model.adaptation_strategy`:

| Strategy | Flag (`--pipeline.model.adaptation_strategy`) | Trainable Parameters | Description |
|---|---|---|---|
| **Zero-Shot** | `zero-shot` | 0% (0 params) | Pure generalization without updating any weights at test time. |
| **Full Fine-Tuning (FFT)** | `full-ft` | 100% (~21.5M params) | All 2D ResNet encoder and NeRF synthesis MLP weights receive gradients. |
| **PFT (Decoder-Only)** | `pft-decoder` | ~0.77% (~165k params) | Freezes the 2D ResNet encoder; optimizes only the synthesis MLPs (`mlp_coarse` / `mlp_fine`). |
| **PFT (Backbone-Only)** | `pft-backbone` | ~99.23% (~21.3M params) | Adapts 2D feature extraction to novel image distributions while keeping synthesis MLPs frozen. |
| **BitFit** | `bitfit` | ~0.04% (~9.4k params) | Freezes all weight matrices $W$ and trains only additive bias vectors $b$. |
| **PEFT (LoRA)** | `peft-lora` | ~0.07% (~14.4k params) | Keeps 100% of base weights frozen; injects low-rank adapters ($W = W_0 + \frac{\alpha}{r} BA$) into target linear layers. |

---

### 1. Zero-Shot Generalization (Default)
Runs test-time evaluation without updating model weights:

```bash
ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy zero-shot
```

### 2. Full Fine-Tuning (FFT)
Updates both the ResNet encoder and NeRF MLPs:

```bash
ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy full-ft \
  --optimizers.encoder.optimizer.lr 1e-5 \
  --optimizers.nerf.optimizer.lr 1e-4
```

### 3. Partial Fine-Tuning (PFT)

#### PFT-Decoder (Freeze 2D Feature Extractor)
Keeps the 2D ResNet encoder frozen and adapts only the neural synthesis MLPs:

```bash
ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy pft-decoder
```

#### PFT-Backbone (Freeze Neural Renderer)
Optimizes the 2D feature extractor for distinct domain shifts (e.g. histopathology, microscopy) while freezing synthesis MLPs:

```bash
ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy pft-backbone
```

#### BitFit (Bias-Only Fine-Tuning)
Freezes all weight matrices and updates only additive bias terms:

```bash
ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy bitfit
```

---

### 4. Parameter-Efficient Fine-Tuning (PEFT with LoRA)

Injects low-rank trainable decomposition matrices $A$ and $B$ into the synthesis MLP layers while keeping base weights 100% frozen:

$$W = W_0 + \frac{\alpha}{r} (B \cdot A)$$

```bash
ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy peft-lora \
  --pipeline.model.lora_rank 4 \
  --pipeline.model.lora_alpha 8.0 \
  --pipeline.model.lora_target_modules '["q_proj", "v_proj", "fc"]'
```

#### Zero-Overhead Inference (Weight Merging)
Before evaluation or deployment, call `model.merge_lora_weights()` to fold the adapter update directly into the base weights ($W_0 \leftarrow W_0 + \Delta W$). This guarantees **zero FPS overhead** during rendering. To resume fine-tuning, call `model.unmerge_lora_weights()`.

```python
# In Python
model.merge_lora_weights()  # Merges delta W into base weights (zero runtime overhead)
# ... run high-FPS inference ...
model.unmerge_lora_weights()  # Restores separate adapters if continuing training
```

---

## Console Diagnostic Logs

When initializing any adaptation strategy, PixelNeRF logs precise parameter accounting to the console:

```
==============================================================================
 [Few-Shot NeRF] Adaptation & Fine-Tuning Configuration
==============================================================================
 Model:               PixelNeRFModel
 Strategy:            PEFT-LORA (peft-lora)
 Total Parameters:    21,464,433
 Trainable Parameters:14,376
 Trainable Ratio:     0.0670% (peft-lora: 0.07% trainable)
 Configuration:       PEFT-LoRA (rank=4, alpha=8.0, targets=['q_proj', 'v_proj', 'fc']): 16 linear layers adapted.
==============================================================================
```

---

## Smoke Check

```bash
python test.py
```
