Metadata-Version: 2.4
Name: nerfstudio-gnt
Version: 0.0.29
Summary: Unnoficial Implementation for GNT (Generalizable NeRF Transformer | Varma et al) on NeRFStudio
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: gdown>=6.0.0
Requires-Dist: lpips>=0.1.4
Requires-Dist: nerfstudio

# nerfstudio-gnt

Unofficial GNT (Generalizable NeRF Transformer) integration for Nerfstudio.

## Install

```bash
uv pip install -e .
```

## Verify method registration

```bash
ns-train --help | grep -i gnt
```

## Dataset format

This integration uses Nerfstudio's native `VanillaDataManager` with `NerfstudioDataParserConfig`.
Point the dataparser to a scene root containing `transforms.json`.

## Configure GNT

### Basic Training (Full Network Training - Default)
By default, all components (ResUNet feature extractor and GNT transformer network) are fully trainable (`freeze_mode = "none"`):

```bash
ns-train gnt --pipeline.datamanager.data-root /path/to/datasets
```

### Transfer Learning & Freezing Strategies

When initializing with pretrained weights (or transferring priors across distinct image domains such as histology datasets), you can specify how the feature network is frozen via `--pipeline.model.freeze_mode`:

1. **Full Training (`none` - Default):**
   ```bash
   ns-train gnt \
     --pipeline.datamanager.dataparser.data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode none
   ```

2. **Freeze Late Layers (`late`):**
   Keeps the ResNet encoder (`conv1`, `bn1`, `layer1`, `layer2`, `layer3`) trainable to adapt to domain-specific low-level image distributions (e.g. histology), while freezing the U-Net decoder (`upconv3`, `iconv3`, `upconv2`, `iconv2`, `out_conv`):
   ```bash
   ns-train gnt \
     --pipeline.datamanager.dataparser.data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode late
   ```

3. **Freeze Early Layers (`early`):**
   Freezes the ResNet encoder backbone (`conv1`, `bn1`, `layer1`, `layer2`, `layer3`) and trains only the U-Net decoder:
   ```bash
   ns-train gnt \
     --pipeline.datamanager.dataparser.data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode early
   ```

4. **Freeze All Extractor Weights (`all`):**
   Freezes the entire ResUNet feature network:
   ```bash
   ns-train gnt \
     --pipeline.datamanager.dataparser.data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode all
   ```

### Differential Learning Rates
You can configure different learning rates for the feature network and transformer networks via CLI optimizer flags:

```bash
ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --optimizers.feature_net.optimizer.lr 5e-5 \
  --optimizers.network.optimizer.lr 5e-4
```

## Smoke check

```bash
python test.py /path/to/scene_root/transforms.json
```

This runs one train step through `GNTPipeline.get_train_loss_dict` and checks source-view tensor shapes.
