Metadata-Version: 2.4
Name: spelunk-ml
Version: 0.1.0a1
Summary: Terminal-native IDE for learned representations.
Project-URL: Homepage, https://github.com/sre0089/spelunk
Project-URL: Repository, https://github.com/sre0089/spelunk
Project-URL: Issues, https://github.com/sre0089/spelunk/issues
Author-email: Sarthak Engala <sarthakengala@gmail.com>
License: MIT
License-File: LICENSE
Keywords: interpretability,machine-learning,representations,terminal,tui
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: rich>=13.7
Requires-Dist: textual>=0.70
Requires-Dist: typer>=0.12
Provides-Extra: arrays
Requires-Dist: numpy>=1.26; extra == 'arrays'
Requires-Dist: zarr>=2.18; extra == 'arrays'
Provides-Extra: datasets
Requires-Dist: numpy>=1.26; extra == 'datasets'
Requires-Dist: pillow>=10; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5.1; extra == 'dev'
Provides-Extra: pytorch
Requires-Dist: torch>=2.2; extra == 'pytorch'
Provides-Extra: tui
Description-Content-Type: text/markdown

# Spelunk

Spelunk is a terminal-native toolkit for inspecting learned representations. It captures PyTorch activations, stores them locally, computes layer and feature statistics, runs diagnostics, compares runs, and exports Markdown or JSON reports.

The current release target is a local-first pre-alpha for researchers who are comfortable using a Python model factory and CLI workflow.

## Install

From PyPI:

```bash
python -m pip install spelunk-ml
```

From a checkout:

```bash
python -m pip install -e ".[dev,arrays,datasets,tui]"
```

Add PyTorch support when you want to capture activations:

```bash
python -m pip install -e ".[pytorch]"
```

Run checks:

```bash
python -m pytest
python -m ruff check .
python -m mypy
```

## Quickstart

Generate the example dataset:

```bash
python examples/generate_samples.py
```

Discover valid layer selectors from a PyTorch model factory:

```bash
spelunk layers --model-path examples/model_factory.py --factory build_model
```

Run the lowest-friction workflow:

```bash
spelunk quickstart \
  --run runs/tiny-autoencoder.spelunk \
  --model-path examples/model_factory.py \
  --factory build_model \
  --dataset examples/samples.npy \
  --layers encoder
```

Or capture directly from flags:

```bash
spelunk capture \
  --run runs/tiny-autoencoder.spelunk \
  --model-path examples/model_factory.py \
  --factory build_model \
  --dataset examples/samples.npy \
  --layers encoder
```

For reproducible workflows, generate or edit a config:

```bash
spelunk init --model-path model_factory.py --dataset samples.npy --layers encoder
spelunk capture examples/capture.json
```

Scan the captured run:

```bash
spelunk scan runs/tiny-autoencoder.spelunk
spelunk scan runs/tiny-autoencoder.spelunk --json
```

Inspect one feature:

```bash
spelunk inspect runs/tiny-autoencoder.spelunk --layer encoder --feature 0
spelunk inspect runs/tiny-autoencoder.spelunk --layer encoder --feature 0 --json
```

Generate reports:

```bash
spelunk report runs/tiny-autoencoder.spelunk --format markdown
spelunk report runs/tiny-autoencoder.spelunk --format json
```

Compare two runs:

```bash
spelunk compare runs/baseline.spelunk runs/experiment.spelunk
spelunk compare runs/baseline.spelunk runs/experiment.spelunk --json
```

Open the TUI:

```bash
spelunk
spelunk open runs/tiny-autoencoder.spelunk
```

## Capture Configs

Capture configs are JSON or TOML files. See:

- `examples/capture.json`
- `examples/capture.toml`
- `examples/model_factory.py`
- `docs/CAPTURE_CONFIG.md`
- `docs/EXAMPLE_SMOKE.md`

The model factory must be callable with no arguments and return a `torch.nn.Module`. Layer names in the capture config are PyTorch `named_modules()` paths.

## Python API

```python
from spelunk import Session

session = Session.open("runs/tiny-autoencoder.spelunk")
scan = session.scan()
feature = session.inspect_feature(layer_id="encoder", feature_id="0")
report = session.report(format="markdown")
```

See `docs/PYTHON_API.md`.

## What Works Today

- layer discovery for PyTorch model factories
- direct flag-based capture
- one-shot quickstart capture, scan, and report generation
- starter config generation
- JSON and TOML capture configs
- Spelunk-owned dataset loading for NumPy, CSV, JSONL, and image folders
- PyTorch activation capture through selected forward hooks
- NumPy shard and Zarr activation stores
- layer statistics
- feature statistics and top examples
- activation health diagnostics
- run comparison
- Markdown and JSON reports
- Textual TUI shell with run overview, layers, diagnostics, reports, and report generation

## Current Limitations

- capture requires a local PyTorch model factory
- model loading does not handle checkpoint files directly yet
- PyPI distribution name is `spelunk-ml`; the CLI and import name remain `spelunk`
- diagnostics are intentionally limited to activation health for now

## Documentation

- `docs/VISION.md`
- `docs/ARCHITECTURE.md`
- `docs/DOMAIN_MODEL.md`
- `docs/PYTHON_API.md`
- `docs/CAPTURE_ARCHITECTURE.md`
- `docs/CAPTURE_CONFIG.md`
- `docs/EXAMPLE_SMOKE.md`
- `docs/STORAGE_FORMAT.md`
- `docs/TUI_DESIGN.md`
- `docs/TUI_COMPONENTS.md`
- `docs/DESIGN_LANGUAGE.md`
- `docs/CLI_SPEC.md`
- `docs/TESTING_STRATEGY.md`
- `docs/ROADMAP.md`
- `docs/RELEASE.md`
- `docs/CLEAN_INSTALL.md`
- `docs/PYPI_RELEASE.md`
- `docs/CONTRIBUTING.md`
- `docs/DECISIONS.md`
