Metadata-Version: 2.4
Name: neurodags
Version: 0.3.0
Summary: A slurm friendly MEEG derivative extraction package leveraging bids-like data organization and DAG processing.
Project-URL: Homepage, https://github.com/yjmantilla/neurodags
Project-URL: Documentation, https://yjmantilla.github.io/neurodags/
Project-URL: Repository, https://github.com/yjmantilla/neurodags
Project-URL: Issues, https://github.com/yjmantilla/neurodags/issues
Author-email: Yorguin-Jose Mantilla-Ramos <yjmantilla@gmail.com>
License: MIT
License-File: LICENSE
Keywords: bids,derivative,eeg,extraction,meeg,meg
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: antropy
Requires-Dist: dash
Requires-Dist: fooof
Requires-Dist: h5py
Requires-Dist: imageio
Requires-Dist: joblib
Requires-Dist: mne
Requires-Dist: mne-bids
Requires-Dist: netcdf4
Requires-Dist: networkx
Requires-Dist: neurokit2
Requires-Dist: pandas
Requires-Dist: plotly
Requires-Dist: pybv>=0.7.3
Requires-Dist: pydantic
Requires-Dist: pyyaml
Requires-Dist: structlog
Requires-Dist: xarray
Requires-Dist: xpublish
Provides-Extra: dev
Requires-Dist: black>=24.3; extra == 'dev'
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: docs
Requires-Dist: furo>=2024.1.29; extra == 'docs'
Requires-Dist: matplotlib>=3.7; extra == 'docs'
Requires-Dist: myst-parser>=2.0; extra == 'docs'
Requires-Dist: sphinx-autoapi; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints>=2.1; extra == 'docs'
Requires-Dist: sphinx-gallery>=0.16; extra == 'docs'
Requires-Dist: sphinx>=7.3; extra == 'docs'
Provides-Extra: test
Requires-Dist: pytest-cov>=5.0; extra == 'test'
Requires-Dist: pytest>=8.2; extra == 'test'
Requires-Dist: textual>=0.70.0; extra == 'test'
Provides-Extra: tui
Requires-Dist: textual>=0.70.0; extra == 'tui'
Description-Content-Type: text/markdown

[![CI](https://img.shields.io/github/actions/workflow/status/yjmantilla/neurodags/ci.yml?branch=main&label=tests)](https://github.com/yjmantilla/neurodags/actions?query=workflow%3ACI)
[![Docs](https://img.shields.io/github/actions/workflow/status/yjmantilla/neurodags/docs.yml?branch=main&label=docs)](https://yjmantilla.github.io/neurodags/)
[![codecov](https://img.shields.io/codecov/c/github/yjmantilla/neurodags)](https://app.codecov.io/gh/yjmantilla/neurodags)
[![PyPI](https://img.shields.io/pypi/v/neurodags)](https://pypi.org/project/neurodags/)

# NeuroDAGs

**An Extensible and Declarative DAG Framework for Reproducible Neuroscience Workflows**

M/EEG studies generate many interdependent intermediate derivatives. Recomputing full pipelines is wasteful; reusing valid intermediates is non-trivial. Large-scale studies require reproducible, extensible, and efficient workflows. NeuroDAGs addresses this with a declarative, graph-based framework for scalable and reusable derivative computation.

 **[Docs](https://yjmantilla.github.io/neurodags/)** | **[Comparison with Snakemake/Pydra](docs/COMPARISON.md)** | **[Poster BRaIN Symposium 2026 Montreal](https://canva.link/c2b2fm0mk2wotwq)**

## Core Idea

Pipelines are defined as a **directed acyclic graph (DAG)** of computation nodes that output reusable derivatives, executed for each input file.

## Design Principles

- **Reproducible, transparent workflows** defined declaratively in YAML — version-controllable and LLM-friendly.
- **Uniform node abstraction** — preprocessing, features, and any custom nodes are treated identically.
- **Directory-agnostic** — outputs mirror inputs' organization. Derivatives are labeled with a `@DerivativeName` suffix.
- **xarray-centered outputs** — derivatives stored as language-agnostic, metadata-rich, dimension-aware xarray → NetCDF.
- **Graph-based reuse** — if a derivative is already computed and `overwrite=False`, it is skipped automatically.

## Features

- Agnostic to data organization / directory hierarchy
- SLURM / HPC friendly with file-level parallelism via joblib
- Graph-based caching: skip already-computed derivatives
- Extensible node system — add nodes without forking the package
- YAML-based declarative configuration
- Unified CLI: `neurodags run`, `dry-run`, `dataframe`, `dag`, `view`, `validate`, `tui`
- Built-in Terminal User Interface (TUI) for pipeline management and execution
- Built-in nodes for preprocessing, spectral analysis, entropy, complexity, and data transformations
- Dataframe assembly (wide or long format) from derivative artifacts
- Dry-run mode — inspect planned computations without executing
- Built-in Dash-Plotly explorer for `.fif` and `.nc` files

## Installation

```bash
pip install neurodags
# Or with TUI support
pip install neurodags[tui]
```

With [uv](https://docs.astral.sh/uv/) (recommended):

```bash
uv add neurodags
# Or with TUI support
uv add neurodags[tui]
```

## Quickstart

See the [quickstart example](https://yjmantilla.github.io/neurodags/auto_examples/plot_quickstart_synthetic.html) — full synthetic pipeline, no real data required.

## CLI Reference

NeuroDAGs installs a unified `neurodags` command. Global flags (placed before the subcommand) apply everywhere:

```bash
neurodags --log-level WARNING run pipeline.yml      # suppress INFO output
neurodags --log-file run.jsonl run pipeline.yml     # also write logs to JSONL file
```

All subcommands accept `-d/--datasets <path>` to override the datasets YAML defined in the pipeline file.

The JSONL log file loads directly as a dataframe:

```python
import pandas as pd
df = pd.read_json("run.jsonl", lines=True)
```

### Validation

```bash
neurodags validate pipeline.yml              # load config, print datasets / derivatives summary
neurodags validate pipeline.yml -d alt.yml   # override datasets
```

### Execution

```bash
neurodags run pipeline.yml                          # run all derivatives in DerivativeList
neurodags run pipeline.yml --derivative CleanedEEG  # run a specific derivative
neurodags run pipeline.yml --derivative A --derivative B  # run multiple

# parallelism
neurodags run pipeline.yml --n-jobs 4          # 4 workers
neurodags run pipeline.yml --n-jobs -1         # all cores
neurodags run pipeline.yml --n-jobs 4 --joblib-backend loky --joblib-prefer processes

# subset / error control
neurodags run pipeline.yml --max-files-per-dataset 10
neurodags run pipeline.yml --only-index 0 5 12   # process only these file indices
neurodags run pipeline.yml --skip-errors          # skip files with a prior .error marker
neurodags run pipeline.yml --raise-on-error       # stop on first failure
```

### Dry Run

Inspect the execution plan without running any nodes. Returns a CSV/Parquet describing
each file, derivative, and whether the output is already cached.

```bash
neurodags dry-run pipeline.yml                                   # all derivatives
neurodags dry-run pipeline.yml --derivative CleanedEEG           # one derivative
neurodags dry-run pipeline.yml --output plan.csv                 # save to CSV
neurodags dry-run pipeline.yml --output plan.parquet             # or Parquet
neurodags dry-run pipeline.yml --n-jobs 4                        # parallel dry-run
neurodags dry-run pipeline.yml --skip-errors                     # exclude errored files from plan
```

### Status

Quick summary of done / missing / errored counts per derivative — no CSV needed.

```bash
neurodags status pipeline.yml                        # summary table
neurodags status pipeline.yml --derivative Alpha     # filter to one derivative
neurodags status pipeline.yml --list-errors          # print errored file paths + .error paths
neurodags status pipeline.yml --list-missing         # print missing file paths
neurodags status pipeline.yml --list-errors --list-missing
neurodags status pipeline.yml --n-jobs 4             # parallelize underlying dry-run
neurodags status pipeline.yml --format json          # machine-readable JSON
```

Exit code `0` only when all derivatives are complete (no missing, no errored); `1` otherwise.

### Source File Count

```bash
neurodags count-inputs pipeline.yml                         # number of source (input) files the pipeline will process
neurodags count-inputs pipeline.yml --derivative CleanedEEG # count for a specific derivative
```

### Dataframe Assembly

```bash
neurodags dataframe pipeline.yml --format wide --output features.csv
neurodags dataframe pipeline.yml --format long --output features.parquet
neurodags dataframe pipeline.yml --include-derivative PowerSpectrum --include-derivative BandPower
neurodags dataframe pipeline.yml --max-files-per-dataset 5
neurodags dataframe pipeline.yml --n-jobs 4   # parallel file-level collection
neurodags dataframe pipeline.yml --n-jobs -1  # all cores
```

### DAG Visualization

```bash
neurodags dag pipeline.yml                                        # print Mermaid text
neurodags dag pipeline.yml --html pipeline_dag.html               # export to HTML (ELK layout by default)
neurodags dag pipeline.yml --html pipeline_dag.html --open        # export and open in browser
neurodags dag pipeline.yml --derivative CleanedEEG --html d.html  # node-level DAG for one derivative
neurodags dag pipeline.yml --html pipeline_dag.html --layout dagre # offline fallback (no CDN)
```

HTML output uses the ELK layout engine by default — orthogonal edge routing with crossing minimisation, significantly cleaner than curved edges for dense pipelines. Use `--layout dagre` for offline environments.

### File Explorer

```bash
neurodags view path/to/file.fif   # launch Dash-Plotly explorer for MNE .fif files
neurodags view path/to/file.nc    # launch Dash-Plotly explorer for xarray NetCDF files
```

### SLURM / HPC Scripts

Generate ready-to-submit SLURM array job scripts:

```bash
neurodags slurm-script pipeline.yml                              # per-file pattern (default)
neurodags slurm-script pipeline.yml --pattern flat               # file × derivative flat array
neurodags slurm-script pipeline.yml --pattern chained            # chained per-derivative arrays
neurodags slurm-script pipeline.yml --output run_array.sh        # write to file
neurodags slurm-script pipeline.yml --derivative CleanedEEG      # restrict to specific derivatives
```

See [HPC guide](https://yjmantilla.github.io/neurodags/user_guide/hpc.html) for details on each pattern.

### TUI (Terminal User Interface)

Requires `pip install neurodags[tui]`:

```bash
neurodags tui                          # launch empty TUI, load config interactively
neurodags tui pipeline.yml             # launch with config pre-loaded
neurodags tui pipeline.yml -d alt.yml  # with datasets override
```

## Development

```bash
git clone https://github.com/yjmantilla/neurodags
cd neurodags
uv sync --all-extras --all-groups # creates .venv and installs all deps incl. dev/test/docs
uv run pre-commit install
```

Key commands (all via `uv run`):

```bash
uv run ruff check src/              # lint  (fix: uv run ruff check src/ --fix)
uv run black --check .              # format check  (fix: uv run black .)
uv run pytest -q                    # run tests
uv run pytest -s -q --no-cov --pdb  # debug a failing test

uv run sphinx-build -b html docs docs/_build/html -W --keep-going  # build docs
rm -rf docs/_build                                                   # clean docs
```

> **No uv?** Install it with `pip install uv` or `curl -Ls https://astral.sh/uv/install.sh | sh`.
> All commands above work with plain `python`/`pip` too — swap `uv run` → activate `.venv`, `uv sync` → `pip install -e .[dev,test,docs]`.

## Project Structure

```
my_project/
├── datasets.yml      # Dataset sources and paths
├── pipeline.yml      # Derivative definitions and execution list
└── custom_nodes.py   # Optional custom node definitions
```

## Quick Example

**`datasets.yml`**
```yaml
my_dataset:
  name: MyDataset
  file_pattern:
    local: data/**/*.vhdr
    hpc: /cluster/BIDS/**/*.vhdr
  derivatives_path:
    local: outputs/
    hpc: /cluster/scratch/out
```

**`pipeline.yml`**
```yaml
datasets: datasets.yml
mount_point: local
new_definitions: custom_nodes.py  # optional

DerivativeDefinitions:
  CleanedEEG:
    nodes:
      - id: 0
        derivative: SourceFile
      - id: 1
        node: basic_preprocessing
        args:
          mne_object: id.0
          resample: 256
          filter_args:
            l_freq: 0.5
            h_freq: 110

  PowerSpectrum:
    for_dataframe: True
    nodes:
      - id: 0
        derivative: CleanedEEG.fif
      - id: 1
        node: mne_spectrum_array
        args:
          meeg: id.0
          method: multitaper

DerivativeList:
  - CleanedEEG
  - PowerSpectrum
```

**Python**
```python
from neurodags.loaders import load_configuration
from neurodags.orchestrators import run_pipeline

config = load_configuration("pipeline.yml")

# Run all derivatives in "DerivativeList", auto-sorted by dependency order
run_pipeline(config)

# Or run specific ones (also sorted by dependency order)
run_pipeline(config, derivatives=["CleanedEEG"])
```

**CLI**
```bash
neurodags validate pipeline.yml

# Run all derivatives in DerivativeList (dependency-sorted)
neurodags run pipeline.yml

# Or run specific ones
neurodags run pipeline.yml --derivative CleanedEEG
```

## Custom Nodes

Add nodes without modifying or forking the package:

```python
# custom_nodes.py
from neurodags.nodes import register_node
from neurodags.definitions import Artifact, NodeResult

@register_node
def my_node(data) -> NodeResult:
    result = compute(data)
    return NodeResult(
        artifacts={
            ".nc": Artifact(
                item=result,
                writer=lambda path: result.to_netcdf(path),
            ),
        },
    )
```

Key rules:
1. A node is a function decorated with `@register_node`.
2. It returns a `NodeResult`.
3. A `NodeResult` contains `artifacts` — a dict mapping file extension to `Artifact(item, writer)`.

## Dataframe Assembly

```python
from neurodags.orchestrators import build_derivative_dataframe

df = build_derivative_dataframe("pipeline.yml", output_format="wide")
```

Derivatives marked `for_dataframe: True` are collected automatically. Supports `"wide"` (one row per file) and `"long"` (one row per value) formats.

CLI equivalent:

```bash
neurodags dataframe pipeline.yml --format wide --output derivative_dataframe.csv
```

## Parallel Execution

```yaml
# pipeline.yml
n_jobs: 4           # -1 = all cores, 1 or null = serial
joblib_backend: loky
joblib_prefer: processes
```

Or via Python:

```python
run_pipeline(config, derivatives=["MyDerivative"], n_jobs=4)
```

Or via CLI:

```bash
neurodags run pipeline.yml --derivative MyDerivative --n-jobs 4
```

## Visualization

```bash
neurodags view path/to/file.fif
neurodags view path/to/file.nc

# Alternative module entry point
python -m neurodags.visualization path/to/file.fif
python -m neurodags.visualization path/to/file.nc
```

Built-in Dash-Plotly explorer with dimension-aware UI — dropdown per axis, plot types: Line, Scatter, Bar, Heatmap.

## Inspection (Dry Run)

```python
# All derivatives in DerivativeList
run_pipeline(config, dry_run=True)

# Or a specific one
run_pipeline(config, derivatives=["MyDerivative"], dry_run=True)
```

Returns a dataframe describing the execution plan without running any nodes. When a node fails, a `.error` marker file is written with the error message — failed files are retried on the next run. If a retry succeeds, the `.error` marker is automatically removed.

CLI equivalent:

```bash
# All derivatives in DerivativeList
neurodags dry-run pipeline.yml --output dry_run_results.csv

# Or a specific one
neurodags dry-run pipeline.yml --derivative MyDerivative --output dry_run_results.csv
```

## Derivative Flags

| Flag | Default | Description |
|------|---------|-------------|
| `save` | `True` | Persist artifacts to disk. `False` = compute but don't write. |
| `overwrite` | `False` | Force recompute even if output exists. |
| `for_dataframe` | `False` | Include this derivative in `build_derivative_dataframe`. |

## Custom Node Definitions

Point `new_definitions` to one or more Python files:

```yaml
new_definitions:
  - custom_nodes/my_nodes.py
  - /abs/path/to/other_nodes.py
```

Relative paths are resolved from the pipeline YAML location.

## Documentation

[https://yjmantilla.github.io/neurodags/](https://yjmantilla.github.io/neurodags/)

## HDF5 / NetCDF Note

If you encounter `RuntimeError: NetCDF: HDF error`:

```bash
uv run pip install --no-binary=h5py h5py
# or without uv:
pip install --no-binary=h5py h5py
```

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md).

## License

MIT. See [`LICENSE`](LICENSE).
