Metadata-Version: 2.4
Name: hedtools
Version: 1.1.1
Summary: HED validation, summary, and analysis tools for annotating events and experimental metadata.
Author: VisLab, Ian Callanan, Monique Dennisen, Jeremy Cockfield, Alexander Jones, Owen Winterberg
Author-email: Kay Robbins <Kay.Robbins@utsa.edu>
Maintainer-email: Kay Robbins <Kay.Robbins@utsa.edu>
License: MIT
Project-URL: Homepage, https://www.hedtags.org/
Project-URL: Documentation, https://www.hedtags.org/hed-python/
Project-URL: Repository, https://github.com/hed-standard/hed-python/
Project-URL: Bug Tracker, https://github.com/hed-standard/hed-python/issues
Project-URL: Changelog, https://github.com/hed-standard/hed-python/blob/main/CHANGELOG.md
Project-URL: HED Resources, https://www.hedtags.org/hed-resources/
Keywords: HED,hierarchical event descriptors,validation,neuroscience,BIDS,events,metadata,annotation
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0.0
Requires-Dist: click-option-group>=0.5.0
Requires-Dist: defusedxml>=0.7.1
Requires-Dist: inflect>=7.5.0
Requires-Dist: numpy>=2.0.2
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas<4.0.0,>=2.2.3
Requires-Dist: portalocker>=3.1.1
Requires-Dist: semantic-version>=2.10.0
Requires-Dist: typeguard>=4.5.2
Provides-Extra: dev
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: typos>=1.29.0; extra == "dev"
Requires-Dist: mdformat>=0.7.0; extra == "dev"
Requires-Dist: mdformat-myst>=0.1.5; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx<10.0,>=7.1.0; extra == "docs"
Requires-Dist: furo>=2024.1.29; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5.2; extra == "docs"
Requires-Dist: myst-parser>=3.0.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.24.0; extra == "docs"
Requires-Dist: linkify-it-py>=2.0.3; extra == "docs"
Provides-Extra: test
Requires-Dist: coverage>=7.0.0; extra == "test"
Provides-Extra: examples
Requires-Dist: jupyter>=1.0.0; extra == "examples"
Requires-Dist: notebook>=7.0.0; extra == "examples"
Requires-Dist: nbformat>=5.9.0; extra == "examples"
Requires-Dist: nbconvert>=7.0.0; extra == "examples"
Requires-Dist: ipykernel>=6.25.0; extra == "examples"
Dynamic: license-file

![PyPI - Status](https://img.shields.io/pypi/v/hedtools) ![Python3](https://img.shields.io/badge/python-%3E=3.10-yellow.svg) [![Maintainability](https://qlty.sh/gh/hed-standard/projects/hed-python/maintainability.svg)](https://qlty.sh/gh/hed-standard/projects/hed-python) [![Code Coverage](https://qlty.sh/gh/hed-standard/projects/hed-python/coverage.svg)](https://qlty.sh/gh/hed-standard/projects/hed-python) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8056010.svg)](https://doi.org/10.5281/zenodo.8056010) [![Docs](https://img.shields.io/badge/docs-hed--python-blue.svg)](https://www.hedtags.org/hed-python)

# HEDTools - Python

> Python tools for validation, analysis, and transformation of HED (Hierarchical Event Descriptors) tagged datasets.

## Overview

HED (Hierarchical Event Descriptors) is a framework for systematically describing both laboratory and real-world events as well as other experimental metadata. HED tags are comma-separated path strings that provide a standardized vocabulary for annotating events and experimental conditions.

**Key Features:**

- Validate HED annotations against schema specifications
- Analyze and summarize HED-tagged datasets
- Full HED support in BIDS (Brain Imaging Data Structure)
- HED support in NWB (Neurodata Without Borders) when used the [ndx-hed](https://github.com/hed-standard/ndx-hed) extension.
- Platform-independent and data-neutral
- Command-line tools and Python API

**Note:** Table remodeling tools have been moved to a separate package. See [table-remodeler](https://pypi.org/project/table-remodeler/) on PyPI or visit [https://www.hedtags.org/table-remodeler](https://www.hedtags.org/table-remodeler) for more information.

## Quick start

### Online tools (no installation required)

For simple validation or transformation tasks, use the online tools at [https://hedtools.org/hed](https://hedtools.org/hed) - no installation needed!

Browser-based validation (no data upload) is available at [https://www.hedtags.org/hed-web](https://www.hedtags.org/hed-web)

A development version of the online tools is available at: [https://hedtools.org/hed_dev](https://hedtools.org/hed_dev)

### Python installation

**Requirements:** Python 3.10 or higher

Install from PyPI:

```bash
pip install hedtools
```

Or install from GitHub (latest):

```bash
pip install git+https://github.com/hed-standard/hed-python/@main
```

### Development installation

For development work or to access optional features, install from the cloned repository:

```bash
# Clone the repository
git clone https://github.com/hed-standard/hed-python.git
cd hed-python

# Install in editable mode with base dependencies
pip install -e .

# Install with optional dependency groups
pip install -e ".[dev]"       # Development tools (ruff, typos)
pip install -e ".[docs]"      # Documentation tools (sphinx, furo)
pip install -e ".[test]"      # Testing tools (coverage)
pip install -e ".[examples]"  # Jupyter notebook support

# Install all optional dependencies
pip install -e ".[dev,docs,test,examples]"
```

**Optional dependency groups:**

- `dev` - Code quality tools: ruff (linter + formatter), typos, mdformat
- `docs` - Documentation generation: sphinx, furo theme, myst-parser
- `test` - Code coverage reporting: coverage
- `examples` - Jupyter notebook support: jupyter, notebook, ipykernel

### Basic usage

```python
from hed import HedString, get_printable_issue_string, load_schema_version


# Load the latest HED schema
schema = load_schema_version("8.4.0")

# Create and validate a HED string
hed_string = HedString("Sensory-event, Visual-presentation, (Red, Square)", schema)
issues = hed_string.validate()

if issues:
    print(get_printable_issue_string(issues, title="Validation issues found"))
else:
    print("HED string is valid!")
```

### Command-line tools

HEDTools provides a unified command-line interface with git-like subcommands:

```bash
# Main command (new unified interface)
hedpy --help

# Validate a BIDS dataset
hedpy validate bids-dataset /path/to/bids/dataset

# Validate a HED string
hedpy validate string "Sensory-event, (Red, Square)" -sv 8.4.0

# Extract sidecar template from a BIDS dataset
hedpy extract bids-sidecar /path/to/dataset

# Extract column-value summary from a set of TSV files
hedpy extract tabular-summary /path/to/data

# Validate HED schemas
hedpy schema validate /path/to/schema.xml

# Convert schema between formats (XML, MEDIAWIKI, TSV, JSON)
hedpy schema convert /path/to/schema.xml

# Add HED IDs to a schema under development
hedpy schema add-ids /path/to/hed-schemas standard 8.4.0
```

**Legacy commands** (deprecated, use `hedpy` instead):

```bash
validate_bids /path/to/dataset
hed_validate_schemas /path/to/schema.xml
```

**Note:** The `run_remodel` command has been removed. Table remodeling functionality is now available in the separate [table-remodeler](https://pypi.org/project/table-remodeler/) package.

**Note:** The visualization tools such as the word cloud visualization have been moved to a separate [hed-vis](https://pypi.org/project/hedvis/) project.

For more examples, see the [user guide](https://www.hedtags.org/hed-python/user_guide.html).

### Jupyter notebook examples

**Note:** Example notebooks are available in the [GitHub repository](https://github.com/hed-standard/hed-python/tree/main/examples) only, not in the PyPI package.

The [`examples/`](examples/) directory contains Jupyter notebooks demonstrating common HED workflows with BIDS datasets:

- **validate_bids_dataset.ipynb** - Validate HED annotations in a BIDS dataset
- **validate_bids_dataset_nondefault.ipynb** - Validate a BIDS dataset that has HED in files other than `events.tsv`
- **validate_bids_datasets.ipynb** - Batch-validate a list of BIDS datasets
- **validate_bids_dataset_with_libraries.ipynb** - Validate with HED library schemas
- **summarize_events.ipynb** - Summarize event file contents and value counts
- **sidecar_to_spreadsheet.ipynb** - Convert JSON sidecars to spreadsheet format
- **merge_spreadsheet_into_sidecar.ipynb** - Merge spreadsheet annotations into JSON sidecars
- **extract_json_template.ipynb** - Generate JSON sidecar templates from event files
- **find_event_combinations.ipynb** - Find unique combinations of event values

To use these notebooks:

```bash
# Clone the repository to get the examples
git clone https://github.com/hed-standard/hed-python.git
cd hed-python

# Install HEDTools with Jupyter support
pip install -e .[examples]

# Launch Jupyter
jupyter notebook examples/
```

See [`examples/README.md`](examples/README.md) for more details.

## Experimental features

**String-based search** (`hed.models.string_search` and `hed.models.schema_lookup`) is a middle-ground search facility that operates on raw HED strings without requiring pre-parsed `HedString` objects or a loaded schema. It supports the full `QueryHandler` query syntax (logical operators, grouping, wildcards) and optionally accepts a pre-generated schema lookup dictionary to enable ancestor-aware matching on short-form strings.

> **This facility is still experimental.** Its API — including class names, function signatures, and module structure — may change in future releases without notice. These modules are **not** exported from the top-level `hed` package; import them directly:
>
> ```python
> from hed.models.string_search import StringQueryHandler, string_search
> from hed.models.schema_lookup import generate_schema_lookup
> ```

See the [search details documentation](https://www.hedtags.org/hed-python/search_details.html) for a full comparison of all three search implementations and performance benchmarks.

## Documentation

📖 **Full Documentation:** [https://www.hedtags.org/hed-python](https://www.hedtags.org/hed-python)

- [User guide](https://www.hedtags.org/hed-python/user_guide.html) - Usage instructions
- [API reference](https://www.hedtags.org/hed-python/api/index.html) - Detailed API documentation
- [HED specification](https://www.hedtags.org/hed-specification) - Full HED standard specification

### Building docs locally

```bash
# Install documentation dependencies
pip install -e .[docs]

# Build the documentation
cd docs
sphinx-build -b html . _build/html
```

To view the built documentation open `docs/_build/html/index.html` in your browser

### Code Formatting

This project uses [ruff format](https://docs.astral.sh/ruff/formatter/) for consistent code formatting.

```bash
# Check if code is properly formatted (without making changes)
ruff format --check .

# Check and show what would change
ruff format --check --diff .

# Format all Python code in the repository
ruff format .

# Format specific files or directories
ruff format hed/
ruff format tests/
```

**Configuration:** Formatter settings are in `pyproject.toml` under `[tool.ruff.format]` with a line length of 120 characters.

**Exclusions:** Ruff automatically excludes `.venv/`, `__pycache__/`, auto-generated files (`hed/_version.py`), and external repos (`spec_tests/hed-examples/`, `spec_tests/hed-schemas/`).

**CI integration:** All code is automatically checked for formatting in GitHub Actions. Run `ruff format .` before committing to ensure your code passes CI checks.

## Related repositories

The HED ecosystem consists of several interconnected repositories:

| Repository                                                             | Description                                      |
| ---------------------------------------------------------------------- | ------------------------------------------------ |
| [hed-python](https://github.com/hed-standard/hed-python)               | Python validation and analysis tools (this repo) |
| [hed-server](https://github.com/hed-standard/hed-server)               | Web interface and deployable Docker services     |
| [hed-resources](https://github.com/hed-standard/hed-resources)         | Tutorials and other HED resources                |
| [hed-specification](https://github.com/hed-standard/hed-specification) | Official HED specification documents             |
| [hed-schemas](https://github.com/hed-standard/hed-schemas)             | Official HED schema repository                   |
| [table-remodeler](https://github.com/hed-standard/table-remodeler)     | Table transformation and remodeling tools        |
| [hed-vis](https://github.com/hed-standard/hed-vis)                     | HED visualization tools (word clouds, etc.)      |
| [ndx-hed](https://github.com/hed-standard/ndx-hed)                     | HED support for NWB                              |
| [hed-javascript](https://github.com/hed-standard/hed-javascript)       | JavaScript HED validation tools                  |
| [hed-web](https://github.com/hed-standard/hed-web)                     | Source for browser-based validation tools        |

## Contributing

We welcome contributions! Here's how you can help:

1. **Report issues:** Use [GitHub Issues](https://github.com/hed-standard/hed-python/issues) for bug reports and feature requests
2. **Submit pull requests (PRs):** PRs should be from a non-main fork and target the `main` branch
3. **Improve documentation:** Help us make HED easier to use
4. **Share examples:** Contribute example code and use cases

**Development setup:**

```bash
# Clone the repository
git clone https://github.com/hed-standard/hed-python.git
cd hed-python

# Install in development mode with all dependencies
pip install -e .[dev,test,docs,examples]

# Or just core + test dependencies
pip install -e .[test]

# Run tests
python -m unittest discover tests

# Run specific test file
python -m unittest tests/path/to/test_file.py

# Test notebooks (requires examples dependencies)
python -m unittest tests.test_notebooks
```

For detailed contribution guidelines, please see [CONTRIBUTING.md](CONTRIBUTING.md).

## Configuration

### Schema caching

By default, HED schemas are cached in `~/.hedtools/` (location varies by OS).

```python
# Change the cache directory
import hed
hed.schema.set_cache_directory('/custom/path/to/cache')
```

Starting with `hedtools 0.2.0`, local copies of recent schema versions are bundled within the package for offline access.

## Citation

If you use HEDTools in your research, please cite:

```bibtex
@software{hedtools,
  author = {Callanan, Ian and Robbins, Kay and others},
  title = {HEDTools: Python tools for HED},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/hed-standard/hed-python},
  doi = {10.5281/zenodo.8056010}
}
```

## License

HEDTools is licensed under the MIT License. See [LICENSE](LICENSE) for details.

## Support

- HED documentation: [www.hedtags.org/hed-resources](https://www.hedtags.org/hed-resources)
- HED homepage: [www.hedtags.org](https://www.hedtags.org)
- GitHub issues: [https://github.com/hed-standard/hed-python/issues](https://github.com/hed-standard/hed-python/issues)
- Questions or ideas: [HED discussions](https://github.com/orgs/hed-standard/discussions)
- Contact: [hed-maintainers@gmail.com](mailto:hed-maintainers@gmail.com)

## Funding

Partial support for this project was provided by [NIH 1R01MH126700-01A1](https://reporter.nih.gov/project-details/10480619).
