scitex_dataset API Reference

SciTeX Dataset - Unified interface for scientific dataset discovery.

Domains: - neuroscience: OpenNeuro, DANDI, PhysioNet - general: Scientific Data, Zenodo, Figshare, OpenML, HuggingFace - biology: GEO (Gene Expression Omnibus) - pharmacology: ChEMBL, MoleculeNet - medical: ClinicalTrials.gov - ai-for-science: CORE-Bench, BixBench, BioMysteryBench (agentic

reproducibility / bioinformatics / biology cohorts — download | prepare | mask verb tree)

Usage:

>>> from scitex_dataset import neuroscience
>>> datasets = neuroscience.fetch_all_datasets(max_datasets=10)

>>> # Or direct import for convenience
>>> from scitex_dataset import fetch_all_datasets, search_datasets

>>> # Local database for fast searching
>>> from scitex_dataset import database as db
>>> db.build()  # Fetch all sources and index
>>> results = db.search("alzheimer EEG", min_subjects=20)

>>> # Prepare an agentic benchmark (mask only — safe, fast)
>>> from scitex_dataset import ai_for_science
>>> paths = ai_for_science.resolve_paths("corebench")
>>> ai_for_science.corebench.mask(
...     raw_dir=paths.raw_dir,
...     masked_dir=paths.masked_dir,
... )

scitex_dataset.db_build(sources=None, db_path=None, logger=None)

Build the local database from all sources.

Parameters:

sources (list, optional) – Sources to fetch: [“openneuro”, “dandi”, “physionet”]. Default: all sources.
db_path (Path, optional) – Database file path. Default: $SCITEX_DIR/dataset/runtime/datasets.db (~/.scitex/dataset/runtime/datasets.db when SCITEX_DIR is unset).
logger (optional) – Logger for progress messages.

Returns:

Count of datasets indexed per source.

Return type:

dict

scitex_dataset.db_search(query=None, source=None, modality=None, min_subjects=None, max_subjects=None, min_downloads=None, has_readme=False, limit=50, offset=0, order_by='downloads', db_path=None)

Search the local database.

Parameters:

query (str, optional) – Full-text search query (searches name, readme, tasks).
source (str, optional) – Filter by source: “openneuro”, “dandi”, “physionet”.
modality (str, optional) – Filter by modality (e.g., “mri”, “eeg”).
min_subjects (int, optional) – Minimum number of subjects.
max_subjects (int, optional) – Maximum number of subjects.
min_downloads (int, optional) – Minimum download count.
has_readme (bool) – Only include datasets with readme.
limit (int) – Maximum results (default: 50).
offset (int) – Skip first N results (for pagination).
order_by (str) – Order by: downloads, views, n_subjects, size_gb, name.
db_path (Path, optional) – Database file path.

Returns:

List of matching datasets.

Return type:

list

scitex_dataset.db_show_stats(db_path=None)

Get database statistics.

Returns:: Statistics including counts per source, last build time, etc.
Return type:: dict

scitex_dataset.filter_results(datasets, **kwargs)[source]: Filter and rank dataset dicts — matches dataset_filter_results MCP tool.

scitex_dataset.list_sources()[source]

Return the 11-source registry — matches dataset_list_sources MCP tool.

Return type:: dict

scitex_dataset.openneuro_fetch(batch_size=100, max_datasets=None, logger=None)

Fetch every dataset record from OpenNeuro by paginating GraphQL.

Walks the public crn/graphql endpoint with cursor-based pagination until exhausted (or max_datasets is reached). Use format_dataset to project each raw record into the package’s common dataset schema.

Parameters:

batch_size (int, default 100) – Records per HTTP request. The OpenNeuro server caps this; the function does not validate the upper bound.
max_datasets (int, optional) – Stop after this many records. None (default) fetches the entire catalog.
logger (logging.Logger, optional) – If provided, HTTP and GraphQL errors are logged. Errors are otherwise silent (the function returns whatever it has so far).

Returns:

Raw GraphQL node dicts, in catalog order. Pass each through format_dataset for the normalized schema.

Return type:

list[dict]

Examples

>>> records = fetch_all_datasets(max_datasets=10)
>>> len(records) <= 10
True

scitex_dataset.dandi_fetch(max_datasets=None, page_size=100, logger=None)

Fetch all dandisets from DANDI Archive with pagination.

Return type:: list[dict]

scitex_dataset.physionet_fetch(max_datasets=None, logger=None)

Fetch all databases from PhysioNet with pagination.

Return type:: list[dict]

scitex_dataset.gin_fetch(max_datasets=None, batch_size=50, logger=None)

Walk GIN’s repo catalog by paginating /repos/search.

Mirrors the OpenNeuro / DANDI conventions so the unified search and database modules can index GIN uniformly.

Return type:: list[dict]

scitex_dataset.gin_search(query='', limit=50, page=1, logger=None)[source]

Search GIN repos via /api/v1/repos/search.

Parameters:

query (str) – Free-text query. Empty string returns the global feed.
limit (int, default 50) – Server-side page size.
page (int, default 1) – Page number, 1-indexed.
logger (logging.Logger, optional) – Errors are logged here; otherwise silent (returns empty list).

Returns:

Raw repo records as returned by GIN. Run each through gin_format() for the package’s normalized schema.

Return type:

list[dict]

scitex_dataset.gin_info(repo_id, logger=None)[source]

Fetch metadata for one GIN repo via /api/v1/repos/{owner}/{repo}.

Returns the empty dict on HTTP error (logged if logger given).

Return type:: dict

scitex_dataset.gin_download(repo_id, local_dir=None, files=None, branch='master', prefer='auto', metadata_only=False, first_only=False, logger=None)[source]

Download a GIN repo (metadata + annexed content).

The default flow:

git clone https://gin.g-node.org/<owner>/<repo>.git — pulls metadata and annex pointer text files (~MB; no auth).
For each file matching files (default: every file under the repo whose worktree content is an annex pointer), stream the bytes from /raw/<branch>/<path> and replace the pointer atomically.

Parameters:

repo_id (str) – owner/repo GIN identifier (e.g. "USZ_NCH/Human_MTL_units_scalp_EEG_and_iEEG_verbal_WM").
local_dir (str | Path, optional) – Local destination. Falls back to the Spartan project FS or <scope-root>/runtime/gin/<owner>__<repo>/ per _resolve_local_dir().
files (iterable of str, optional) – Repo-relative paths to fetch. None (default) fetches every annex pointer encountered.
branch (str, default "master") – Source branch for the /raw/ URL.
prefer ({“auto”, “https”, “datalad”}, default "auto") – Backend selector. "datalad" requires the optional datalad extra; "auto" uses datalad if importable, else "https". No silent fallback: an explicit "datalad" request raises if the optional dep is missing.
metadata_only (bool, default False) – Skip annex content; just clone the repo.
first_only (bool, default False) – Fetch only the FIRST pointer encountered (smoke-test path).
logger (logging.Logger, optional) – Per-step progress is logged here if given.

Returns:

``{“repo_dir”: Path, “files”: [{“path”: Path, “bytes”: int}, …],: ”total_bytes”: int, “backend”: str}``.

Return type:

dict

Raises:

ImportError – If prefer="datalad" is explicit and the optional dep is missing; the install hint can be read via scitex_dev.last_install_hint().

scitex_dataset.zenodo_fetch(query='', max_datasets=None, page_size=25, type_filter='dataset', logger=None)

Fetch all datasets from Zenodo with pagination.

Parameters:

query (str) – Search query string.
max_datasets (int, optional) – Maximum number of datasets to fetch.
page_size (int) – Datasets per request.
type_filter (str) – Resource type filter (default: ‘dataset’).
logger (optional) – Logger for progress messages.

Returns:

List of raw record dictionaries.

Return type:

list[dict]

scitex_dataset.figshare_fetch(query='', max_datasets=None, page_size=25, logger=None)

Fetch all datasets from Figshare with pagination.

Parameters:

query (str) – Search query string.
max_datasets (int, optional) – Maximum number of datasets to fetch.
page_size (int) – Datasets per request.
logger (optional) – Logger for progress messages.

Returns:

List of raw article dictionaries.

Return type:

list[dict]

scitex_dataset.openml_fetch(max_datasets=None, page_size=100, logger=None)

Fetch all datasets from OpenML with pagination.

Parameters:

max_datasets (int, optional) – Maximum number of datasets to fetch.
page_size (int) – Datasets per request.
logger (optional) – Logger for progress messages.

Returns:

List of raw dataset dictionaries.

Return type:

list[dict]

scitex_dataset.moleculenet_fetch(max_datasets=None, logger=None)

Fetch all MoleculeNet datasets.

Parameters:

max_datasets (int, optional) – Maximum number of datasets to return.
logger (optional) – Logger for progress messages.

Returns:

List of MoleculeNet dataset records.

Return type:

list[dict]

scitex_dataset.geo_fetch(max_datasets=None, logger=None)

Fetch all datasets from GEO with pagination.

Return type:: list[dict]

scitex_dataset.chembl_fetch(max_datasets=None, logger=None)

Fetch all assays from ChEMBL with pagination.

Return type:: list[dict]

scitex_dataset.clinicaltrials_fetch(max_datasets=None, logger=None)

Fetch all studies from ClinicalTrials.gov with pagination.

Return type:: list[dict]

scitex_dataset.huggingface_fetch(query='', max_datasets=None, logger=None, **_unused)

Catalog-style adapter so HuggingFace can plug into database.build.

Unlike OpenNeuro/DANDI/etc., HuggingFace has no bounded catalog — query is required for meaningful results. Without one this calls search_hub("") which lists by recency up to max_datasets.

Parameters:

query (str) – Search query. Empty string lists by recency (HF default).
max_datasets (int, optional) – Cap on results. Default 1000 to avoid runaway indexing.

Return type:

List[Dict]

scitex_dataset.huggingface_search(query, limit=50)

Search for datasets on HuggingFace.

Parameters:

query (str) – Search query string.
limit (int) – Maximum number of results (default: 50).

Returns:

List of search result dictionaries with fields: id, name, description, likes, downloads, private, gated, etc.

Return type:

list[dict]

scitex_dataset.huggingface_info(repo_id, repo_type='dataset')

Get metadata about a HuggingFace dataset or model.

Parameters:

repo_id (str) – Repository ID (e.g., “username/dataset_name”).
repo_type (str) – Repository type: “dataset” (default) or “model”.

Returns:

Dataset metadata: id, name, description, downloads, likes, private, gated, size_gb, created_at, last_modified, etc.

Return type:

dict

scitex_dataset.huggingface_download_file(repo_id, filename, local_dir=None, repo_type='dataset')

Download a single file from a HuggingFace repository.

Parameters:

repo_id (str) – Repository ID (e.g., “username/dataset_name”).
filename (str) – Path within the repository (e.g., “data/train.csv”).
local_dir (str, optional) – Local directory for download. If None, uses the SciTeX runtime directory via <scope-root>/runtime/huggingface/<repo_id>/.
repo_type (str) – Repository type: “dataset” (default) or “model”.

Returns:

Path to the downloaded file.

Return type:

Path

Raises:

Exception – If download fails.

scitex_dataset.download_dataset(source, id, dest=None, **kwargs)[source]

Unified download_dataset(source, id, dest, **opts) dispatcher.

Parameters:

source (str) – Source id, case-insensitive. Matched against scitex_dataset._sources.ALL_SOURCES.
id (str) – Source-native identifier — HF org/name, GIN owner/repo, etc.
dest (str | Path, optional) – Local destination. Per-source default applies if None.
**kwargs – Forwarded to the underlying <source>_download function.

Returns:

Whatever the underlying downloader returns (path or manifest dict).

Return type:

Any

Raises:

ValueError – If source is not in the registry.
NotImplementedError – If the matched source has no download adapter wired up yet.

scitex_dataset.corebench_mask(*, raw_dir, masked_dir, **_)

Mask the oracle JSONs and build the masked view in masked_dir.

raw_dir must already contain the upstream-pristine dataset/core_train.json and core_test.json — either from a prior download(...) or hand-staged by the operator. The two record lists are concatenated (train first, then test) into a single 90-record questions.json under masked_dir, then the answer-free capsule content is symlinked alongside.

Return type:: dict

scitex_dataset.bixbench_mask(*, raw_dir, masked_dir, **_)

Read the oracle manifest, build the masked view in masked_dir.

Output is JSONL: one record per line, sort_keys=True and ensure_ascii=False for deterministic byte output across runs. Symlinks the answer-free capsule content into masked_dir and recreates the legacy BixBench_masked.jsonl backward-compat link.

Return type:: dict

scitex_dataset.biomysterybench_mask(*, raw_dir, masked_dir, **_)

Read raw_dir/problems.csv, build the masked view in masked_dir.

Writes questions.jsonl (JSONL, sort_keys=True, ensure_ascii=False — symmetric with the other benchmarks) and symlinks the answer-free upstream content (problem environments) into masked_dir so the agent gets the data without the rubric.

Return type:: dict

Search Module

Unified search interface for neuroscience datasets.

Currently supports: - OpenNeuro (BIDS neuroimaging)

Future sources: - DANDI (NWB neurophysiology) - PhysioNet (EEG/ECG/physiology) - Zenodo (general scientific)

scitex_dataset.search.search_datasets(datasets, modality=None, min_subjects=None, max_subjects=None, task_contains=None, text_query=None, min_downloads=None, has_readme=False)[source]

Filter datasets by various criteria.

Parameters:

datasets (list[dict]) – List of formatted dataset dictionaries
modality (Optional[str]) – Filter by modality (e.g., “mri”, “eeg”, “meg”)
min_subjects (Optional[int]) – Minimum number of subjects
max_subjects (Optional[int]) – Maximum number of subjects
task_contains (Optional[str]) – Filter by task name substring
text_query (Optional[str]) – Search in name and readme text
min_downloads (Optional[int]) – Minimum download count
has_readme (bool) – Only include datasets with readme

Return type:

list[dict]

Returns:

Filtered list of datasets

Example

>>> from scitex_dataset import fetch_all_datasets, format_dataset
>>> from scitex_dataset.search import search_datasets
>>> raw = fetch_all_datasets(max_datasets=100)
>>> datasets = [format_dataset(d) for d in raw]
>>> eeg_data = search_datasets(datasets, modality="eeg", min_subjects=20)

scitex_dataset.search.sort_datasets(datasets, by='downloads', descending=True)[source]

Sort datasets by a field.

Parameters:

datasets (list[dict]) – List of formatted dataset dictionaries
by (str) – Field to sort by (downloads, views, n_subjects, size_gb, created)
descending (bool) – Sort in descending order

Return type:

list[dict]

Returns:

Sorted list of datasets

Database Module

Local SQLite database for fast dataset searching.

Usage:

>>> from scitex_dataset import database as db
>>> db.build()  # Fetch all sources and build database
>>> results = db.search("alzheimer EEG", min_subjects=20)

scitex_dataset.database.build(sources=None, db_path=None, logger=None)[source]

Build the local database from all sources.

Parameters:

sources (list, optional) – Sources to fetch: [“openneuro”, “dandi”, “physionet”]. Default: all sources.
db_path (Path, optional) – Database file path. Default: $SCITEX_DIR/dataset/runtime/datasets.db (~/.scitex/dataset/runtime/datasets.db when SCITEX_DIR is unset).
logger (optional) – Logger for progress messages.

Returns:

Count of datasets indexed per source.

Return type:

dict

scitex_dataset.database.update(source, db_path=None, logger=None)[source]

Update a single source in the database.

Parameters:

source (str) – Source to update: “openneuro”, “dandi”, or “physionet”.
db_path (Path, optional) – Database file path.
logger (optional) – Logger for progress messages.

Returns:

Number of datasets indexed.

Return type:

int

scitex_dataset.database.search(query=None, source=None, modality=None, min_subjects=None, max_subjects=None, min_downloads=None, has_readme=False, limit=50, offset=0, order_by='downloads', db_path=None)[source]

Search the local database.

Parameters:

query (str, optional) – Full-text search query (searches name, readme, tasks).
source (str, optional) – Filter by source: “openneuro”, “dandi”, “physionet”.
modality (str, optional) – Filter by modality (e.g., “mri”, “eeg”).
min_subjects (int, optional) – Minimum number of subjects.
max_subjects (int, optional) – Maximum number of subjects.
min_downloads (int, optional) – Minimum download count.
has_readme (bool) – Only include datasets with readme.
limit (int) – Maximum results (default: 50).
offset (int) – Skip first N results (for pagination).
order_by (str) – Order by: downloads, views, n_subjects, size_gb, name.
db_path (Path, optional) – Database file path.

Returns:

List of matching datasets.

Return type:

list

scitex_dataset.database.get_stats(db_path=None)[source]

Get database statistics.

Returns:: Statistics including counts per source, last build time, etc.
Return type:: dict

scitex_dataset.database.get_db_path()[source]

Get the database file path.

Return type:: Path

scitex_dataset.database.clear(db_path=None)[source]

Delete the database file.

Returns:: True if deleted, False if didn’t exist.
Return type:: bool

Neuroscience Sources

OpenNeuro

OpenNeuro dataset fetcher using GraphQL API.

Example

>>> from scitex_dataset import fetch_all_datasets, format_dataset
>>> datasets = fetch_all_datasets(max_datasets=10)
>>> formatted = [format_dataset(ds) for ds in datasets]

scitex_dataset.neuroscience.openneuro.fetch_datasets(first=10, after=None)[source]

Fetch a single page of datasets from OpenNeuro.

Return type:: dict

scitex_dataset.neuroscience.openneuro.fetch_all_datasets(batch_size=100, max_datasets=None, logger=None)[source]

Fetch every dataset record from OpenNeuro by paginating GraphQL.

Walks the public crn/graphql endpoint with cursor-based pagination until exhausted (or max_datasets is reached). Use format_dataset to project each raw record into the package’s common dataset schema.

Parameters:

batch_size (int, default 100) – Records per HTTP request. The OpenNeuro server caps this; the function does not validate the upper bound.
max_datasets (int, optional) – Stop after this many records. None (default) fetches the entire catalog.
logger (logging.Logger, optional) – If provided, HTTP and GraphQL errors are logged. Errors are otherwise silent (the function returns whatever it has so far).

Returns:

Raw GraphQL node dicts, in catalog order. Pass each through format_dataset for the normalized schema.

Return type:

list[dict]

Examples

>>> records = fetch_all_datasets(max_datasets=10)
>>> len(records) <= 10
True

scitex_dataset.neuroscience.openneuro.format_dataset(node)[source]

Project a raw OpenNeuro GraphQL node into the common dataset schema.

Every catalog source exposes format_dataset returning the same shape so they can plug into database.build and search.search_datasets uniformly.

Parameters:: node (dict) – A single edges[].node element from the OpenNeuro GraphQL response (the draft / analytics keys are read; missing fields fall back to None / 0).
Returns:: Normalized record with keys: id, name, n_subjects, modalities, tasks, size_gb, downloads, views, readme, license, doi, url, source.
Return type:: dict

DANDI

DANDI Archive dataset fetcher.

DANDI (Distributed Archives for Neurophysiology Data Integration) hosts neurophysiology data in NWB (Neurodata Without Borders) format.

API: https://api.dandiarchive.org/api

Example

>>> from scitex_dataset.neuroscience import dandi
>>> datasets = dandi.fetch_all_datasets(max_datasets=10)
>>> formatted = [dandi.format_dataset(ds) for ds in datasets]

scitex_dataset.neuroscience.dandi.fetch_datasets(page=1, page_size=100, ordering='-modified')[source]

Fetch a single page of dandisets from DANDI Archive.

Return type:: dict

scitex_dataset.neuroscience.dandi.fetch_all_datasets(max_datasets=None, page_size=100, logger=None)[source]

Fetch all dandisets from DANDI Archive with pagination.

Return type:: list[dict]

scitex_dataset.neuroscience.dandi.format_dataset(dandiset)[source]

Extract and format dandiset information.

Return type:: dict

PhysioNet

PhysioNet dataset fetcher.

PhysioNet hosts physiological signal databases including EEG, ECG, EMG, and other biomedical signals.

API: https://physionet.org/api/v1/

Example

>>> from scitex_dataset.neuroscience import physionet
>>> datasets = physionet.fetch_all_datasets(max_datasets=10)
>>> formatted = [physionet.format_dataset(ds) for ds in datasets]

scitex_dataset.neuroscience.physionet.fetch_datasets(page=1)[source]

Fetch a single page of databases from PhysioNet.

Return type:: dict

scitex_dataset.neuroscience.physionet.fetch_all_datasets(max_datasets=None, logger=None)[source]

Fetch all databases from PhysioNet with pagination.

Return type:: list[dict]

scitex_dataset.neuroscience.physionet.format_dataset(database)[source]

Extract and format PhysioNet database information.

Return type:: dict

General Sources

Zenodo

Zenodo API client for scientific dataset discovery.

Zenodo is a general-purpose open repository developed under the European OpenAIRE program and operated by CERN. It allows researchers to deposit research papers, data sets, research software, reports, and any other research related digital artifacts.

API Documentation: https://developers.zenodo.org/

scitex_dataset.general.zenodo.fetch_datasets(query='', page=1, size=25, sort='mostrecent', type_filter='dataset')[source]

Fetch datasets from Zenodo.

Parameters:

query (str) – Search query string (Elasticsearch query syntax).
page (int) – Page number (1-indexed).
size (int) – Number of results per page (max 10000).
sort (str) – Sort order: ‘bestmatch’, ‘mostrecent’, ‘-mostrecent’.
type_filter (str) – Resource type filter: ‘dataset’, ‘software’, ‘publication’, etc.

Returns:

API response with ‘hits’ containing records.

Return type:

dict

scitex_dataset.general.zenodo.fetch_all_datasets(query='', max_datasets=None, page_size=25, type_filter='dataset', logger=None)[source]

Fetch all datasets from Zenodo with pagination.

Parameters:

query (str) – Search query string.
max_datasets (int, optional) – Maximum number of datasets to fetch.
page_size (int) – Datasets per request.
type_filter (str) – Resource type filter (default: ‘dataset’).
logger (optional) – Logger for progress messages.

Returns:

List of raw record dictionaries.

Return type:

list[dict]

scitex_dataset.general.zenodo.format_dataset(record)[source]

Format a Zenodo record into a standardized dataset dictionary.

Parameters:: record (dict) – Raw Zenodo record from API.
Returns:: Standardized dataset dictionary.
Return type:: dict

Figshare

Figshare API client for scientific dataset discovery.

Figshare is a repository where users can make all of their research outputs available in a citable, shareable and discoverable manner.

API Documentation: https://docs.figshare.com/

scitex_dataset.general.figshare.fetch_datasets(query='', page=1, page_size=25, order='published_date', order_direction='desc')[source]

Fetch datasets from Figshare.

Parameters:

query (str) – Search query string.
page (int) – Page number (1-indexed).
page_size (int) – Number of results per page (max 1000).
order (str) – Sort field: ‘published_date’, ‘modified_date’, ‘views’, ‘shares’.
order_direction (str) – Sort direction: ‘asc’ or ‘desc’.

Returns:

List of article records.

Return type:

list[dict]

scitex_dataset.general.figshare.fetch_all_datasets(query='', max_datasets=None, page_size=25, logger=None)[source]

Fetch all datasets from Figshare with pagination.

Parameters:

query (str) – Search query string.
max_datasets (int, optional) – Maximum number of datasets to fetch.
page_size (int) – Datasets per request.
logger (optional) – Logger for progress messages.

Returns:

List of raw article dictionaries.

Return type:

list[dict]

scitex_dataset.general.figshare.format_dataset(record)[source]

Format a Figshare article into a standardized dataset dictionary.

Parameters:: record (dict) – Raw Figshare article from API.
Returns:: Standardized dataset dictionary.
Return type:: dict

OpenML

OpenML API client for machine learning dataset discovery.

OpenML is an open platform for sharing machine learning datasets, tasks, and experiments. It hosts thousands of curated ML datasets.

API Documentation: https://www.openml.org/apis

scitex_dataset.general.openml.fetch_datasets(offset=0, limit=100, status='active')[source]

Fetch datasets from OpenML.

Parameters:

offset (int) – Offset for pagination.
limit (int) – Number of results per request (max 10000).
status (str) – Dataset status filter: ‘active’, ‘deactivated’, ‘all’.

Returns:

API response with dataset list.

Return type:

dict

scitex_dataset.general.openml.fetch_all_datasets(max_datasets=None, page_size=100, logger=None)[source]

Fetch all datasets from OpenML with pagination.

Parameters:

max_datasets (int, optional) – Maximum number of datasets to fetch.
page_size (int) – Datasets per request.
logger (optional) – Logger for progress messages.

Returns:

List of raw dataset dictionaries.

Return type:

list[dict]

scitex_dataset.general.openml.format_dataset(record)[source]

Format an OpenML dataset into a standardized dataset dictionary.

Parameters:: record (dict) – Raw OpenML dataset from list API.
Returns:: Standardized dataset dictionary.
Return type:: dict

HuggingFace Hub

HuggingFace Hub client for dataset and model downloads.

HuggingFace Hub (https://huggingface.co) hosts large language models, vision models, and datasets. This module provides utilities to fetch, search, and download datasets from HuggingFace with project-FS awareness (on Spartan, caches to /data/gpfs/ instead of home to avoid quota issues).

API Documentation: https://huggingface.co/docs/hub/

Local-state layout

Downloaded snapshots are regenerable cache data and live under the SciTeX local-state runtime directory:

<scope-root>/runtime/huggingface/<repo_id>/

where <scope-root> is project-scope (<project>/.scitex/dataset/) or user-scope ($SCITEX_DIR/dataset/, default ~/.scitex/dataset/). See general/01_ecosystem_06_local-state-directories for the canonical layout.

scitex_dataset.general.huggingface.fetch_dataset(repo_id, local_dir=None, repo_type='dataset', gated_token_var='HF_TOKEN_PATH', max_workers=4, hf_home_override=None)[source]

Fetch a complete HuggingFace dataset to disk.

On Spartan, if hf_home_override is provided, sets HF_HOME to that directory so the content-addressed cache doesn’t grow home quotas.

Parameters:

repo_id (str) – HuggingFace repository ID (e.g., “Anthropic/BioMysteryBench-full”).
local_dir (str, optional) – Local directory for dataset. If None, uses Spartan project FS if detected, else <scope-root>/runtime/huggingface/<repo_id>/ via the SciTeX local-state resolver (project scope wins over $SCITEX_DIR).
repo_type (str) – Repository type: “dataset” (default) or “model”.
gated_token_var (str) – Environment variable name for token file path (default: HF_TOKEN_PATH).
max_workers (int) – Parallel download workers (default: 4).
hf_home_override (str, optional) – Override HF_HOME cache directory (recommended on Spartan).

Returns:

Path to the downloaded dataset directory.

Return type:

Path

Raises:

Exception – If token resolution fails for gated repositories or network errors occur.

scitex_dataset.general.huggingface.search_datasets(query, limit=50)[source]

Search for datasets on HuggingFace.

Parameters:

query (str) – Search query string.
limit (int) – Maximum number of results (default: 50).

Returns:

List of search result dictionaries with fields: id, name, description, likes, downloads, private, gated, etc.

Return type:

list[dict]

scitex_dataset.general.huggingface.search_hub(query, limit=50)

Search for datasets on HuggingFace.

Parameters:

query (str) – Search query string.
limit (int) – Maximum number of results (default: 50).

Returns:

List of search result dictionaries with fields: id, name, description, likes, downloads, private, gated, etc.

Return type:

list[dict]

scitex_dataset.general.huggingface.fetch_all_datasets(query='', max_datasets=None, logger=None, **_unused)[source]

Catalog-style adapter so HuggingFace can plug into database.build.

Unlike OpenNeuro/DANDI/etc., HuggingFace has no bounded catalog — query is required for meaningful results. Without one this calls search_hub("") which lists by recency up to max_datasets.

Parameters:

query (str) – Search query. Empty string lists by recency (HF default).
max_datasets (int, optional) – Cap on results. Default 1000 to avoid runaway indexing.

Return type:

List[Dict]

scitex_dataset.general.huggingface.format_dataset(ds)[source]

Normalize an HF search-result dict to the common dataset schema.

HuggingFace records lack the n_subjects / modalities / tasks fields that BIDS/NWB sources expose, so those keys are emitted as None or empty lists. downloads and likes are preserved.

Return type:: Dict

scitex_dataset.general.huggingface.dataset_info(repo_id, repo_type='dataset')[source]

Get metadata about a HuggingFace dataset or model.

Parameters:

repo_id (str) – Repository ID (e.g., “username/dataset_name”).
repo_type (str) – Repository type: “dataset” (default) or “model”.

Returns:

Dataset metadata: id, name, description, downloads, likes, private, gated, size_gb, created_at, last_modified, etc.

Return type:

dict

scitex_dataset.general.huggingface.download_file(repo_id, filename, local_dir=None, repo_type='dataset')[source]

Download a single file from a HuggingFace repository.

Parameters:

repo_id (str) – Repository ID (e.g., “username/dataset_name”).
filename (str) – Path within the repository (e.g., “data/train.csv”).
local_dir (str, optional) – Local directory for download. If None, uses the SciTeX runtime directory via <scope-root>/runtime/huggingface/<repo_id>/.
repo_type (str) – Repository type: “dataset” (default) or “model”.

Returns:

Path to the downloaded file.

Return type:

Path

Raises:

Exception – If download fails.

Biology Sources

GEO

GEO (Gene Expression Omnibus) dataset fetcher.

GEO hosts gene expression and genomics datasets from NCBI.

API: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/

Example

>>> from scitex_dataset.biology import geo
>>> datasets = geo.fetch_all_datasets(max_datasets=10)
>>> formatted = [geo.format_dataset(ds) for ds in datasets]

scitex_dataset.biology.geo.fetch_datasets(retstart=0, retmax=100, term='gds[Entry Type]')[source]

Fetch a single page of datasets from GEO via NCBI E-utilities.

Return type:: dict

scitex_dataset.biology.geo.fetch_all_datasets(max_datasets=None, logger=None)[source]

Fetch all datasets from GEO with pagination.

Return type:: list[dict]

scitex_dataset.biology.geo.format_dataset(dataset)[source]

Extract and format GEO dataset information.

Return type:: dict

Pharmacology Sources

MoleculeNet

MoleculeNet dataset catalog for molecular machine learning.

MoleculeNet is a benchmark for molecular machine learning containing curated datasets spanning quantum mechanics, physical chemistry, biophysics, and physiology.

Reference: https://moleculenet.org/ Data source: DeepChem’s MoleculeNet catalog (static dataset list).

scitex_dataset.pharmacology.moleculenet.fetch_datasets()[source]

Fetch the MoleculeNet dataset catalog.

MoleculeNet is a static benchmark suite, so this returns the curated catalog rather than querying a live API.

Returns:: List of MoleculeNet dataset records.
Return type:: list[dict]

scitex_dataset.pharmacology.moleculenet.fetch_all_datasets(max_datasets=None, logger=None)[source]

Fetch all MoleculeNet datasets.

Parameters:

max_datasets (int, optional) – Maximum number of datasets to return.
logger (optional) – Logger for progress messages.

Returns:

List of MoleculeNet dataset records.

Return type:

list[dict]

scitex_dataset.pharmacology.moleculenet.format_dataset(record)[source]

Format a MoleculeNet dataset into a standardized dictionary.

Parameters:: record (dict) – Raw MoleculeNet dataset record.
Returns:: Standardized dataset dictionary.
Return type:: dict

ChEMBL

ChEMBL dataset fetcher.

ChEMBL hosts bioactivity data for drug-like molecules from EMBL-EBI.

API: https://www.ebi.ac.uk/chembl/api/data/

Example

>>> from scitex_dataset.pharmacology import chembl
>>> datasets = chembl.fetch_all_datasets(max_datasets=10)
>>> formatted = [chembl.format_dataset(ds) for ds in datasets]

scitex_dataset.pharmacology.chembl.fetch_datasets(limit=100, offset=0)[source]

Fetch a single page of assays from ChEMBL.

Return type:: dict

scitex_dataset.pharmacology.chembl.fetch_all_datasets(max_datasets=None, logger=None)[source]

Fetch all assays from ChEMBL with pagination.

Return type:: list[dict]

scitex_dataset.pharmacology.chembl.format_dataset(dataset)[source]

Extract and format ChEMBL assay information.

Return type:: dict

Medical Sources

ClinicalTrials.gov

ClinicalTrials.gov dataset fetcher.

ClinicalTrials.gov hosts clinical study records from the U.S. National Library of Medicine.

API: https://clinicaltrials.gov/api/v2/studies

Example

>>> from scitex_dataset.medical import clinicaltrials
>>> datasets = clinicaltrials.fetch_all_datasets(max_datasets=10)
>>> formatted = [clinicaltrials.format_dataset(ds) for ds in datasets]

scitex_dataset.medical.clinicaltrials.fetch_datasets(page_size=100, page_token=None)[source]

Fetch a single page of studies from ClinicalTrials.gov.

Return type:: dict

scitex_dataset.medical.clinicaltrials.fetch_all_datasets(max_datasets=None, logger=None)[source]

Fetch all studies from ClinicalTrials.gov with pagination.

Return type:: list[dict]

scitex_dataset.medical.clinicaltrials.format_dataset(dataset)[source]

Extract and format ClinicalTrials.gov study information.

Return type:: dict

CLI

Command-line interface for scitex-dataset.

The command grammar is:

scitex-dataset <domain> <dataset> <action> [OPTIONS]

For example:

scitex-dataset neuroscience openneuro fetch -n 50
scitex-dataset general huggingface fetch Anthropic/BioMysteryBench-full
scitex-dataset pharmacology chembl fetch
scitex-dataset db build

The flat fetch-<source> and hf <verb> shapes from earlier versions are kept as hidden deprecation aliases that print the new path and exit with status 2.

See general/03_interface_02_cli/02_subcommand-structure-noun-verb.md for the SciTeX CLI grammar.

scitex_dataset._cli._repositories_block()[source]

Render the per-domain bullet list shown in top-level --help.

Return type:: str