scitex_dataset API Reference
SciTeX Dataset - Unified interface for scientific dataset discovery.
Domains: - neuroscience: OpenNeuro, DANDI, PhysioNet - general: Scientific Data, Zenodo, Figshare, OpenML, HuggingFace - biology: GEO (Gene Expression Omnibus) - pharmacology: ChEMBL, MoleculeNet - medical: ClinicalTrials.gov - ai-for-science: CORE-Bench, BixBench, BioMysteryBench (agentic
reproducibility / bioinformatics / biology cohorts —
download | prepare | maskverb tree)
- Usage:
>>> from scitex_dataset import neuroscience >>> datasets = neuroscience.fetch_all_datasets(max_datasets=10)
>>> # Or direct import for convenience >>> from scitex_dataset import fetch_all_datasets, search_datasets
>>> # Local database for fast searching >>> from scitex_dataset import database as db >>> db.build() # Fetch all sources and index >>> results = db.search("alzheimer EEG", min_subjects=20)
>>> # Prepare an agentic benchmark (standardize only — safe, fast) >>> from scitex_dataset import ai_for_science >>> paths = ai_for_science.resolve_paths("corebench") >>> ai_for_science.corebench.standardize( ... raw_dir=paths.raw_dir, ... for_solver_dir=paths.for_solver_dir, ... eval_dir=paths.eval_dir, ... )
- scitex_dataset.db_build(sources=None, store=None, logger=None)
Build the index from all sources.
- Parameters:
- Returns:
Count of datasets indexed per source.
- Return type:
- scitex_dataset.db_search(query=None, source=None, modality=None, min_subjects=None, max_subjects=None, min_downloads=None, has_readme=False, limit=50, offset=0, order_by='downloads', store=None)
Search the index.
- Parameters:
Full-text search over id, name, readme and tasks. Bare words are ANDed,
"quoted phrases"are phrases,ordisjoins and a leading-negates. Malformed input matches nothing rather than raising, because the text came from a person.WORDS ARE STEMMED and English stopwords are dropped, which the previous tokeniser did neither of. So
studiesnow finds a dataset that saysstudy— a superset, and the reason to prefer it — but a query consisting only of stopwords (the,of) matches nothing rather than everything. Passtext_config="simple"on the schema if a package ever needs the literal behaviour back.source (
Optional[str]) – Filter by source: “openneuro”, “dandi”, “physionet”.modality (
Optional[str]) – Filter by modality (e.g., “mri”, “eeg”).has_readme (
bool) – Only include datasets with a non-empty readme.limit (
int) – Maximum results (default: 50).offset (
int) – Skip first N results (for pagination).order_by (
str) – Order by: downloads, views, n_subjects, size_gb, name, created.store (
Store|None) – The store to read from. Default: this host’s.
- Returns:
Matching datasets, in the shape the fetchers produced.
- Return type:
- scitex_dataset.db_show_stats(store=None)
Index statistics.
- Return type:
- Returns:
dict – Counts per source, the total, where the store is, and when the index was last written.
existsreports whether anything is INDEXED, not whether a file is ondisk. The store always exists; an empty one is the state that used to be
”no database yet”, and it is the one a caller has to act on.
- scitex_dataset.filter_results(datasets, **kwargs)[source]
Filter and rank dataset dicts — matches
dataset_filter_resultsMCP tool.
- scitex_dataset.list_sources()[source]
Return the 11-source registry — matches
dataset_list_sourcesMCP tool.- Return type:
- scitex_dataset.openneuro_fetch(batch_size=100, max_datasets=None, logger=None)
Fetch every dataset record from OpenNeuro by paginating GraphQL.
Walks the public
crn/graphqlendpoint with cursor-based pagination until exhausted (ormax_datasetsis reached). Useformat_datasetto project each raw record into the package’s common dataset schema.- Parameters:
batch_size (
int) – Records per HTTP request. The OpenNeuro server caps this; the function does not validate the upper bound.max_datasets (
Optional[int]) – Stop after this many records.None(default) fetches the entire catalog.logger (logging.Logger, optional) – If provided, HTTP and GraphQL errors are logged. Errors are otherwise silent (the function returns whatever it has so far).
- Returns:
Raw GraphQL
nodedicts, in catalog order. Pass each throughformat_datasetfor the normalized schema.- Return type:
Examples
>>> records = fetch_all_datasets(max_datasets=10) >>> len(records) <= 10 True
- scitex_dataset.dandi_fetch(max_datasets=None, page_size=100, logger=None)
Fetch all dandisets from DANDI Archive with pagination.
- scitex_dataset.physionet_fetch(max_datasets=None, logger=None)
Fetch all databases from PhysioNet with pagination.
- scitex_dataset.gin_fetch(max_datasets=None, batch_size=50, logger=None)
Walk GIN’s repo catalog by paginating
/repos/search.Mirrors the OpenNeuro / DANDI conventions so the unified search and database modules can index GIN uniformly.
- scitex_dataset.gin_search(query='', limit=50, page=1, logger=None)[source]
Search GIN repos via
/api/v1/repos/search.- Parameters:
query (
str) – Free-text query. Empty string returns the global feed.limit (
int) – Server-side page size.page (
int) – Page number, 1-indexed.logger (logging.Logger, optional) – Errors are logged here; otherwise silent (returns empty list).
- Returns:
Raw repo records as returned by GIN. Run each through
gin_format()for the package’s normalized schema.- Return type:
- scitex_dataset.gin_info(repo_id, logger=None)[source]
Fetch metadata for one GIN repo via
/api/v1/repos/{owner}/{repo}.Returns the empty dict on HTTP error (logged if
loggergiven).- Return type:
- scitex_dataset.gin_download(repo_id, local_dir=None, files=None, branch='master', prefer='auto', metadata_only=False, first_only=False, logger=None)[source]
Download a GIN repo (metadata + annexed content).
The default flow:
git clone https://gin.g-node.org/<owner>/<repo>.git— pulls metadata and annex pointer text files (~MB; no auth).For each file matching
files(default: every file under the repo whose worktree content is an annex pointer), stream the bytes from/raw/<branch>/<path>and replace the pointer atomically.
- Parameters:
repo_id (
str) –owner/repoGIN identifier (e.g."USZ_NCH/Human_MTL_units_scalp_EEG_and_iEEG_verbal_WM").local_dir (
Union[str,Path,None]) – Local destination. Falls back to the Spartan project FS or<scope-root>/runtime/gin/<owner>__<repo>/per_resolve_local_dir().files (
Optional[Iterable[str]]) – Repo-relative paths to fetch.None(default) fetches every annex pointer encountered.branch (
str) – Source branch for the/raw/URL.prefer (
str) – Backend selector."datalad"requires the optionaldataladextra;"auto"usesdataladif importable, else"https". No silent fallback: an explicit"datalad"request raises if the optional dep is missing.metadata_only (
bool) – Skip annex content; just clone the repo.first_only (
bool) – Fetch only the FIRST pointer encountered (smoke-test path).logger (logging.Logger, optional) – Per-step progress is logged here if given.
- Returns:
- ``{“repo_dir”: Path, “files”: [{“path”: Path, “bytes”: int}, …],
”total_bytes”: int, “backend”: str}``.
- Return type:
- Raises:
ImportError – If
prefer="datalad"is explicit and the optional dep is missing; the install hint can be read viascitex_dev.last_install_hint().
- scitex_dataset.zenodo_fetch(query='', max_datasets=None, page_size=25, type_filter='dataset', logger=None)
Fetch all datasets from Zenodo with pagination.
- Parameters:
- Returns:
List of raw record dictionaries.
- Return type:
Fetch all datasets from Figshare with pagination.
- scitex_dataset.openml_fetch(max_datasets=None, page_size=100, logger=None)
Fetch all datasets from OpenML with pagination.
- scitex_dataset.moleculenet_fetch(max_datasets=None, logger=None)
Fetch all MoleculeNet datasets.
- scitex_dataset.geo_fetch(max_datasets=None, logger=None)
Fetch all datasets from GEO with pagination.
- scitex_dataset.chembl_fetch(max_datasets=None, logger=None)
Fetch all assays from ChEMBL with pagination.
- scitex_dataset.clinicaltrials_fetch(max_datasets=None, logger=None)
Fetch all studies from ClinicalTrials.gov with pagination.
- scitex_dataset.huggingface_fetch(query='', max_datasets=None, logger=None, **_unused)
Catalog-style adapter so HuggingFace can plug into
database.build.Unlike OpenNeuro/DANDI/etc., HuggingFace has no bounded catalog —
queryis required for meaningful results. Without one this callssearch_hub("")which lists by recency up tomax_datasets.
- scitex_dataset.huggingface_search(query, limit=50)
Search for datasets on HuggingFace.
- scitex_dataset.huggingface_info(repo_id, repo_type='dataset')
Get metadata about a HuggingFace dataset or model.
- scitex_dataset.huggingface_download_file(repo_id, filename, local_dir=None, repo_type='dataset')
Download a single file from a HuggingFace repository.
- Parameters:
repo_id (
str) – Repository ID (e.g., “username/dataset_name”).filename (
str) – Path within the repository (e.g., “data/train.csv”).local_dir (
Optional[str]) – Local directory for download. If None, uses the SciTeX runtime directory via<scope-root>/runtime/huggingface/<repo_id>/.repo_type (
str) – Repository type: “dataset” (default) or “model”.
- Returns:
Path to the downloaded file.
- Return type:
- Raises:
Exception – If download fails.
- scitex_dataset.download_dataset(source, id, dest=None, **kwargs)[source]
Unified
download_dataset(source, id, dest, **opts)dispatcher.- Parameters:
source (
str) – Source id, case-insensitive. Matched againstscitex_dataset._sources.ALL_SOURCES.id (
str) – Source-native identifier — HForg/name, GINowner/repo, etc.dest (str | Path, optional) – Local destination. Per-source default applies if
None.**kwargs – Forwarded to the underlying
<source>_downloadfunction.
- Returns:
Whatever the underlying downloader returns (path or manifest dict).
- Return type:
Any
- Raises:
ValueError – If
sourceis not in the registry.NotImplementedError – If the matched source has no download adapter wired up yet.
- scitex_dataset.corebench_standardize(*, raw_dir, for_solver_dir, eval_dir, only=None, force=False, **_)
Split the oracle JSONs into the for_solver + eval views.
raw_dirmust already contain the upstream-pristinedataset/core_train.jsonandcore_test.json— either from a priordownload(...)or hand-staged by the operator. The two record lists are concatenated (train first, then test); eachresultsentry becomes one task (leak-safe) + one answer.for_solveris written in the PER-CAPSULE shape: one self-containedcapsule-NNN/dir per native capsule (friendly id), each holding the EXTRACTED capsule archive ininput/, atask.jsonlof only that capsule’s rows, the uniform submission schema/example, and a README — plus a rootindex.jsonlMAPPER (friendly_id ↔ native_id). An agent binds exactly onecapsule-NNN/dir.only(a friendlycapsule-NNNid OR a native capsule id, e.g.capsule-0201225) materializes just that one capsule’s dir; the mapper is always written in full.forcere-extracts capsules that are already present (default skips them).- Return type:
- scitex_dataset.bixbench_standardize(*, raw_dir, for_solver_dir, eval_dir, only=None, force=False, **_)
Read the oracle manifest, build the for_solver + eval views.
Each upstream record becomes one leak-safe task (
{task_id, benchmark, prompt, data}) + one answer ({answer, ideal}). Thetask_idis keyed on the record’s UNIQUEquestion_id(bixbench/<short_id>-qN), NOT the capsule-scopedshort_id— a single capsule holds multiple questions that share oneshort_id, so keying onshort_idwould collapse distinct questions. Each task’sdatapoints at itsdata_folderarchive (e.g.CapsuleFolder-<uuid>.zip); a capsule’s multiple questions still group into ONEcapsule-NNN/dir viadata_folder.for_solveris written in the PER-CAPSULE shape: one self-containedcapsule-NNN/dir per native capsule (friendly id), each holding the EXTRACTED capsule archive ininput/, atask.jsonlof only that capsule’s rows, the uniform submission schema/example, and a README — plus a rootindex.jsonlMAPPER (friendly_id ↔ native_id). An agent binds exactly onecapsule-NNN/dir. Records with nodata_folderhavedata: nulland materialize no capsule.only(a friendlycapsule-NNNid OR a native capsule id) materializes just that one capsule’s dir; the mapper is always written in full.forcere-extracts capsules already present.- Return type:
- scitex_dataset.biomysterybench_standardize(*, raw_dir, for_solver_dir, eval_dir, only=None, force=False, **_)
Read
raw_dir/problems.csv, build the for_solver + eval views.Each CSV row becomes one leak-safe task (
{task_id, benchmark, prompt, data}) + one answer ({rubric}). A task’sdatapoints atdata/<id>.zipwhen that environment exists underraw/data, elsenull. The rubric stays in operator-onlyeval/answers.jsonl.for_solveris written in the PER-CAPSULE shape: one self-containedcapsule-NNN/dir per problem that ships adata/<id>.zipenvironment (friendly id), each holding the EXTRACTED archive ininput/, atask.jsonlof only that problem’s row, the uniform submission schema/example, and a README — plus a rootindex.jsonlMAPPER (friendly_id ↔ native_id). An agent binds exactly onecapsule-NNN/dir. A problem withdata: null(no environment) materializes no capsule.only(a friendlycapsule-NNNid OR a native capsule id) materializes just that one capsule’s dir; the mapper is always written in full.forcere-extracts capsules already present.- Return type:
Search Module
Unified search interface for neuroscience datasets.
Currently supports: - OpenNeuro (BIDS neuroimaging)
Future sources: - DANDI (NWB neurophysiology) - PhysioNet (EEG/ECG/physiology) - Zenodo (general scientific)
- scitex_dataset.search.search_datasets(datasets, modality=None, min_subjects=None, max_subjects=None, task_contains=None, text_query=None, min_downloads=None, has_readme=False)[source]
Filter datasets by various criteria.
- Parameters:
- Return type:
- Returns:
Filtered list of datasets
Example
>>> from scitex_dataset import fetch_all_datasets, format_dataset >>> from scitex_dataset.search import search_datasets >>> raw = fetch_all_datasets(max_datasets=100) >>> datasets = [format_dataset(d) for d in raw] >>> eeg_data = search_datasets(datasets, modality="eeg", min_subjects=20)
Database Module
The dataset index, held in the fleet’s shared store.
- Usage:
>>> from scitex_dataset import database as db >>> db.build() # Fetch all sources and index them >>> results = db.search("alzheimer EEG", min_subjects=20)
WHAT CHANGED, AND WHY IT IS NOT A PORT
This module used to keep its own private database file with its own full-text index, its own schema and its own triggers. That was a second storage engine living inside a leaf package, which is the shape the fleet ruled out: state has exactly one home, and a private file is not it. A file also has no notion of WHO — anyone who can open it holds every permission — so an index built that way can be handed over but never collaborated on.
The index now lives in scitex_dev.store, used DIRECTLY. There is no
scitex_dataset database layer between this module and the primitive and
there is not meant to be one: a wrapper is how two packages end up with two
answers to the same question. What stays here is the DOMAIN — the shape
lives in _index_schema, the verbs live below.
THE ONE THING THE PRIMITIVE DID NOT HAVE was a way to read by criteria
rather than by key, which is a large part of why a private index looked
reasonable. Rather than rebuild one here, that surface was added to the
store (Store.search / count / tally, Query), where every
package gets it. If something here still feels awkward, that is a gap in
the primitive and belongs there too.
NOTHING IS DELETED ANY MORE
The store has no delete verb; clear() hides rows instead. A hidden
row leaves the default view — so a search after clear finds nothing,
exactly as before — but the record and its history stay readable, and a
rebuild revives it rather than starting it over.
- scitex_dataset.database.build(sources=None, store=None, logger=None)[source]
Build the index from all sources.
- Parameters:
- Returns:
Count of datasets indexed per source.
- Return type:
- scitex_dataset.database.clear(store=None)[source]
Retire every indexed dataset.
- Return type:
- Returns:
bool – True if anything was retired, False if the index was already empty.
NOT A DELETE, and the difference is the point. The store has no delete
verb (the failure it exists to make impossible is a row disappearing)
because some code decided it should. So this HIDES. The datasets leave
the default view, so
search()andget_stats()answer exactlyas they did when the old file had been removed, while the records and
their history stay readable and a later
build()brings them back.
- scitex_dataset.database.get_stats(store=None)[source]
Index statistics.
- Return type:
- Returns:
dict – Counts per source, the total, where the store is, and when the index was last written.
existsreports whether anything is INDEXED, not whether a file is ondisk. The store always exists; an empty one is the state that used to be
”no database yet”, and it is the one a caller has to act on.
- scitex_dataset.database.get_store(store=None)[source]
The index store — this host’s, unless one is handed in.
storeis how a test points at a throwaway schema without patching anything: pass aStoreand it is used verbatim. Every public function here takes the same argument for the same reason.host_storeis the single resolver for WHERE that is; nothing in this package reads a connection string. The result is cached because opening a store creates its tables under an advisory lock, and a CLI doing that per subcommand would pay for it every time.- Return type:
Store
- scitex_dataset.database.search(query=None, source=None, modality=None, min_subjects=None, max_subjects=None, min_downloads=None, has_readme=False, limit=50, offset=0, order_by='downloads', store=None)[source]
Search the index.
- Parameters:
Full-text search over id, name, readme and tasks. Bare words are ANDed,
"quoted phrases"are phrases,ordisjoins and a leading-negates. Malformed input matches nothing rather than raising, because the text came from a person.WORDS ARE STEMMED and English stopwords are dropped, which the previous tokeniser did neither of. So
studiesnow finds a dataset that saysstudy— a superset, and the reason to prefer it — but a query consisting only of stopwords (the,of) matches nothing rather than everything. Passtext_config="simple"on the schema if a package ever needs the literal behaviour back.source (
Optional[str]) – Filter by source: “openneuro”, “dandi”, “physionet”.modality (
Optional[str]) – Filter by modality (e.g., “mri”, “eeg”).has_readme (
bool) – Only include datasets with a non-empty readme.limit (
int) – Maximum results (default: 50).offset (
int) – Skip first N results (for pagination).order_by (
str) – Order by: downloads, views, n_subjects, size_gb, name, created.store (
Store|None) – The store to read from. Default: this host’s.
- Returns:
Matching datasets, in the shape the fetchers produced.
- Return type:
- scitex_dataset.database.store_description(store=None)[source]
A credential-free one-line description of where the index lives.
Replaces the old path accessor. There is no file any more, and returning a path would be a lie a caller could act on — the CLI printed it beside a file size, and both facts stopped existing at once.
RESOLVES WITHOUT CONNECTING when no store is open yet. Asking where the index WOULD be must not create it:
db build --dry-runprints this line, and going throughget_store()would open a connection and run the schema DDL — a dry run whose whole promise is that it writes nothing.- Return type:
Neuroscience Sources
OpenNeuro
OpenNeuro dataset fetcher using GraphQL API.
Example
>>> from scitex_dataset import fetch_all_datasets, format_dataset
>>> datasets = fetch_all_datasets(max_datasets=10)
>>> formatted = [format_dataset(ds) for ds in datasets]
- scitex_dataset.neuroscience.openneuro.fetch_datasets(first=10, after=None)[source]
Fetch a single page of datasets from OpenNeuro.
- Return type:
- scitex_dataset.neuroscience.openneuro.fetch_all_datasets(batch_size=100, max_datasets=None, logger=None)[source]
Fetch every dataset record from OpenNeuro by paginating GraphQL.
Walks the public
crn/graphqlendpoint with cursor-based pagination until exhausted (ormax_datasetsis reached). Useformat_datasetto project each raw record into the package’s common dataset schema.- Parameters:
batch_size (
int) – Records per HTTP request. The OpenNeuro server caps this; the function does not validate the upper bound.max_datasets (
Optional[int]) – Stop after this many records.None(default) fetches the entire catalog.logger (logging.Logger, optional) – If provided, HTTP and GraphQL errors are logged. Errors are otherwise silent (the function returns whatever it has so far).
- Returns:
Raw GraphQL
nodedicts, in catalog order. Pass each throughformat_datasetfor the normalized schema.- Return type:
Examples
>>> records = fetch_all_datasets(max_datasets=10) >>> len(records) <= 10 True
- scitex_dataset.neuroscience.openneuro.format_dataset(node)[source]
Project a raw OpenNeuro GraphQL node into the common dataset schema.
Every catalog source exposes
format_datasetreturning the same shape so they can plug intodatabase.buildandsearch.search_datasetsuniformly.- Parameters:
node (
dict) – A singleedges[].nodeelement from the OpenNeuro GraphQL response (thedraft/analyticskeys are read; missing fields fall back toNone/ 0).- Returns:
Normalized record with keys:
id, name, n_subjects, modalities, tasks, size_gb, downloads, views, readme, license, doi, url, source.- Return type:
DANDI
DANDI Archive dataset fetcher.
DANDI (Distributed Archives for Neurophysiology Data Integration) hosts neurophysiology data in NWB (Neurodata Without Borders) format.
API: https://api.dandiarchive.org/api
Example
>>> from scitex_dataset.neuroscience import dandi
>>> datasets = dandi.fetch_all_datasets(max_datasets=10)
>>> formatted = [dandi.format_dataset(ds) for ds in datasets]
- scitex_dataset.neuroscience.dandi.fetch_datasets(page=1, page_size=100, ordering='-modified')[source]
Fetch a single page of dandisets from DANDI Archive.
- Return type:
PhysioNet
PhysioNet dataset fetcher.
PhysioNet hosts physiological signal databases including EEG, ECG, EMG, and other biomedical signals.
API: https://physionet.org/api/v1/
Example
>>> from scitex_dataset.neuroscience import physionet
>>> datasets = physionet.fetch_all_datasets(max_datasets=10)
>>> formatted = [physionet.format_dataset(ds) for ds in datasets]
- scitex_dataset.neuroscience.physionet.fetch_datasets(page=1)[source]
Fetch a single page of databases from PhysioNet.
- Return type:
General Sources
Zenodo
Zenodo API client for scientific dataset discovery.
Zenodo is a general-purpose open repository developed under the European OpenAIRE program and operated by CERN. It allows researchers to deposit research papers, data sets, research software, reports, and any other research related digital artifacts.
API Documentation: https://developers.zenodo.org/
- scitex_dataset.general.zenodo.fetch_datasets(query='', page=1, size=25, sort='mostrecent', type_filter='dataset')[source]
Fetch datasets from Zenodo.
- Parameters:
query (
str) – Search query string (Elasticsearch query syntax).page (
int) – Page number (1-indexed).size (
int) – Number of results per page (max 10000).sort (
str) – Sort order: ‘bestmatch’, ‘mostrecent’, ‘-mostrecent’.type_filter (
str) – Resource type filter: ‘dataset’, ‘software’, ‘publication’, etc.
- Returns:
API response with ‘hits’ containing records.
- Return type:
OpenML
OpenML API client for machine learning dataset discovery.
OpenML is an open platform for sharing machine learning datasets, tasks, and experiments. It hosts thousands of curated ML datasets.
API Documentation: https://www.openml.org/apis
- scitex_dataset.general.openml.fetch_datasets(offset=0, limit=100, status='active')[source]
Fetch datasets from OpenML.
HuggingFace Hub
HuggingFace Hub client for dataset and model downloads.
HuggingFace Hub (https://huggingface.co) hosts large language models, vision models, and datasets. This module provides utilities to fetch, search, and download datasets from HuggingFace with project-FS awareness (on Spartan, caches to /data/gpfs/ instead of home to avoid quota issues).
API Documentation: https://huggingface.co/docs/hub/
Local-state layout
Downloaded snapshots are regenerable cache data and live under the SciTeX local-state runtime directory:
<scope-root>/runtime/huggingface/<repo_id>/
where <scope-root> is project-scope (<project>/.scitex/dataset/) or
user-scope ($SCITEX_DIR/dataset/, default ~/.scitex/dataset/). See
general/01_ecosystem_06_local-state-directories for the canonical layout.
- scitex_dataset.general.huggingface.fetch_dataset(repo_id, local_dir=None, repo_type='dataset', gated_token_var='HF_TOKEN_PATH', max_workers=4, hf_home_override=None)[source]
Fetch a complete HuggingFace dataset to disk.
On Spartan, if hf_home_override is provided, sets HF_HOME to that directory so the content-addressed cache doesn’t grow home quotas.
- Parameters:
repo_id (
str) – HuggingFace repository ID (e.g., “Anthropic/BioMysteryBench-full”).local_dir (
Optional[str]) – Local directory for dataset. If None, uses Spartan project FS if detected, else<scope-root>/runtime/huggingface/<repo_id>/via the SciTeX local-state resolver (project scope wins over$SCITEX_DIR).repo_type (
str) – Repository type: “dataset” (default) or “model”.gated_token_var (
str) – Environment variable name for token file path (default: HF_TOKEN_PATH).max_workers (
int) – Parallel download workers (default: 4).hf_home_override (
Optional[str]) – Override HF_HOME cache directory (recommended on Spartan).
- Returns:
Path to the downloaded dataset directory.
- Return type:
- Raises:
Exception – If token resolution fails for gated repositories or network errors occur.
- scitex_dataset.general.huggingface.search_datasets(query, limit=50)[source]
Search for datasets on HuggingFace.
- scitex_dataset.general.huggingface.search_hub(query, limit=50)
Search for datasets on HuggingFace.
- scitex_dataset.general.huggingface.fetch_all_datasets(query='', max_datasets=None, logger=None, **_unused)[source]
Catalog-style adapter so HuggingFace can plug into
database.build.Unlike OpenNeuro/DANDI/etc., HuggingFace has no bounded catalog —
queryis required for meaningful results. Without one this callssearch_hub("")which lists by recency up tomax_datasets.
- scitex_dataset.general.huggingface.format_dataset(ds)[source]
Normalize an HF search-result dict to the common dataset schema.
HuggingFace records lack the n_subjects / modalities / tasks fields that BIDS/NWB sources expose, so those keys are emitted as
Noneor empty lists.downloadsandlikesare preserved.- Return type:
- scitex_dataset.general.huggingface.dataset_info(repo_id, repo_type='dataset')[source]
Get metadata about a HuggingFace dataset or model.
- scitex_dataset.general.huggingface.download_file(repo_id, filename, local_dir=None, repo_type='dataset')[source]
Download a single file from a HuggingFace repository.
- Parameters:
repo_id (
str) – Repository ID (e.g., “username/dataset_name”).filename (
str) – Path within the repository (e.g., “data/train.csv”).local_dir (
Optional[str]) – Local directory for download. If None, uses the SciTeX runtime directory via<scope-root>/runtime/huggingface/<repo_id>/.repo_type (
str) – Repository type: “dataset” (default) or “model”.
- Returns:
Path to the downloaded file.
- Return type:
- Raises:
Exception – If download fails.
Biology Sources
GEO
GEO (Gene Expression Omnibus) dataset fetcher.
GEO hosts gene expression and genomics datasets from NCBI.
API: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/
Example
>>> from scitex_dataset.biology import geo
>>> datasets = geo.fetch_all_datasets(max_datasets=10)
>>> formatted = [geo.format_dataset(ds) for ds in datasets]
- scitex_dataset.biology.geo.fetch_datasets(retstart=0, retmax=100, term='gds[Entry Type]')[source]
Fetch a single page of datasets from GEO via NCBI E-utilities.
- Return type:
Pharmacology Sources
MoleculeNet
MoleculeNet dataset catalog for molecular machine learning.
MoleculeNet is a benchmark for molecular machine learning containing curated datasets spanning quantum mechanics, physical chemistry, biophysics, and physiology.
Reference: https://moleculenet.org/ Data source: DeepChem’s MoleculeNet catalog (static dataset list).
- scitex_dataset.pharmacology.moleculenet.fetch_datasets()[source]
Fetch the MoleculeNet dataset catalog.
MoleculeNet is a static benchmark suite, so this returns the curated catalog rather than querying a live API.
ChEMBL
ChEMBL dataset fetcher.
ChEMBL hosts bioactivity data for drug-like molecules from EMBL-EBI.
API: https://www.ebi.ac.uk/chembl/api/data/
Example
>>> from scitex_dataset.pharmacology import chembl
>>> datasets = chembl.fetch_all_datasets(max_datasets=10)
>>> formatted = [chembl.format_dataset(ds) for ds in datasets]
- scitex_dataset.pharmacology.chembl.fetch_datasets(limit=100, offset=0)[source]
Fetch a single page of assays from ChEMBL.
- Return type:
Medical Sources
ClinicalTrials.gov
ClinicalTrials.gov dataset fetcher.
ClinicalTrials.gov hosts clinical study records from the U.S. National Library of Medicine.
API: https://clinicaltrials.gov/api/v2/studies
Example
>>> from scitex_dataset.medical import clinicaltrials
>>> datasets = clinicaltrials.fetch_all_datasets(max_datasets=10)
>>> formatted = [clinicaltrials.format_dataset(ds) for ds in datasets]
- scitex_dataset.medical.clinicaltrials.fetch_datasets(page_size=100, page_token=None)[source]
Fetch a single page of studies from ClinicalTrials.gov.
- Return type:
CLI
Command-line interface for scitex-dataset.
The command grammar is:
scitex-dataset <domain> <dataset> <action> [OPTIONS]
For example:
scitex-dataset neuroscience openneuro fetch -n 50
scitex-dataset general huggingface fetch Anthropic/BioMysteryBench-full
scitex-dataset pharmacology chembl fetch
scitex-dataset db build
The flat fetch-<source> and hf <verb> shapes from earlier
versions are kept as hidden deprecation aliases that print the new path
and exit with status 2.
See general/03_interface_02_cli/02_subcommand-structure-noun-verb.md
for the SciTeX CLI grammar.