Qualifier Vocabulary Categorization

§1 — Why this plan exists

The catalog SPA's Grammar composer (site/src/views/Grammar.jsx, backed by site/src/lib/grammar-compose.js — a JS mirror of imas_standard_names/grammar/render.py) lets a user toggle grammar segments and pick tokens from each segment's closed vocabulary. Four qualifier sub-kinds — aggregation, orbit, population, subject — were promoted out of the qualifier group into their own segments because they are orthogonal, closed, single-occurrence dimensions. Everything else is the open, recursive, multi-occurrence qualifier class (<qualifier>_<base>, e.g. major in major_area, external/heating in total_external_heating_power).

Building the composer surfaced two problems in how the qualifier vocabulary is published and consumed. The parser accepts 348 qualifier tokens (Vocabularies.qualifiers). Of those, 71 are the named sub-kinds and 277 are "other". But that 277 is not a clean set:

136 of the 277 are cross-listed with other segments — locus entities (bolometer, antenna_strap, beam_tracing_beam, coil_conductor) and operators (gradient, derivative, flux_surface_averaged, cumulative). The parser accepts them as qualifiers for matching, but they are not qualifiers and must not be offered as such.
The 141 genuine generic qualifiers (absorbed, alfven, anomalous, bootstrap, collisional, diamagnetic, doppler, e_cross_b, …) are not a jumble — they already fall into ~30 coherent families that exist as section comments in qualifiers.yml (transport regime, source/mechanism, geometry/position, diagnostic, wave polarization, sign, temporal, normalized, …). But that structure is comments only: the tokens are a flat list, and the IR's Qualifier.category field (imas_standard_names/grammar/ir.py) is a forward-compat slot that is never populated.

Partition of the parser's qualifier acceptance set. The work below removes the red group from the qualifier role and gives the green group machine-readable categories.

Interim mitigation already shipped (site): the Grammar composer's generic "qualifier" picker computes the genuine set by subtracting the named sub-kinds and the locus/operator/component/base vocabularies in the SPA, so it offers ~141 tokens, not 277. That is a client-side workaround; this plan makes the partition authoritative in ISN and adds categories the composer can sub-group by.

§2 — Scope of ISN changes ✓ landed 2026-06-25

Landed (commit on main + RC). Added qualifier_categories.yml mapping all 161 qualifiers.yml tokens to 12 normalized categories; the parser populates IR Qualifier.category from it; dataset.py now emits GRAMMAR_VOCAB.qualifiers as the genuine modifier set (159 = qualifiers.yml − subjects), each with its category — the 136 cross-listed operator/locus/subject tokens no longer leak into the qualifier role (decision keep-flag: they stay in the parser's acceptance union for stripping, excluded only from the published role). The SPA qualifier picker now sub-groups by the emitted category, dropping the subtraction workaround. Full round-trip stays 1364/1364; full Python suite + 154 SPA tests green. Detail: §2 landed. Decisions: category-taxonomy=normalized-set, category-storage=sidecar-map.

§2a — Audit & de-pollute the qualifier acceptance set

Determine, for each of the 136 cross-listed tokens, whether it is genuinely needed in Vocabularies.qualifiers for the parser to match real names, or whether it is an artifact that should be removed from the qualifier role and resolved only via its true segment (locus / operator / component / base). Constraint: the parser strips qualifiers recursively before matching the base, so removing a token the parser depends on can break parse() round-trip. Every removal MUST be verified against full-catalog round-trip (build_site_dataset over the ISNC catalog: 1364/1364 today). Produce a keep/remove/reclassify verdict per token, not a blanket delete.

§2b — Categorize the genuine generic qualifiers

Promote the ~30 comment-families in qualifiers.yml to machine-readable per-token categories, and populate the IR's existing Qualifier.category field at parse time so the category travels with each parsed qualifier. Decide the category taxonomy (reuse the existing comment-family labels vs. a smaller normalized set) under §3.

§2c — Emit qualifier categories in the dataset

Extend imas_standard_names/catalog/dataset.py::_build_grammar_vocab so the SPA's GRAMMAR_VOCAB.qualifiers entries carry their category (and the cross-listed tokens are excluded from the qualifier role authoritatively, not by SPA subtraction). This is the single contract point the site consumes.

§2d — Site Grammar composer consumption (links back to the design)

Once categories are emitted, replace the SPA's subtraction-based "genuine qualifier" computation (otherQualifiers in site/src/views/Grammar.jsx) with the authoritative emitted set, and sub-group the generic qualifier picker by category — mirroring the named sub-kind grouping. No change to the round-trip/compose contract (grammar-compose.js); this is presentation + data-source only.

§3 — Open decisions

See the decisions panel below. These gate §2a and §2b.

§ Decisions

How to handle the 136 tokens cross-listed between the qualifier role and locus/operator/ component/base?

The 136 cross-listed tokens are NOT in qualifiers.yml — they enter Vocabularies.qualifiers only because the parser unions unary_prefix operators + locus/object tokens into its acceptance set so they can be stripped during parsing. Publishing qualifiers.yml's genuine modifier set (minus subjects) as the qualifier role excludes them with zero parser/round-trip risk. No change to the acceptance union.

What category taxonomy for the genuine qualifiers?

qualifiers.yml's 39 comment-section families are redundant/inconsistent (e.g. 'Source/mechanism' vs 'Source / drive mechanism'; three near-duplicate 'Diagnostic / measurement' families). Collapse them into ~12 normalized categories for a usable grouped picker; the 39 families bootstrap the mapping.

Where do per-token categories live?

Add qualifier_categories.yml (category -> tokens) in the vocabularies dir as the authoritative category source, bootstrapped from the comment families. qualifiers.yml stays the flat token source (no churn, no parser risk); a validation test enforces every qualifier token has exactly one category. Populate the IR Qualifier.category from it at parse time.

§ Followups

§2a — Audit the 136 cross-listed qualifier tokens

First shippable step: produce the per-token keep/remove/reclassify verdict for the qualifier tokens that are also locus/operator/component/base, gated on full-catalog round-trip. This unblocks the de-pollution (§2a) and informs the category work (§2b).

Project: imas-standard-names
Plan:    qualifier-vocabulary-categorization (http://localhost:8765/imas-standard-names/qualifier-vocabulary-categorization.html)
Section: §2a
Tier:    opus

Context
  The parser accepts 348 qualifier tokens; 136 are cross-listed with locus/
  operator/component/base and are not genuinely qualifiers. Honour the Open
  decisions shown live above this brief (esp. cross-listed-tokens) before
  acting.

State to read  (code/files, not the plan)
  imas_standard_names/grammar/vocabularies/qualifiers.yml
  imas_standard_names/grammar/parser.py  (load_default_vocabularies, the
    recursive qualifier-stripping in _match_base_with_qualifiers)
  imas_standard_names/grammar/vocab_loaders.py
  imas_standard_names/catalog/dataset.py::_build_grammar_vocab
  site/src/views/Grammar.jsx  (otherQualifiers — the current SPA workaround)

Scope locks / constraints  (non-decision)
  - Do NOT regress parse()/compose round-trip: build_site_dataset over the
    ISNC catalog must stay 1364/1364 (8 binary-operator names use the verbatim
    fallback). Verify with the round-trip harness before/after any removal.
  - No change to grammar-compose.js compose/seed contract.

Done-when
  1. A per-token verdict table (keep / remove / reclassify-to-segment) for the
     136 cross-listed tokens, committed under the plan or docs/.
  2. Round-trip verified green at each removal.
  3. Followup written for §2b (categorize the genuine set) + this one resolved.

§2 landed (all sub-sections). Finding: the 136 cross-listed tokens are NOT in qualifiers.yml — they enter the parser's acceptance union via operator/locus stripping; fixed by publishing qualifiers.yml (genuine set) at the emitter, parser untouched, round-trip 1364/1364. Added qualifier_categories.yml (12 cats), IR Qualifier.category populated, GRAMMAR_VOCAB emits {token,category}, SPA picker grouped by category.

Optional: domain-expert review of the 12-category bucketing + deeper acceptance-union audit

The 12-category mapping was derived family-by-family from qualifiers.yml's comment sections (deterministic, not per-token tuned). A domain-expert pass could re-bucket edge cases (e.g. ferritic: material vs source; normalized currently lands under state→normalized via its comment family). Separately, a deeper audit could decide whether any genuinely mis-filed locus/operator tokens should be removed from the parser's qualifier acceptance union (not just excluded from the published role). Neither blocks the shipped picker.

Project: imas-standard-names
Plan:    qualifier-vocabulary-categorization (http://localhost:8765/imas-standard-names/qualifier-vocabulary-categorization.html)
Section: refinement (optional)
Tier:    sonnet

Context
  §2 shipped: qualifiers categorized into 12 buckets (qualifier_categories.yml),
  emitted + grouped in the SPA picker. This is an OPTIONAL polish pass; the
  picker already works. Honour the locked decisions above.

State to read (code/files)
  imas_standard_names/grammar/vocabularies/qualifier_categories.yml
  imas_standard_names/grammar/vocabularies/qualifiers.yml  (comment families)
  tests/grammar/test_qualifier_categories.py  (coverage invariant)

Scope locks / constraints
  - Keep the coverage invariant green: every qualifiers.yml token categorized
    exactly once; categories stay within the 12-member normalized set unless
    the test's EXPECTED_CATEGORIES is updated in the same change.
  - Do NOT regress parse()/compose round-trip (1364/1364) or the full suite.

Done-when
  1. Re-bucketed tokens (if any) committed with rationale; tests green.
  2. Verdict recorded on the acceptance-union audit (remove vs keep-excluded).
  3. This followup resolved (or re-scoped) in the plan.