# Search Mode & Hybrid Alpha: Proper Noun / SDK Name Improvement Research

**Date:** 2026-03-10
**Scope:** `src/mcp_vector_search/core/`, `src/mcp_vector_search/mcp/`, `src/mcp_vector_search/cli/commands/search.py`
**Trigger:** Search failure for "getstream.io" — semantic embedding (all-MiniLM-L6-v2) treats it as a generic term

---

## 1. Summary of Findings

The search pipeline is well-architected for this improvement. `search_mode` and `hybrid_alpha` are already fully wired end-to-end (CLI, MCP, engine). The default `hybrid_alpha=0.7` strongly favors vector search, which is exactly the wrong setting for proper nouns and SDK names. There is currently zero query pre-processing that looks at the structure of the query (dots, camelCase, known package patterns) to auto-adjust routing. All five improvement points identified below map to specific, small code changes.

---

## 2. Current Defaults and Where They Live

### 2.1 `search_mode`

| Location | Default | Type |
|---|---|---|
| `core/search.py:136` — `SemanticSearchEngine.search()` signature | `SearchMode.HYBRID` | `SearchMode` enum |
| `core/search.py:222` — `_search_internal()` signature | `SearchMode.HYBRID` | `SearchMode` enum |
| `cli/commands/search.py:230` — CLI `--search-mode` option | `"hybrid"` | string |
| `cli/commands/search.py:521` — `run_search()` function | `"hybrid"` | string |
| `mcp/search_handlers.py:121` — `handle_search_code()` | `"hybrid"` (via `args.get("search_mode", "hybrid")`) | string |
| `mcp/tool_schemas.py:126` — MCP JSON schema default | `"hybrid"` | string |

The `SearchMode` enum is defined at `core/search.py:35-40`:
```python
class SearchMode(StrEnum):
    VECTOR = "vector"
    BM25 = "bm25"
    HYBRID = "hybrid"
```

### 2.2 `hybrid_alpha`

| Location | Default | Meaning |
|---|---|---|
| `core/search.py:137` — `SemanticSearchEngine.search()` | `0.7` | 70% vector, 30% BM25 |
| `core/search.py:223` — `_search_internal()` | `0.7` | same |
| `cli/commands/search.py:237` — CLI `--hybrid-alpha` | `0.7` | same |
| `cli/commands/search.py:522` — `run_search()` | `0.7` | same |
| `mcp/search_handlers.py:122` — `handle_search_code()` | `0.7` (via `args.get("hybrid_alpha", 0.7)`) | same |
| `mcp/tool_schemas.py:131` — MCP JSON schema default | `0.7` | same |

### 2.3 How `hybrid_alpha` is used in RRF fusion (`core/search.py:1659-1668`)

```python
vector_score = (
    hybrid_alpha / (RRF_K + vector_rank)      # alpha = 0.7 → heavy vector weight
    if vector_rank != float("inf") else 0.0
)
bm25_score = (
    (1.0 - hybrid_alpha) / (RRF_K + bm25_rank)  # 1-alpha = 0.3 → light BM25 weight
    if bm25_rank != float("inf") else 0.0
)
rrf_scores[chunk_id] = vector_score + bm25_score
```

`RRF_K = 60` is a module-level constant at `core/search.py:32`. The formula is standard weighted RRF — lowering `hybrid_alpha` directly reduces vector influence and increases BM25 influence.

### 2.4 `similarity_threshold`

- Project config default: `0.3` (in `config/settings.py:26-27` as a `Field` default)
- `SemanticSearchEngine.__init__` accepts it as `similarity_threshold: float = 0.3` (`core/search.py:58`)
- MCP schema default: `0.3` (`mcp/tool_schemas.py:64`)

---

## 3. What Is Already Configurable vs. Hard-Coded

### Already fully configurable (end-to-end wired)

- `search_mode`: CLI `--search-mode`, MCP `search_code` parameter, engine `search()` parameter
- `hybrid_alpha`: CLI `--hybrid-alpha`, MCP `search_code` parameter, engine `search()` parameter
- `similarity_threshold`: CLI `--threshold`, MCP parameter, engine parameter
- `expand` (query expansion): CLI `--expand/--no-expand`, MCP parameter
- `use_rerank` / `rerank_top_n`: CLI and MCP
- `use_mmr` / `diversity`: CLI and MCP

### Hard-coded / not user-facing

- **BM25 tokenizer** (`core/bm25_backend.py:301-327`): Uses `re.findall(r"\w+", text.lower())` — splits on non-word characters. This means `getstream.io` is tokenized into `["getstream", "io"]`, losing the dot-joined form. There is no subword handling.
- **BM25 corpus construction** (`core/bm25_backend.py:86-118`): Function names are repeated twice as a weight boost. No special handling for dotted names or camelCase splitting.
- **`RRF_K = 60`** (`core/search.py:32`): Module-level constant, not configurable via API.
- **Query pre-processing** (`core/query_processor.py`): Applies only simple abbreviation expansion. No structural analysis of the query (no dot/camelCase detection, no proper-noun detection).
- **`search_similar` and `search_context`**: Neither the MCP schema nor the engine methods accept `search_mode` or `hybrid_alpha`. Both are hard-wired to call `self.search()` with default parameters (pure vector for `search_similar`, hybrid default for `search_by_context`).

---

## 4. What `search_similar` and `search_context` Expose

### `search_similar` (MCP tool)

`mcp/tool_schemas.py` schema only exposes: `file_path`, `function_name`, `limit`, `similarity_threshold`.

In `mcp/search_handlers.py:192-196`, the handler calls:
```python
results = await self.search_engine.search_similar(
    file_path=file_path_obj,
    function_name=function_name,
    limit=limit,
    similarity_threshold=similarity_threshold,
)
```

In `core/search.py:444-450`, `search_similar` calls `self.search()` with no `search_mode` or `hybrid_alpha` — uses all defaults.

### `search_context` (MCP tool)

`mcp/tool_schemas.py` schema exposes: `description`, `focus_areas`, `limit`. No `search_mode` or `hybrid_alpha`.

In `core/search.py:480-484`, `search_by_context` calls `self.search()` with no `search_mode` or `hybrid_alpha` — uses all defaults.

### CLI `search --context` and `search --similar`

The `search_main` CLI handler routes to `run_similar_search` or `run_context_search` when `--similar` or `--context` flags are passed (`cli/commands/search.py:321-352`). Neither of those runner functions accept `search_mode` or `hybrid_alpha` as parameters — they are silently dropped even though the top-level CLI option parses them.

---

## 5. Query Pre-Processing Pipeline (No Proper-Noun Detection Today)

The query goes through two stages before reaching the search backends:

**Stage 1 — `QueryProcessor.preprocess_query()` (`core/query_processor.py:54-86`)**
- Collapses whitespace
- Applies a static abbreviation expansion table (e.g., `"auth"` → `"authentication authorize login"`)
- The expansion table has 30 entries, all generic programming terms
- No handling for dots, camelCase, package names, version strings

**Stage 2 — `QueryExpander.expand()` (`core/query_expander.py`, called from `core/search.py:271`)**
- Generates multiple query variants for synonym expansion
- Controlled by `expand` flag

The `QueryAnalyzer` class (`core/query_analyzer.py`) has some structural detection:
- Detects CamelCase at line 82 (`re.search(r"\b[A-Z][a-zA-Z]*\b", query)`)
- Detects dot notation at line 86 (`re.search(r"\b\w+\.\w+", query)`)
- But these only generate *display suggestions* — they do not affect `search_mode` or `hybrid_alpha`

There is no code path that auto-adjusts routing parameters based on query structure.

---

## 6. The "Getstream.io" Failure — Root Cause Analysis

Query: `"getstream.io"` (or similar SDK names)

1. `QueryProcessor.preprocess_query()` collapses whitespace only; no expansion for `"getstream.io"` (not in table). The query arrives at the vector model unchanged.
2. `all-MiniLM-L6-v2` encodes `"getstream.io"` as a near-random point in semantic space — it was trained on natural language, not package registry tokens. It sees no meaningful signal in `"getstream.io"`.
3. In hybrid mode with `hybrid_alpha=0.7`, vector search contributes 70% of the RRF score. Even if BM25 ranks a matching chunk at position 1, the final RRF score is heavily diluted by the poor vector score.
4. The BM25 tokenizer splits `"getstream.io"` into `["getstream", "io"]`. If the codebase contains `import getstream` or `from getstream import ...`, BM25 will find it. But at `hybrid_alpha=0.7`, BM25's RRF contribution is small.
5. `"io"` is a very common BM25 token (appears in imports, file paths, etc.) which adds noise to the BM25 score.

With `hybrid_alpha=0.2`, BM25 contributes 80% — exact token matches dominate and the SDK name is found. With `search_mode=bm25`, the vector model is bypassed entirely.

---

## 7. Specific Code Locations for Each Improvement

### Improvement 1: Lower `hybrid_alpha` default (or add per-call auto-detection)

**Current default location:** `core/search.py:137`
```python
hybrid_alpha: float = 0.7,
```
**Recommended value for SDK/package queries:** `0.2` (BM25-heavy)

No other defaults need changing since the CLI and MCP both read the engine default when the user does not specify.

### Improvement 2: Auto-detect "identifier-style" queries in `_search_internal`

**Target file:** `core/search.py`, inside `_search_internal()` between lines 250-267 (after threshold calculation, before query preprocessing):

```python
# Detect identifier-style queries and override hybrid_alpha
if _is_identifier_query(query):
    if hybrid_alpha == 0.7:          # only override when using the default
        hybrid_alpha = 0.2           # lean heavily on BM25 for exact token matching
        logger.debug(
            f"Identifier-style query detected ('{query}'), "
            f"lowering hybrid_alpha to 0.2 for better keyword matching"
        )
```

The detection function would live in `core/query_processor.py` (alongside existing query analysis):

```python
import re

# Patterns that indicate identifier/package/SDK-style queries
_IDENTIFIER_PATTERNS = [
    re.compile(r"\b[\w-]+\.[\w-]+"),          # dotted names: getstream.io, react-query.dev
    re.compile(r"\b[a-z][\w]*[A-Z][\w]*\b"),  # camelCase: StreamApp, getStream
    re.compile(r"\b[A-Z][A-Z0-9_]{2,}\b"),    # SCREAMING_SNAKE or acronyms: SDK, API_KEY
    re.compile(r"\b@[\w/-]+\b"),               # npm scoped: @tanstack/query
    re.compile(r"\b[\w-]+-[\w-]+\b"),          # hyphenated packages: react-activity-feed
    re.compile(r"\b\w+\.\w+\.\w+\b"),          # deep dotted: io.getstream.chat
]
# Words that look like package identifiers even without these patterns
_PACKAGE_KEYWORDS = frozenset(["sdk", "npm", "pip", "pypi", "crate", "gem", "maven"])

def is_identifier_query(query: str) -> bool:
    """Return True if query looks like an identifier, package name, or SDK name."""
    for pattern in _IDENTIFIER_PATTERNS:
        if pattern.search(query):
            return True
    words = query.lower().split()
    if any(w in _PACKAGE_KEYWORDS for w in words):
        return True
    return False
```

### Improvement 3: Fix BM25 tokenizer to preserve dotted names

**Target file:** `core/bm25_backend.py`, `_tokenize()` method at line 301:

Current:
```python
tokens = re.findall(r"\w+", text)
```

Recommended — two-pass tokenization:
```python
# First extract whole dotted/hyphenated identifiers, then fallback to word tokens
# This preserves "getstream.io", "react-activity-feed", "io.getstream.chat"
compound_tokens = re.findall(r"[\w][\w.\-/]*[\w]", text)   # compound identifiers
word_tokens = re.findall(r"\w+", text)                      # individual words
# Combine: compound forms first (higher BM25 term frequency weight), then components
tokens = compound_tokens + word_tokens
tokens = [t for t in tokens if t]
```

This ensures `"getstream.io"` is indexed as both `"getstream.io"` AND `"getstream"` AND `"io"`, so a query of `"getstream.io"` matches the compound token exactly.

**Warning:** Changing the tokenizer invalidates existing BM25 indexes. A version bump in the index metadata and a re-index prompt would be needed on startup, similar to the embedding model version check already present.

### Improvement 4: Expose `search_mode` and `hybrid_alpha` on `search_context` and `search_similar`

**`search_context` — MCP schema:** `mcp/tool_schemas.py`, `_get_search_context_schema()` (currently around line 177)
Add to `inputSchema.properties`:
```python
"search_mode": {
    "type": "string",
    "enum": ["vector", "bm25", "hybrid"],
    "default": "hybrid",
    "description": "Search mode: 'vector', 'bm25', or 'hybrid'"
},
"hybrid_alpha": {
    "type": "number",
    "default": 0.7,
    "minimum": 0.0,
    "maximum": 1.0,
    "description": "BM25/vector balance in hybrid mode (0.0=pure BM25, 1.0=pure vector)"
},
```

**`search_context` — handler:** `mcp/search_handlers.py`, `handle_search_context()` at line 222:
```python
search_mode_str = args.get("search_mode", "hybrid")
hybrid_alpha = args.get("hybrid_alpha", 0.7)
search_mode = SearchMode(search_mode_str.lower())

results = await self.search_engine.search_by_context(
    context_description=description,
    focus_areas=focus_areas,
    limit=limit,
    search_mode=search_mode,       # add param
    hybrid_alpha=hybrid_alpha,     # add param
)
```

**`search_by_context` engine method:** `core/search.py:456-484` — add `search_mode` and `hybrid_alpha` to signature and pass through to `self.search()`.

**`search_similar` MCP + CLI:** Same pattern — expose in schema, pass through handler to `search_similar()` engine method, forward to `self.search()` call at line 444.

**CLI `run_similar_search` and `run_context_search`:** `cli/commands/search.py` at lines 851 and 965 — neither function accepts `search_mode` or `hybrid_alpha`. The top-level `search_main` parses them but does not pass them to these runners. The fix is to add those parameters to both runner signatures and pass them in.

### Improvement 5: Add identifier-style query tip to the "no results" flow

**Target file:** `cli/commands/search.py`, line 763 (the `else:` branch after empty results):

```python
# Existing suggestions ...
# Add:
if any(c in query for c in [".", "-", "@", "/"]) or re.search(r"[a-z][A-Z]", query):
    print_info(
        "  - For SDK/package names, try: "
        "[cyan]--search-mode bm25[/cyan] or [cyan]--hybrid-alpha 0.2[/cyan]"
    )
    print_info(
        "  - Or rephrase as code identifiers: "
        "[cyan]\"import getstream StreamApp\"[/cyan]"
    )
```

---

## 8. Impact Summary

| Improvement | Effort | Impact | Risk |
|---|---|---|---|
| Lower `hybrid_alpha` default to 0.5 (balanced) | 1 line | Medium — helps all queries a little | Low — slight regression on pure semantic queries |
| Auto-detect identifier queries, override alpha to 0.2 | ~30 lines in `query_processor.py` + 10 in `_search_internal` | High — directly fixes the failure case | Low — only activates on identifier patterns |
| Fix BM25 tokenizer to preserve dotted names | 3 lines in `_tokenize()` + index migration | High — BM25 now matches `"getstream.io"` exactly | Medium — requires re-index |
| Expose `search_mode`/`hybrid_alpha` on `search_context` and `search_similar` | ~20 lines per tool | Low — users can override manually | Low |
| Add SDK search tip to "no results" feedback | 5 lines in CLI | Low — UX only | None |

---

## 9. Recommended Query Reformulation (No Code Change)

Until auto-detection is implemented, users can get good results today with:

```
# Instead of:
mcp-vector-search search "getstream.io"

# Use BM25 mode:
mcp-vector-search search "getstream.io" --search-mode bm25

# Or lower alpha:
mcp-vector-search search "getstream.io" --hybrid-alpha 0.2

# Or use code identifier syntax (vector model handles these better):
mcp-vector-search search "getstream StreamApp react-activity-feed"
```

For MCP callers (Claude agents):
```json
{
  "query": "getstream.io StreamApp",
  "search_mode": "bm25",
  "hybrid_alpha": 0.2
}
```

---

## 10. Files Requiring Changes

| File | Change | Lines |
|---|---|---|
| `core/search.py` | Auto-detect identifier queries, override `hybrid_alpha` | ~287 (before preprocessing loop) |
| `core/query_processor.py` | Add `is_identifier_query()` function and regex patterns | After line 159 |
| `core/bm25_backend.py` | Update `_tokenize()` to preserve compound identifiers | Lines 314-325 |
| `mcp/tool_schemas.py` | Add `search_mode`/`hybrid_alpha` to `search_context` and `search_similar` schemas | After line 174 (context), after line 163 (similar) |
| `mcp/search_handlers.py` | Read new params in `handle_search_context()` and `handle_search_similar()` | Lines 222, 155 |
| `core/search.py` | Add `search_mode`/`hybrid_alpha` to `search_by_context()` and `search_similar()` signatures | Lines 411, 456 |
| `cli/commands/search.py` | Pass `search_mode`/`hybrid_alpha` to `run_similar_search()` and `run_context_search()` | Lines 328, 344 |
| `cli/commands/search.py` | Add SDK name tip to "no results" feedback | Line 763 |
