# hound-mcp v10.0.0 — Plan (DRAFT, pending Dondai approval)

Theme: **"The free web research tool that never gives up — and tells your
agent what to do next."** v10 makes hound the tool an LLM picks because every
response carries trust + freshness + the next step, and the fetch path stops
returning dead-ends. No new tools (still 6). 1 new feature + 1 major "make the
current one far better" + code-quality/speed hardening.

Pitch for the release post: *"Hound v10: when the live site hard-blocks your
agent, Hound silently pulls the page from the Internet Archive — dated and
honest. Every response now says whether a source is official, whether it's
stale, and exactly what to fetch next. Still $0, still keyless, still one
`pip install`."*

---

## Scope discipline — what is NOT in v10 (respects prior cuts)

Do NOT re-propose (deliberately cut, reasoning in CHANGELOG):
- Residential proxy rotation, LLM extraction via user endpoint, CSS-schema
  extraction, captcha-solving hook, diff/monitor tool, deep content-aware
  rerank (search mode=deep).
- **Research mode / multi-page content fusion** — shipped v5.0.0, CUT v7.1.0
  as bloat. Any "fetch N URLs, merge/rerank into one blob" feature = re-proposal
  of this. OFF THE TABLE. (Confirmed: focus already runs post-cache locally,
  so single-URL re-focus is already free — no work there either.)
- No 7th tool. 6 tools is the clean surface; v10 enriches existing ones.

---

## Pillar 1 — Archive resilience (NEW feature, the headline)

**Problem:** the #1 pain with free fetch tools is the hard dead-end — 403 bot
challenge, 451 legal block, 404, or all_tiers_failed (DataDome/Akamai). The
agent gets an error and stops. Archive.org usually has the page.

**Feature:** when smart_fetch's live tiers hard-fail, automatically try the
Wayback Machine's closest snapshot and return THAT, honestly marked.

- Trigger ONLY on genuine hard-fail: `all_tiers_failed`, status in {403, 451,
  503-with-bot-challenge, 404}, or `bot_challenge_detected`/`auth_wall` after
  escalation. Never on 200-with-content, never on soft errors (timeout that
  might resolve on retry), never when the live fetch already succeeded.
- Path: call `https://archive.org/wayback/available?url=<url>` (JSON). If
  `archived_snapshots.closest.available` is true, fetch the snapshot `url`
  through the EXISTING fetch path (HTTP tier is fine; archive serves static
  HTML). Retry the availability API up to 3x with 2-4s backoff (it throws
  transient errors ~1-in-10, confirmed 2026). Cap the whole fallback at ~10s.
- Honesty fields (additive to ResponseModel, all default-valued):
  - `source`: `"live"` | `"archive.org"` (default `"live"`). The agent always
    knows where the bytes came from.
  - `archived_at`: ISO date from the snapshot `timestamp` (YYYYMMDD... → ISO).
    Empty when `source=="live"`.
- `next_action` when archived: *"archive.org snapshot from <DATE>; the live
  site is blocking automated fetches — try again later or use a different
  source for current data."*
- Opt-out: `options.archive_fallback=false` and env `HOUND_ARCHIVE_FALLBACK=0`
  (default on). Opt-out is for agents that want the raw failure signal.
- `fetcher_used` becomes `"archive"` so the summary line shows it.
- Cache the archive result under a key that encodes `source=archive` so a
  repeat fetch of a hard-blocked URL is instant from cache (and the cache
  entry still expires by TTL, so a later retry can find the live page unblocked).

**Why it's god-tier, not bloat:** one fallback path, reuses existing fetch
infra, two additive fields, turns the most common dead-end into a result.
The only other tool doing Wayback fallback (TadMSTR/searxng-mcp) needs
Firecrawl-key + Crawl4AI + self-hosted SearXNG + Ollama. Hound = `pip install
hound-mcp`. That's the moat.

**Honest novelty claim for the post:** "the only free, keyless, zero-setup
fetch tool with automatic Internet Archive recovery." NOT "nobody does this."

---

## Pillar 2 — Intelligent response envelope (make the current one far better)

The envelope is already rich (summary, content_ok, next_action, metadata with
published_time/modified_time/author/canonical, links, quality_score). It is
dumb: next_action ignores freshness and page structure, and there is no
trust/authority signal. v10 makes it a brain. All additive fields, near-zero
cost (regex/heuristics on already-extracted HTML).

### 2a. `page_type` on every smart_fetch result
Classify: `article` | `docs` | `list` | `forum` | `qa` | `pdf` | `js_shell` |
`auth_wall` | `paywall` | `redirect` | `unknown`. Heuristics on the raw HTML
(count of `<a>` in main vs text length, `<article>` presence, forum markers
`class*=thread`/`class*=post`, QA markers stackoverflow-style, login form
markers, paywall phrases, `<meta http-equiv=refresh>`/JS redirect). PDF and
js_shell already detected — reuse. One field.

### 2b. Freshness as a first-class signal
- `content_date`: best-of {JSON-LD datePublished, meta article:published_time,
  modified_time, PDF creation_date} → ISO. (metadata.py already extracts the
  raw values; v10 picks the best and normalizes.)
- `content_age_days`: integer, computed from content_date vs fetched_at. -1
  when no date is recoverable.
- `is_stale`: bool, `content_age_days > 365` (tunable). Agents constantly need
  "is this current" — this answers it without parsing ISO strings.

### 2c. Authority / source_type
- `source_type`: `vendor-docs` | `official-docs` | `news` | `blog` | `forum` |
  `qa` | `gov` | `edu` | `github` | `docs-site` | `ecommerce` | `unknown`.
  Domain-suffix + path heuristics (docs.* / *.gov / *.edu / github.com /
  *stackoverflow.com / news TLDs etc.).
- `is_official`: bool — true when the domain is the canonical vendor/owner
  domain for the page's subject (best-effort: domain == og:site_name's
  implied owner, or domain is a known official-docs host). Conservative
  default false; only true on strong signal.

### 2d. `next_action` becomes genuinely useful (the payoff)
Reuses the existing field — zero new fields, just smarter branches that
consume 2a/2b/2c + existing links:
- `page_type=="list"` → *"this is a list of <N> links; the content you want is
  likely behind them. Top targets: <url1>, <url2>, <url3>. Or call smart_crawl
  on this URL."* (uses existing links.citations).
- `is_stale && page_type in (article,docs)` → *"published <DATE> (<age>d old);
  for current info, smart_search '<query> 2026'."*
- `page_type=="auth_wall"|"paywall"` → *"content behind login/paywall;
  archive_fallback may have a snapshot, or switch sources."*
- `page_type=="redirect"` → *"page redirected; the real URL is <canonical or
  final url>."*
- Existing branches (paginate, robots, js_shell, bot_challenge, PDF errors,
  all_tiers_failed) stay; all_tiers_failed now also notes archive fallback
  was tried (or will be).

Net: the agent gets trust + freshness + next-step in every response without
asking. No competitor does page-class + freshness-aware next_action. Unique.

---

## Pillar 3 — Code-quality + speed hardening ("professional, not a side project")

### 3a. Kill the ResponseModel vanish-on-truncation bug class
`_apply_chunking` builds ResponseModel twice (dev notes flag this as the
contextvar-vanish class). Refactor to ONE factory
`_truncate_response(result, content, max_chars, offset)` that derives the
truncated response from the original via `model_copy(update={...})` so EVERY
field is preserved by construction. Regression test: truncate a response
with all envelope fields set, assert none vanish.

### 3b. Kill the "dev venv ships a built wheel, pytest runs stale copy" trap
Add `pythonpath = ["src"]` to `[tool.pytest.ini_options]` (pytest 7+). pytest
then imports from src/ regardless of what's installed in .venv. This trap has
bitten the project 4+ times (pdfplumber, onnxruntime, etc.). One line, permanent.
Pin it with a test asserting `master_fetch.__file__` is under `src/`.

### 3c. Type-annotate the public surface
Full annotations + tight docstrings on the 6 tool methods and the key helpers
(`_agent_hints`, `_apply_chunking`/`_truncate_response`, `_translate_response`,
the new envelope/archive functions). Static-analysis friendly, senior-review
clean. No behavior change.

### 3d. Profile + cut hot-path waste
Add `scripts/profile_hotpath.py`: cProfile a cold fetch, a cache hit, a
search, and (new) an archive fallback. Goal: confirm cache-hit is
zero-overhead (no browser touch, no extraction, only focus BM25 on cached
content — should be <50ms), and cut whatever the profile flags. Deliverable:
a CHANGELOG entry listing concrete cuts (e.g., redundant regex compiles,
repeated url parsing, double metadata passes). No speculative optimization —
only what the profile proves.

### 3e. Dead-code / dup pass
rg for unused imports, unreachable branches, duplicated logic across
server.py/crawl.py/search.py. The 9.1.1 cleanup did some; finish it. Remove
the `keyword` remnants if any survive.

---

## Testing plan (paranoid, per AGENTS testing principle)

- `tests/test_v10_archive.py`: fallback triggers on hard-fail (403/503/404/
  all_tiers_failed), does NOT trigger on 200/soft-timeout/200-with-content,
  opt-out works, transient availability-API error retries then succeeds,
  archived_at parsed correctly, result cached under archive key, cache hit
  returns instantly. Mock both the availability API and the snapshot fetch.
- `tests/test_v10_envelope.py`: page_type for each class (article/docs/list/
  forum/qa/pdf/js_shell/auth_wall/paywall/redirect), content_age_days +
  is_stale math (incl. no-date → -1), source_type classification, is_official
  conservative default, and every new next_action branch (list→links,
  stale→search, auth→archive, redirect→canonical). Adversarial: a page that
  is BOTH stale AND a list → which next_action wins (defined precedence).
- `tests/test_v10_construction.py`: truncate a fully-populated response,
  assert ALL envelope fields survive (the regression for 3a).
- `tests/test_v10_devworkflow.py`: assert `master_fetch.__file__` under src/
  (pins 3b).
- Live (not pytest): `scripts/live_archive_check.py` — fetch a known
  hard-blocked URL for real, assert archive fallback returns content with
  source="archive.org" and a real archived_at.
- Full suite must stay green: `pytest -m 'not e2e'` (635 → ~660+ tests).

---

## Versioning + compatibility

- **v10.0.0** (major). Justified: default-on archive fallback is a behavior
  change on the failure path (was: error-only; now: archive content). Additive
  envelope fields are non-breaking (all default-valued).
- 6 tools, their schemas, and all existing response fields unchanged. New
  fields are additive. Opt-out exists for the one behavior change.
- Bump pyproject `version`, `__init__.version`, CHANGELOG, README, STATUS in
  lockstep (AGENTS: never cross-publish out of sync).

---

## Risks + mitigations

- **Archive fallback returns stale/wrong content silently** → `source` +
  `archived_at` always set, next_action says "snapshot from DATE," opt-out
  exists. Honesty is the differentiator.
- **Archive.org down / rate-limits** → 3x backoff retry, 10s cap, and on
  archive failure fall through to the original error (never worse than today).
- **page_type misclassification** → conservative heuristics, `unknown`
  default, next_action only adds hints (never blocks). Tests cover each class.
- **Envelope cost on every fetch** → regex on already-extracted HTML, ~µs.
  Profile (3d) confirms <1ms overhead.
- **Construction refactor (3a) introduces a regression** → the
  truncate-preserves-all-fields test is the guard; run the full suite.
- **pytest pythonpath (3b) masks a real install bug** → it only affects the
  test run path; `pip install` / wheel build / twine check still validate the
  real package. The test pinning `__file__` under src/ makes the intent
  explicit.

---

## Execution order (after approval)

1. 3b (pytest pythonpath) + 3a (construction factory) first — they de-risk
   the rest (tests run live src; truncation is safe).
2. Pillar 2 envelope (2a page_type → 2b freshness → 2c authority → 2d
   next_action) — additive, testable in isolation.
3. Pillar 1 archive fallback — depends on nothing, but reuses the envelope's
   `source`/next_action, so do it after 2.
4. 3c type annotations + 3e dead-code pass — mechanical, last.
5. 3d profile pass — run once everything's in, cut what it flags.
6. Bump version, full suite, live checks, CHANGELOG/README/STATUS, then
   publish per `skills/python-publish/SKILL.md` (read first, never skip).

---

## Open decisions for Dondai (need a call before I code)

1. **Archive fallback default-on vs opt-in?** I recommend default-on (the
   wow + agents get results not dead-ends). Opt-out via env/option either way.
2. **`is_stale` threshold** — 365d? 180d? Make it domain-aware (news stale at
   30d, docs stale at 730d)? I recommend a flat 365d default + a tunable, keep
   v10 simple; domain-awareness can be a v10.1.
3. **Scope confirmation** — is "1 new feature (archive) + envelope
   intelligence + hardening" the right v10, or do you want a second new
   feature? Candidates I held back as possible v10.1: (a) second-extractor
   fallback (readability-style when trafilatura returns thin), (b) crawl
   query-focused harvest (DROPPED — re-proposes cut research mode), (c)
   extraction_confidence field. My read: the 3 pillars are a complete major
   release; adding (a) risks bloat. Your call.
