Metadata-Version: 2.4
Name: askfaro-progressive-context
Version: 0.5.0
Summary: Compile any content into a tiered, budget-aware, agent-navigable progressive-disclosure manifest, plus an on-demand expansion protocol — for small/on-device context windows
Project-URL: Homepage, https://github.com/poolside-ventures/askfaro-progressive-context
Project-URL: Repository, https://github.com/poolside-ventures/askfaro-progressive-context
Project-URL: Issues, https://github.com/poolside-ventures/askfaro-progressive-context/issues
Project-URL: Changelog, https://github.com/poolside-ventures/askfaro-progressive-context/blob/main/CHANGELOG.md
Author: Faro
License: MIT
License-File: LICENSE
Keywords: agent,context,context-window,llm,llms-txt,on-device,progressive-disclosure,tokens
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Provides-Extra: llm
Requires-Dist: httpx>=0.27; extra == 'llm'
Provides-Extra: schema
Requires-Dist: jsonschema>=4.21; extra == 'schema'
Provides-Extra: tokenize
Requires-Dist: tiktoken>=0.7; extra == 'tokenize'
Provides-Extra: yaml
Requires-Dist: pyyaml>=6; extra == 'yaml'
Description-Content-Type: text/markdown

# askfaro-progressive-context

**Compile any content into a tiered, budget-aware, agent-navigable progressive-disclosure manifest — for small / on-device context windows.**

On-device models have tiny context windows (~4k today, ~32k near-term). Stuffing everything in — or lossily compressing it — loses information. The alternative is **progressive disclosure**: give the model a compact, accurate *index* of what exists and *when each piece is relevant*, then let it fetch detail on demand within a hard token budget.

`askfaro-progressive-context` is the open-source compiler + format + runtime for that index. It is the **agent-navigated** half of Faro's context tooling (the model reads the index and decides what to expand); its sibling [`askfaro-embedded-search`](https://github.com/poolside-ventures/askfaro-embedded-search) is the retrieval-driven half. The two stay independent — this library has no hard dependency on it.

> Status: **pre-alpha, but published** — `pip install askfaro-progressive-context` (v0.4.0 on PyPI). Shipped and tested: the format, the compiler (`pcx build`) with adapters for `tools`/`docs`/`skills`/`memory`, the descriptor engine (bottom-up + contrastive + self-grade), incremental rebuilds (descriptor reuse keyed on `content_hash`), the expansion runtime, the `NavSession` agent-loop API (`index`/`look`/`open`/`close`, `local`/`remote` modes), identity-revalidated manifest caching (`ManifestLoader` / `AsyncManifestLoader`), and the eval harness. Still to come: per-budget frontier shaping, `website`/`file` adapters with clustering, and the hosted Faro registry.

## The idea in one screen

A **progressive-context manifest** (`pcx.json`) is a tree of nodes. Every node carries:

- a **descriptor** — `what` (one line: what this is) and `when` (one line: when it's relevant). These are the *navigation index*, and their quality is the whole game.
- **token costs** — so the runtime can plan expansion against a budget *without fetching anything*.
- either **children** (a branch) or a **payload pointer** (a leaf). Leaves are never inlined and always **verbatim** — no information is lost to a summary.
- optionally (**v0.2**) **`links`** — see-also cross-links `{to, why}` to related nodes in *other* branches, the lateral navigation a pure tree can't express — and **`facets`** — orthogonal tags (type, status, …) you filter on to narrow before reading descriptors.

Variants are **pre-generated per budget** (`pcx.4k.json`, `pcx.32k.json`, …); budgets are arbitrary integers, so a developer who needs headroom for their own content can build a `31k` variant — or reserve it at runtime.

## What's here (v0.4.0)

| Module | What it does |
|---|---|
| `schema/pcx-0.2.schema.json` | the format, as JSON Schema (v0.2: adds cross-links + facets) |
| `askfaro_progressive_context.types` | `Manifest` / `Node` / `Link` / `Payload` / `Variant` dataclasses |
| `askfaro_progressive_context.validate` | structural (zero-dep) + JSON Schema validation |
| `askfaro_progressive_context.build` (`pcx build`) | the compiler: adapters, descriptor engine (bottom-up + **contrastive convergence** + dual-channel self-grade + optional branch **synthesis**), **distinctiveness/fidelity metrics + CI gates**, cross-link inference, tree-shape lints, incremental rebuilds, per-budget emit + `llms.txt` |
| `askfaro_progressive_context.runtime` | the expansion protocol: `peek` / `expand` / `collapse` / `search` / `related` / `find_by_facets` / `reconcile`, with **hard budget enforcement** and a runtime `reserve` |
| `askfaro_progressive_context.session` | `NavSession` — the agent-loop API (`index`/`look`/`open`/`close`/`related`/`filter`) with `local`/`remote` modes + leaf context envelopes |
| `askfaro_progressive_context.navigator` | `KeywordNavigator` (deterministic baseline, no model) and `LLMNavigator` (bring your own `complete()`) |
| `askfaro_progressive_context.loader` | `ManifestLoader` / `AsyncManifestLoader` + `MemoryStore` / `FileStore` — transport-agnostic, identity-revalidated manifest caching |
| `askfaro_progressive_context.eval` | the **`navigation-success @ budget`** harness — the headline quality metric |

## Quick start

```bash
pip install askfaro-progressive-context          # runtime + CLI
pip install "askfaro-progressive-context[llm]"   # + the LLM descriptor engine for `pcx build`
```

```bash
# build manifest variants from a source tree (use --fake for an offline/CI build)
pcx build ./skills --kind skills --budgets 4k,32k --endpoint $ENDPOINT --model $MODEL

# validate a manifest
pcx validate examples/skills/manifest.pcx.4k.json --schema

# score navigation-success @ budget with the deterministic baseline navigator
pcx eval examples/skills/manifest.pcx.4k.json examples/skills/cases.json -v
```

> Working from a clone? Install editable with the dev + schema extras instead:
> `pip install -e ".[dev,schema]"`.

```python
from askfaro_progressive_context import Manifest, Runtime

m = Manifest.from_dict(json.load(open("examples/skills/manifest.pcx.4k.json")))
rt = Runtime(m, reserve=1024)          # leave 1k for your own content

rt.peek()                               # frontier: tier-1 descriptors + budget_remaining
rt.expand("recurring")                  # reveal a branch's children (charged against budget)
ref = rt.expand("recurring.create")     # splice a leaf's verbatim payload; raises if over budget
```

## The expansion protocol

The runtime — not the model — is the budget authority. `effective_budget = variant.budget − reserve`, and every `expand` is checked against it. When full it auto-collapses LRU leaves (opt-in) or refuses and tells the agent to choose. **The budget is never silently exceeded.**

## Driving it from an agent loop: `NavSession`

`Runtime` is the low-level budget authority; `NavSession` wraps it with the three verbs an agent loop actually drives, plus mode-aware defaults. The model's *choice of verb is the confidence signal* — there is no threshold to tune:

```python
from askfaro_progressive_context import Manifest, NavSession

s = NavSession(manifest, mode="local", reserve=1024)

s.index()                        # current frontier, shortest-useful view first
s.look(["recurring", "one_off"]) # escalate candidates to full descriptors WITHOUT opening them
s.open("recurring")              # branch -> drill into its children;
                                 # leaf   -> splice the verbatim content (budget-enforced)
s.close("recurring")             # collapse a node to reclaim budget

s.shown_tokens                   # everything the model has seen this session (the real "length")
s.budget_remaining
```

If the index is enough, the model calls `open`; if it can't decide, it calls `look` first. **Modes** encode the tokens-vs-round-trips tradeoff:

| mode | frontier view | small leaves | use when |
|---|---|---|---|
| `local` (default) | `brief` | resolved on demand (O(1) resident splice) | on-device / resident manifest — round-trips are ~free, so take many tiny steps |
| `remote` | `full` | inlined into `index()` (≤200 tokens) | network-backed — each hop costs latency, so disclose more per step to need fewer |

Pass `config=ModeConfig(...)` for a custom policy; an unknown `mode` raises with the valid options.

## Beyond the tree: cross-links, facets, freshness (v0.2)

A pure tree gives you *global* (where am I) and *local* (this neighborhood)
navigation, but not the *contextual* "this also relates to that over in another
branch." pcx v0.2 adds three things for that, all optional and additive:

```python
s = NavSession(manifest, mode="local")

s.related("posts.schedule")          # explore: see-also links (with why-phrases)
                                     # to related nodes in OTHER branches
s.filter(kind="template")            # facet-first precision: ids matching every
                                     # facet key=value, before you read descriptors
```

- **Cross-links** (`node.links`, `Runtime.related`) are the lateral edges. Build
  them with `pcx build --cross-links`, which infers see-also links from
  cross-branch descriptor similarity and stamps a why-phrase; `build.links.betweenness`
  then flags the bridge nodes those links create (the descriptors most worth
  getting right). By default similarity is lexical, but the contrastive pass
  drives sibling tokens apart — so on good descriptors lexical finds almost
  nothing. Pass `compile_source(cross_link_vectors={node_id: embedding})` (with a
  cosine-tuned `cross_link_min_sim`) to infer links from *semantic* similarity
  instead; that's what makes cross-links actually useful on a real corpus.
- **Facets** (`node.facets`, `Runtime.find_by_facets`) are orthogonal dimensions.
  Filtering on two independent 5-value facets cuts the space ~25× *before* any
  descriptor ranking — multiplicative precision for cheap.
- **Reconciliation** (`Runtime.reconcile`) treats a stale manifest as the hazard
  it is: an agent trusts a curated map completely, so a stale one misroutes
  silently. Pass the origin's current identity and/or the set of live ids and it
  returns staleness / dangling-route warnings instead of serving them blind.

```python
warnings = rt.reconcile(current_identity=origin_hash, live_ids=still_present)
if warnings:            # refetch before trusting the routes
    ...
```

## Caching the manifest: `ManifestLoader`

A manifest is small, identical for every reader, and changes only when its source content changes — an ideal cache target. But it is also a *routing index*, so a stale copy can point an agent at content that has moved or vanished. A plain time-to-live cache is therefore the wrong tool: the safe pattern is to cache on the manifest's **identity** and *revalidate* against it, never to expire blindly on a clock.

`ManifestLoader` gives every consumer that pattern by default, while staying **transport-agnostic** — it knows nothing about HTTP, files, S3, or ETags. You supply a `Fetcher` that reaches your origin; the loader owns the identity bookkeeping, the store, and the guarantee that a changed identity always wins.

```python
from askfaro_progressive_context import (
    FetchOutcome, FileStore, ManifestKey, ManifestLoader, identity_of,
)
import httpx

def fetch(key: ManifestKey, known_identity: str | None) -> FetchOutcome:
    # Your transport decides "changed?" however it likes. Over HTTP, let the
    # origin do it with a conditional request:
    headers = {"If-None-Match": known_identity} if known_identity else {}
    r = httpx.get(f"https://api.example.com/pcx/manifest?budget={key.budget}", headers=headers)
    if r.status_code == 304:
        return FetchOutcome.unchanged()                       # reuse the cache, no transfer
    body = r.json()
    return FetchOutcome.fresh(r.headers.get("ETag") or identity_of(body), body)

loader = ManifestLoader(fetch=fetch, store=FileStore("~/.cache/pcx"))
manifest = loader.load(ManifestKey(source_id="my-catalog", budget="4k"))   # -> Manifest
```

Your `fetch(key, known_identity)` is handed the identity already in cache (or `None`) and returns one of two outcomes:

- **`FetchOutcome.unchanged()`** — nothing changed since `known_identity` (e.g. a 304). The loader reuses the cached body: zero transfer, zero parsing.
- **`FetchOutcome.fresh(identity, body)`** — fresh content. The loader stores it under the key and returns it.

The safety property is **structural**: a `Fetcher` too dumb to do conditional requests and *always* returning full content still gets correct invalidation, because the store is identity-stamped and re-storing the same identity is a no-op. There is no TTL knob that can serve a stale routing index past an identity change.

| piece | role |
|---|---|
| `ManifestKey(source_id, budget)` | what you want — budget is part of the key (identity is shared across budgets) |
| `identity` (a `str`) | opaque content token: a content hash, an ETag, `"v{n}"` — compared for equality only, never parsed |
| `Fetcher` | **your** transport: `(key, known_identity) -> FetchOutcome`. The one bit you write |
| `MemoryStore` (default) | process-local; all a long-running process needs |
| `FileStore(dir)` | on-disk; earns its keep for short-lived processes (CLIs, serverless, edge cold starts) that would otherwise re-download every run |
| `identity_of(dict)` | derive an identity from a manifest body (prefers `source.content_hash`, else hashes the body) — for transports with no native validator |

Identity comes from the build: `emit` stamps each manifest with a bottom-up `source.content_hash`, surfaced as `manifest.identity`. `load()` means *"give me the current manifest, revalidating."* If you want to skip even the revalidation round-trip, don't call `load()` in a hot loop — hold the returned `Manifest` and reload on your own cadence.

For an async transport (e.g. an `httpx.AsyncClient`), use **`AsyncManifestLoader`** — same contract, but `fetch` is a coroutine and `load()` is awaited. The store stays synchronous (its get/put are fast local operations):

```python
from askfaro_progressive_context import AsyncManifestLoader, FetchOutcome, FileStore, ManifestKey

async def fetch(key, known_identity):
    r = await async_http.get(f"/pcx/manifest?budget={key.budget}",
                             headers={"If-None-Match": known_identity} if known_identity else {})
    if r.status_code == 304:
        return FetchOutcome.unchanged()
    return FetchOutcome.fresh(r.headers.get("ETag") or identity_of(r.json()), r.json())

loader = AsyncManifestLoader(fetch=fetch, store=FileStore("~/.cache/pcx"))
manifest = await loader.load(ManifestKey(source_id="my-catalog", budget="4k"))
```

> **For origins:** to let clients revalidate cheaply, serve the manifest with an `ETag` (the `source.content_hash` is a ready-made one) and honor `If-None-Match` with a `304`. Without that, clients still cache correctly via `identity_of`, they just re-transfer the body on each `load()`.

## Why a benchmark, not vibes

The moat is descriptor quality, so quality is measured, not asserted. The eval harness gives a navigator *only* the manifest and a budget plus `(query → correct leaf)` cases, and reports **navigation-success @ budget**, **first-hop precision**, and **average hops**. The deterministic `KeywordNavigator` establishes an offline floor; swap in an `LLMNavigator` to score a real model.

### Enforced at build time, not just measured after

The eval scores a finished manifest; the compiler also *gates* descriptor quality as it builds, so regressions fail fast:

- **Distinctiveness.** The contrastive pass is a convergence loop over a lexical similarity measure: it rewrites colliding siblings until the worst pair drops below `--collision-threshold`, and — when a level is wider than one contrast call — groups siblings *by similarity* so near-duplicates are actually compared. `--max-collision` fails the build if any siblings stay too alike.
- **Fidelity** (predict-then-verify). `--fidelity` scores whether each descriptor lets you predict its node's content (`lexical` offline, `llm` for real); `--min-fidelity` gates it. This catches descriptors that read well but don't anticipate the content.
- **Vacuity/retrieval channel.** A descriptor that just restates the title, or whose `when` is all connective filler, reads fine but is un-searchable — a deterministic check forces a repair, and the LLM grade scores navigation *and* retrieval channels.
- **Branch synthesis.** `--synthesis` writes branch descriptors from descendant *content* (what unifies the group, tensions, where to start) instead of assembling child descriptors.
- **Shape lints.** Over-wide levels, over-deep/small-corpus trees, and does-too-much branch `what` surface as warnings; `--flatten` collapses single-child hops; `--preset` fills a coherent config with a justification per setting.

## Length is the point

Accuracy without length misses why this exists. The eval reports
`first_view_tokens` and `tokens_to_answer` next to accuracy, and the runtime
renders the frontier at progressively shorter levels (`title` → `brief` →
`full`) so the agent opens with the **shortest** view and escalates only when
unsure. On a 24-tool catalog, a `title`-first view is ~14× smaller than the full
descriptor set, and the model reaches the right tool having seen ~6.9× less
context than loading every schema.

Progressive disclosure trades tokens for round-trips, so the whole artifact is
meant to stay **resident**: `Runtime(resolver=...)` resolves leaves from a local
store, making every `expand` an O(1) splice rather than a network fetch.

## Troubleshooting

Setup mistakes fail with actionable messages; see [`docs/troubleshooting.md`](docs/troubleshooting.md)
for the full table (missing extras, model config, architecture mismatches).

## License

MIT © Faro
