Metadata-Version: 2.4
Name: wiki-forest
Version: 0.1.0
Summary: Forest.Wikipedia.EN: a populated English Wikipedia terrain with sealed semantic retrieval.
Author: Forest contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/schmerbert/Forest.Wikipedia.EN
Project-URL: Repository, https://github.com/schmerbert/Forest.Wikipedia.EN
Project-URL: Issues, https://github.com/schmerbert/Forest.Wikipedia.EN/issues
Project-URL: Populated release, https://huggingface.co/datasets/schmerbertt/forest-wikipedia-en
Keywords: wikipedia,offline,retrieval,semantic-search,faiss,zim
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: libzim<4,>=3.11
Requires-Dist: lxml<7,>=6
Provides-Extra: download
Requires-Dist: huggingface-hub<2,>=0.34; extra == "download"
Provides-Extra: semantic
Requires-Dist: faiss-cpu<2,>=1.11; extra == "semantic"
Requires-Dist: huggingface-hub<2,>=0.34; extra == "semantic"
Requires-Dist: numpy<3,>=2; extra == "semantic"
Requires-Dist: sentence-transformers<6,>=5; extra == "semantic"
Provides-Extra: test
Requires-Dist: pytest<10,>=8; extra == "test"
Provides-Extra: dev
Requires-Dist: build<2,>=1; extra == "dev"
Requires-Dist: pytest<10,>=8; extra == "dev"
Dynamic: license-file

# Forest.Wikipedia.EN

**The English Wikipedia forest: populated, local, and ready for people and agents.**

Forest.Wikipedia.EN turns an offline English Wikipedia canopy into stable article
bearings, authored sections, exact passages, and bounded semantic search. It is
designed to answer *where should I look?* without pretending that retrieval
alone makes something true.

The populated release contains **8,419,072 canonical article bearings** and a
sealed semantic index. Search and canopy navigation work locally. Exact article
depth is fetched from Wikimedia only when a user or agent deliberately enters an
article, then cached by revision and content hash.

> **Canopy search gives directions, not evidence.** A consuming application
> decides what was exposed, why it was selected, and what authority it carries.

## Get the populated Forest

The code lives on GitHub. The large, versioned Wikipedia canopy lives on
[Hugging Face](https://huggingface.co/datasets/schmerbertt/forest-wikipedia-en).

```bash
pip install "wiki-forest[semantic]"
wiki-forest setup
wiki-forest search "milk"
```

`setup` downloads the pinned populated release, resumes interrupted transfers,
verifies every artifact, and creates a fresh local database. Allow about **20 GB
of free disk space**.

Until the package is published, install from a checkout:

```bash
git clone https://github.com/schmerbert/Forest.Wikipedia.EN.git
cd Forest.Wikipedia.EN
python -m venv .venv
# Windows: .venv\Scripts\activate
# Unix:    source .venv/bin/activate
pip install -e ".[semantic,test]"
wiki-forest --root . identity
```

## What is populated?

| Layer | Contents | Approximate size |
|---|---|---:|
| Canopy | Kiwix English Wikipedia `all_mini`, June 2026 | 11.67 GiB |
| Semantic projection | FAISS IVF-PQ index plus SQ8 reranking codes | 3.47 GiB |
| Canonical inventory | Stable mapping for 8.4 million bearings | 0.78 GiB |
| Query model | Pinned `BAAI/bge-small-en-v1.5` snapshot | 0.13 GiB |

Personal runtime state is never part of the populated release. Hydrated article
bodies live under `sources/`, while traversal metadata and revision pins live in
`forest.db`; both begin empty on every new installation.

## Try it

Resolve an exact bearing without entering the article:

```bash
wiki-forest bearing "Milk"
```

Search the unread canopy:

```bash
wiki-forest search "rituals around hospitality" --limit 8
```

Inspect the mounted snapshot and semantic identity:

```bash
wiki-forest identity
wiki-forest status
wiki-forest verify --quick
```

Hydrate exact depth on first entry:

```bash
wiki-forest hydrate "Milk"
wiki-forest children "Milk"
wiki-forest node "Milk" "History.p1"
```

Full verification reads and hashes the entire populated release and can take a
while:

```bash
wiki-forest verify
```

Remove the downloaded canopy, indexes, local database, and hydrated articles:

```bash
wiki-forest remove-data --dry-run
wiki-forest remove-data
pip uninstall wiki-forest
```

The command reports the exact location and size, requires confirmation, and
preserves files it does not own. Use `--yes` only for unattended removal. It
refuses to operate on a source checkout or filesystem root.

Set a custom data location with `WIKI_FOREST_HOME` or pass `--root`. Existing
checkout-based integrations remain supported:

```bash
wiki-forest --root D:\data\wiki-forest identity
```

## Offline boundary

The bundled `all_mini` archive contains every article's title, introduction,
and infobox, without images. Consequently:

- Bearing lookup and canopy search work offline.
- The sealed semantic index works offline after installation.
- Full authored sections and passages require network access on first entry.
- A hydrated revision is pinned and reused locally; it never silently updates.

This is deliberate. Forest.Wikipedia.EN keeps a complete shallow world available while
allowing exact depth to grow only where someone actually walks.

## Stable addresses, readable labels

Every node has a stable machine locator. Existing addresses remain valid:

```text
article              article root
article.p1           first lead passage
History              authored section
History.p1           first passage in History
```

APIs also return `display_title`, so interfaces can show **Lead passage 1** or
**History — passage 1** without exposing internal locator syntax.

## Architecture

```text
canopy bearing
    → article lead
        → section map
            → section
                → subsection
                    → passage
```

- `canopy/` holds the sealed Kiwix archive and checksum.
- `sources/` holds immutable hydrated revision bodies, addressed by SHA-256.
- `projections/` holds rebuildable structural and semantic indexes.
- `forest.db` holds source custody, revision pins, and navigable nodes.
- `models/` holds the pinned query encoder used by semantic retrieval.

The production retriever unions exact and lexical ZIM bearings with FAISS
candidates, reranks a bounded neighborhood from compact SQ8 codes, then applies
diversity selection. Exact title matches cannot be displaced merely because the
approximate index omitted them.

Long-lived applications can keep the verified index and model warm:

```bash
wiki-forest-index serve-production
```

The worker accepts newline-delimited JSON on standard input and emits one result
per line. See
[DESIGN.md](https://github.com/schmerbert/Forest.Wikipedia.EN/blob/main/DESIGN.md)
for the full retrieval design, benchmark
history, integrity model, and rebuild procedure.

## Forest ecosystem

Forest.Wikipedia.EN is usable by itself and is also a populated backend for the
wider Forest ecosystem. Its catalog identity is `forest.wikipedia.en`. It owns
source identity and navigable Wikipedia terrain. A consumer owns the reason for
retrieval: actor, query, exposure, selection,
interpretation, and consequences.

The dotted name is designed to scale as a catalog: `forest.wikipedia.fr`,
`forest.arxiv.cs`, or `forest.docs.personal` can identify other populated
forests without inventing a new project metaphor each time. For compatibility,
the Python package and commands remain `wiki-forest`, the import remains
`wiki_forest`, and the original integration field remains
`forest_id: wikipedia-en-lazy` throughout the `v0.1` line.

That boundary allows many applications to share one sealed Wikipedia substrate
without confusing a retrieved source with locally adopted truth.

## Development

```bash
pip install -e ".[test]"
pytest -q
```

Dense-index construction dependencies are intentionally optional. Build and
benchmark commands live under `wiki-forest-index`; ordinary lexical traversal
does not require FAISS or PyTorch.

## Provenance and license

Forest.Wikipedia.EN code is MIT licensed. The populated release contains independently
licensed Wikipedia, Kiwix/openZIM, and model material. See
[THIRD_PARTY_NOTICES.md](https://github.com/schmerbert/Forest.Wikipedia.EN/blob/main/THIRD_PARTY_NOTICES.md)
before redistributing a bundle.

Forest.Wikipedia.EN is independent and is not endorsed by Wikimedia Foundation, Kiwix,
openZIM, BAAI, or Hugging Face.

## Status

`v0.1.0` is the first public compatibility release. Locators, CLI commands, and
machine-readable identity fields used by existing Forest consumers are treated
as stable within the `v0.1` line.
