Metadata-Version: 2.4
Name: wiki2okf
Version: 0.2.0
Summary: Convert Wikipedia articles into Open Knowledge Format bundles.
Author: Lucas Marchand
License-Expression: MIT
Project-URL: Homepage, https://github.com/LucasMarchnd/wiki2okf
Project-URL: Repository, https://github.com/LucasMarchnd/wiki2okf
Project-URL: Issues, https://github.com/LucasMarchnd/wiki2okf/issues
Keywords: wikipedia,knowledge,markdown,okf,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Natural Language :: French
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.8
Requires-Dist: pyyaml>=6.0
Requires-Dist: python-slugify>=8.0
Requires-Dist: typer>=0.12
Provides-Extra: llm
Requires-Dist: litellm<2,>=1.60; extra == "llm"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-httpx>=0.30; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Dynamic: license-file

# wiki2okf

[![CI](https://github.com/LucasMarchnd/wiki2okf/actions/workflows/ci.yml/badge.svg)](https://github.com/LucasMarchnd/wiki2okf/actions/workflows/ci.yml)
![Python](https://img.shields.io/badge/python-3.11%2B-blue)
![License](https://img.shields.io/badge/license-MIT-green)
![Status](https://img.shields.io/badge/status-experimental-orange)

Convert one Wikipedia article into a small, local, agent-ready
[Open Knowledge Format (OKF) v0.2][okf-spec] bundle.

```bash
wiki2okf convert \
  "https://fr.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing
```

```text
alan-turing/
├── index.md
├── log.md
└── concepts/
    └── alan-turing.md
```

wiki2okf provides two paths:

- a fully deterministic converter with no LLM dependency, account, or API key;
- optional provider-agnostic enrichment for compact, structured knowledge.

Both paths keep the final YAML and Markdown generation inside wiki2okf. An LLM
can return validated data, but it never writes the OKF files directly.

## Why?

Wikipedia contains richly linked public knowledge, but full articles are often
too large and noisy for an agent context. OKF provides a portable,
Markdown-based way to represent concepts, sources, and relationships.

wiki2okf bridges both formats through an inspectable pipeline:

```text
Wikipedia URL
    ↓
MediaWiki API
    ↓
cleaned WikipediaArticle
    ↓
optional two-pass enrichment
    ↓
validated models
    ↓
deterministic OKF renderer
```

## Installation

wiki2okf requires Python 3.11 or newer.

### With uv

Once the first PyPI release is published, add the deterministic package to a
project:

```bash
uv add wiki2okf
uv run wiki2okf --help
```

Include optional LLM enrichment:

```bash
uv add "wiki2okf[llm]"
```

Install it as a standalone command:

```bash
uv tool install wiki2okf
```

Or run it without a permanent installation:

```bash
uvx wiki2okf convert \
  "https://fr.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing
```

Until the first PyPI release is available, install the current GitHub version:

```bash
uv tool install \
  "wiki2okf @ git+https://github.com/LucasMarchnd/wiki2okf.git"
```

### With pip

```bash
pip install wiki2okf
```

For enrichment:

```bash
pip install "wiki2okf[llm]"
```

The base installation intentionally excludes LiteLLM and provider SDKs.

## Deterministic conversion

The default command uses no model and follows the same inspectable conversion
rules for identical Wikipedia content:

```bash
wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing
```

It retrieves the article through MediaWiki, cleans the HTML, extracts
categories and internal links, and renders the result without an LLM.

### Content modes

`concise` is the default:

```bash
wiki2okf convert URL --content-mode concise
```

It keeps the introduction and main encyclopedic sections, excludes references,
bibliographies, media lists, portals, and navigation, and limits the article
body to 12,000 characters with an explicit truncation notice.

`full` retains more meaningful source sections while still removing navigation
and portal noise:

```bash
wiki2okf convert URL --content-mode full
```

## Optional LLM enrichment

Install the `llm` extra, then select any model identifier supported by
[LiteLLM][litellm]:

```bash
wiki2okf convert \
  "https://fr.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --fallback fail \
  --output ./alan-turing
```

Enrichment uses two passes:

1. Every cleaned section is assessed independently. The model assigns a
   relevance score and extracts up to eight grounded points.
2. The introduction and retained points are synthesized into a concise concept
   with a description, facts, typed relations, aliases, related concepts, and
   useful questions.

Every response is validated with Pydantic. Every fact and relation must name
the exact Wikipedia sections supporting it. Unsupported section references,
invalid limits, inconsistent relevance decisions, and malformed responses are
rejected.

### Providers

wiki2okf does not import provider-specific SDKs in its domain or rendering
layers. LiteLLM handles provider routing.

| Provider | Example model | Authentication |
|---|---|---|
| Google AI Studio | `gemini/gemini-3.1-flash-lite` | `GEMINI_API_KEY` |
| OpenAI | `openai/<model>` | `OPENAI_API_KEY` |
| Anthropic | `anthropic/<model>` | `ANTHROPIC_API_KEY` |
| Ollama | `ollama_chat/<model>` | None |

Google AI Studio example:

```bash
export GEMINI_API_KEY="..."

wiki2okf convert URL \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite"
```

Local Ollama example:

```bash
ollama pull qwen3.5:4b

wiki2okf convert URL \
  --enrich \
  --model "ollama_chat/qwen3.5:4b"
```

Small local models can struggle with strict schemas and grounding. Use
`--fallback fail` while evaluating a model so failures are visible.

### Cache and fallback

Validated responses are cached by:

- Wikipedia revision;
- provider/model identifier;
- prompt and schema versions;
- content hash;
- pipeline stage;
- temperature.

```bash
wiki2okf convert URL \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --cache-dir ~/.cache/wiki2okf
```

Cache entries contain validated structured data, never raw prompts or raw API
responses.

The default fallback still produces the deterministic bundle if enrichment
fails:

```bash
wiki2okf convert URL --enrich --model MODEL \
  --fallback deterministic
```

For CI, benchmarking, or model evaluation:

```bash
wiki2okf convert URL --enrich --model MODEL \
  --fallback fail
```

Use `--keep-raw` to append the cleaned source sections after the enriched
concept.

## Generated concept

An enriched concept remains ordinary Markdown with safe YAML frontmatter:

```markdown
---
type: Mathématicien et cryptologue
title: Alan Turing
description: Alan Turing est un mathématicien et cryptologue britannique…
resource: https://fr.wikipedia.org/wiki/Alan_Turing
generated:
  by: wiki2okf/0.2.0
  at: 2026-07-24T23:00:05Z
  enrichment:
    provider: litellm
    model: gemini/gemini-3.1-flash-lite
    prompt_version: 2026-07-25.v2
    schema_version: "2"
    temperature: 0.0
sources:
  - id: wikipedia-page
    resource: https://fr.wikipedia.org/wiki/Alan_Turing
    title: Alan Turing — Wikipédia
    revision_id: 236602045
---

# Alan Turing

## Faits clés

- Alan Turing est né à Londres le 23 juin 1912. (Source: Biographie)

## Relations

- Alan Turing — a travaillé à → Bletchley Park (Source: Biographie)
```

The bundle also contains:

- `index.md` for navigation;
- `log.md` for import history;
- up to 20 related Wikipedia links;
- source URL, revision identifier, retrieval time, and generator provenance.

## CLI reference

```text
wiki2okf convert URL [OPTIONS]

--output, -o PATH
--content-mode concise|full
--enrich
--model PROVIDER/MODEL
--temperature FLOAT
--fallback deterministic|fail
--keep-raw
--cache-dir PATH
```

## Current scope

Included:

- one French or English Wikipedia URL;
- MediaWiki `action=parse`;
- introduction and readable `h2`/`h3` sections;
- categories and internal article links;
- concise and full content modes;
- deterministic Markdown and YAML;
- optional two-pass LLM enrichment;
- Pydantic validation, one JSON repair attempt, caching, and fallback;
- local `index.md`, `log.md`, and concept files.

Intentionally not included:

- recursive crawling;
- downloading linked pages;
- Wikidata and structured infoboxes;
- embeddings or a vector database;
- graph visualization;
- a web interface;
- bulk Wikipedia dump processing.

## Development

```bash
git clone https://github.com/LucasMarchnd/wiki2okf.git
cd wiki2okf

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,llm]"

ruff check .
pytest
python -m build
```

Unit tests use fake LLM implementations and never call external model APIs.

## Publishing to PyPI

The repository includes a Trusted Publisher workflow in
`.github/workflows/publish.yml`. It builds and checks the wheel and source
distribution, then publishes with GitHub OIDC. No long-lived PyPI token is
stored in GitHub.

For the first release, configure a pending publisher on PyPI with:

```text
PyPI project: wiki2okf
GitHub owner: LucasMarchnd
Repository: wiki2okf
Workflow: publish.yml
Environment: pypi
```

Then publish a GitHub Release for the matching version. The workflow uploads
the package, after which these commands become available:

```bash
uv add wiki2okf
uv tool install wiki2okf
pip install wiki2okf
```

## Roadmap

- depth-1 crawling and local OKF links;
- Wikidata identifiers and structured infoboxes;
- incremental updates using `revision_id`;
- official OKF conformance validation;
- graph visualization;
- bulk Wikipedia dump processing;
- stable Python API and source plugins.

## Attribution and independence

Generated bundles preserve the Wikipedia source URL and revision. Wikipedia
content remains subject to the licensing and attribution requirements stated
by Wikimedia and on the source article.

wiki2okf is independent and is not affiliated with or endorsed by Google,
Google Cloud, the Wikimedia Foundation, Wikipedia, or any LLM provider.

## License

wiki2okf is released under the [MIT License](LICENSE).

[litellm]: https://docs.litellm.ai/
[okf-spec]: https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md
