Metadata-Version: 2.4
Name: wiki2okf
Version: 0.2.1
Summary: Convert Wikipedia articles into Open Knowledge Format bundles.
Author: Lucas Marchand
License-Expression: MIT
Project-URL: Homepage, https://github.com/LucasMarchnd/wiki2okf
Project-URL: Repository, https://github.com/LucasMarchnd/wiki2okf
Project-URL: Issues, https://github.com/LucasMarchnd/wiki2okf/issues
Keywords: wikipedia,knowledge,markdown,okf,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Natural Language :: French
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.8
Requires-Dist: pyyaml>=6.0
Requires-Dist: python-slugify>=8.0
Requires-Dist: typer>=0.12
Provides-Extra: llm
Requires-Dist: litellm<2,>=1.60; extra == "llm"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-httpx>=0.30; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Dynamic: license-file

# wiki2okf

[![PyPI](https://img.shields.io/pypi/v/wiki2okf)](https://pypi.org/project/wiki2okf/)
[![Python](https://img.shields.io/pypi/pyversions/wiki2okf)](https://pypi.org/project/wiki2okf/)
[![CI](https://github.com/LucasMarchnd/wiki2okf/actions/workflows/ci.yml/badge.svg)](https://github.com/LucasMarchnd/wiki2okf/actions/workflows/ci.yml)
[![License](https://img.shields.io/github/license/LucasMarchnd/wiki2okf)](LICENSE)

Convert a Wikipedia article into a compact, local
[Open Knowledge Format (OKF) v0.2][okf-spec] bundle.

wiki2okf turns Wikipedia's readable content, categories, and internal links
into inspectable Markdown files with structured YAML metadata. It works
deterministically by default and can optionally use an LLM to produce a more
concise, structured knowledge concept.

```bash
uvx wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing
```

```text
alan-turing/
├── index.md
├── log.md
└── concepts/
    └── alan-turing.md
```

## Highlights

- **Small by default** — concise mode keeps the introduction and main
  encyclopedic sections while removing references, navigation, media lists,
  and other noise.
- **Deterministic** — the standard pipeline requires no LLM, account, API key,
  or optional dependency.
- **Agent-friendly** — output is plain Markdown and YAML that remains easy to
  inspect, version, edit, and process.
- **Traceable** — generated concepts preserve the Wikipedia URL, revision,
  retrieval time, and generator version.
- **Optionally enriched** — provider-agnostic LLM enrichment adds grounded
  facts, typed relations, aliases, and useful questions.
- **Safe rendering** — models return validated structured data; wiki2okf, not
  the model, owns the final YAML and Markdown rendering.

## Installation

wiki2okf requires Python 3.11 or newer.

Install it as a command-line tool:

```bash
uv tool install wiki2okf
```

Add it to a project:

```bash
uv add wiki2okf
```

Or use pip:

```bash
pip install wiki2okf
```

LLM enrichment is an optional extra:

```bash
uv tool install "wiki2okf[llm]"
# or
pip install "wiki2okf[llm]"
```

The base package does not install LiteLLM or provider SDKs.

## Quick start

Convert one English or French Wikipedia article:

```bash
wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing
```

The default `concise` mode:

- retains the introduction and main encyclopedic sections;
- excludes references, bibliographies, external links, portal lists, and
  navigation;
- caps the article body at 12,000 characters;
- adds an explicit notice when content is truncated.

Use `full` mode to retain more meaningful source sections:

```bash
wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --content-mode full \
  --output ./alan-turing-full
```

Both modes still remove empty sections and navigation noise.

## How it works

```text
Wikipedia URL
    │
    ▼
MediaWiki API
    │
    ▼
cleaned WikipediaArticle
    │
    ├── deterministic rendering
    │
    └── optional validated LLM enrichment
             │
             ▼
       deterministic rendering
    │
    ▼
local OKF bundle
```

The output always passes through the same internal article model and
deterministic renderer. LLM responses are validated with Pydantic and never
write frontmatter or final Markdown directly.

## Example output

The generated concept is ordinary Markdown with YAML frontmatter:

```markdown
---
type: Wikipedia Article
title: Alan Turing
description: English mathematician and computer scientist (1912–1954).
resource: https://en.wikipedia.org/wiki/Alan_Turing
tags:
  - Alan Turing
  - British computer scientists
generated:
  by: wiki2okf/0.2.1
  at: 2026-07-25T10:30:00Z
status: stable
sources:
  - id: wikipedia-page
    resource: https://en.wikipedia.org/wiki/Alan_Turing
    title: Alan Turing — Wikipedia
    revision_id: 1300000000
---

# Alan Turing

## Early life and education

Turing was born in Maida Vale, London, and was educated at Sherborne School.

## Related concepts

- [Turing machine](https://en.wikipedia.org/wiki/Turing_machine)
- [Cryptanalysis](https://en.wikipedia.org/wiki/Cryptanalysis)

[^wikipedia-page]: Wikipedia article retrieved automatically.
```

`index.md` provides bundle navigation, while `log.md` records the import.

## Optional LLM enrichment

Enrichment converts the cleaned source into a shorter, structured concept while
keeping the original provenance:

```bash
export GEMINI_API_KEY="your-key"

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --output ./alan-turing
```

The pipeline uses two passes:

1. Each section is assessed independently for relevance and summarized into
   grounded key points.
2. Retained points are combined into a concept type, concise description,
   key facts, typed relations, aliases, related concepts, and useful questions.

Every fact and relation must reference its supporting Wikipedia section.
Unsupported section references, malformed responses, and schema violations are
rejected.

### Supported providers

Model routing is handled by [LiteLLM][litellm]. Use any compatible
provider/model identifier and its standard environment variable.

| Provider | Model identifier example | Authentication |
|---|---|---|
| Google AI Studio | `gemini/gemini-3.1-flash-lite` | `GEMINI_API_KEY` |
| OpenAI | `openai/<model>` | `OPENAI_API_KEY` |
| Anthropic | `anthropic/<model>` | `ANTHROPIC_API_KEY` |
| Ollama | `ollama_chat/qwen3.5:4b` | None |

Local Ollama example:

```bash
ollama pull qwen3.5:4b

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "ollama_chat/qwen3.5:4b" \
  --fallback fail \
  --output ./alan-turing
```

Small local models may struggle with strict schemas and source grounding.
`--fallback fail` is useful when evaluating a model because it makes those
failures visible.

### Caching and fallback

Validated enrichment responses are cached using the article revision, model,
prompt version, schema version, pipeline stage, temperature, and content hash:

```bash
wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --cache-dir ~/.cache/wiki2okf
```

The cache never contains raw prompts or raw provider responses.

If enrichment fails, the default behavior is to print a warning and produce
the deterministic bundle. Use `--fallback fail` to stop instead:

```bash
wiki2okf convert URL \
  --enrich \
  --model PROVIDER/MODEL \
  --fallback fail
```

Use `--keep-raw` to append the cleaned Wikipedia sections after the enriched
concept.

## CLI

```text
wiki2okf convert URL [OPTIONS]

Options:
  -o, --output PATH                 Output directory
      --content-mode concise|full   Content selection (default: concise)
      --enrich                      Enable structured LLM enrichment
      --model PROVIDER/MODEL        LiteLLM model identifier
      --temperature FLOAT           Sampling temperature (default: 0)
      --fallback deterministic|fail Enrichment failure behavior
      --keep-raw                    Keep cleaned source after enrichment
      --cache-dir PATH              Validated response cache directory
```

Run `wiki2okf convert --help` for the authoritative command reference.

## Scope

wiki2okf currently supports:

- one French or English Wikipedia article at a time;
- MediaWiki `action=parse` responses;
- introductions and readable `h2`/`h3` sections;
- categories and up to 20 internal article links;
- concise and full content modes;
- deterministic OKF Markdown and YAML;
- optional two-pass LLM enrichment with validation, caching, and fallback.

It intentionally does not include recursive crawling, Wikidata, structured
infoboxes, embeddings, a vector database, graph visualization, a web interface,
or bulk Wikipedia dump processing.

## Development

```bash
git clone https://github.com/LucasMarchnd/wiki2okf.git
cd wiki2okf

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,llm]"

ruff check .
pytest
python -m build
```

Tests use recorded MediaWiki fixtures and fake LLM implementations. They never
call model provider APIs.

## Roadmap

- depth-one crawling with local OKF links;
- Wikidata identifiers and structured infoboxes;
- incremental updates based on Wikipedia revision IDs;
- official OKF conformance validation;
- graph visualization and bulk dump processing;
- stable Python API and source plugins.

## Attribution

Generated bundles preserve their Wikipedia source URL and revision. Wikipedia
content remains subject to the licensing and attribution requirements stated
by Wikimedia and on each source article.

wiki2okf is an independent project. It is not affiliated with or endorsed by
Google, Google Cloud, the Wikimedia Foundation, Wikipedia, or any LLM provider.

## License

Released under the [MIT License](LICENSE).

[litellm]: https://docs.litellm.ai/
[okf-spec]: https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md
