Metadata-Version: 2.5
Name: mx-opendata-tools
Version: 0.1.0
Summary: Standard-library clients for Wikidata, Wikipedia, Commons and OpenStreetMap, with the wikitext and geometry helpers that go with them
Project-URL: Homepage, https://github.com/maxim75/opendata-tools
Project-URL: Repository, https://github.com/maxim75/opendata-tools
Project-URL: Issues, https://github.com/maxim75/opendata-tools/issues
Project-URL: Changelog, https://github.com/maxim75/opendata-tools/blob/main/CHANGELOG.md
Author-email: Maksym Kozlenko <max@kozlenko.info>
License-Expression: MIT
License-File: LICENSE
Keywords: commons,geojson,mediawiki,open-data,openstreetmap,overpass,sparql,wikidata,wikimedia,wikipedia,wikitext
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Provides-Extra: all
Requires-Dist: ddgs>=9.14; extra == 'all'
Requires-Dist: langchain>=1.3; extra == 'all'
Provides-Extra: ddgs
Requires-Dist: ddgs>=9.14; extra == 'ddgs'
Provides-Extra: langchain
Requires-Dist: langchain>=1.3; extra == 'langchain'
Description-Content-Type: text/markdown

# mx-opendata-tools

Standard-library clients for Wikidata, Wikipedia, Commons and OpenStreetMap, with
the wikitext and geometry helpers that go with them.

> **Installs as `mx-opendata-tools`, imports as `opendata_tools`.** The plain name
> was unavailable on PyPI — it collapses to an existing `opendatatools` — so the
> distribution carries a personal prefix while the module keeps the readable name.
> The same split as `pillow` → `PIL` or `scikit-learn` → `sklearn`.

This is not a general Wikimedia SDK. It is the set of pieces one open-data project
actually needed and got right — a retry policy tuned by things that really went
wrong, an infobox parser that survives nested templates, geometry that puts a
coordinate *on* a road rather than near it — extracted because a second project
needed the same pieces. Where it is opinionated, the docstring says what the
opinion cost to learn.

```bash
pip install mx-opendata-tools
```

**The core imports nothing but the standard library, and it will stay that way.**
Two optional extras exist and nothing else needs them:

```bash
pip install 'mx-opendata-tools[ddgs]'       # the account-free web-search backend
pip install 'mx-opendata-tools[langchain]'  # the tools, wrapped as LangChain tools
```

The two *paid* search providers need no extra — they are plain HTTP, which the
standard library already speaks.

## Two ways to call it

For one call, use the functions. There is no setup:

```python
from opendata_tools import wikidata_search, sparql

wikidata_search("Kyiv", language="en")
# [{'qid': 'Q1899', 'label': 'Kyiv', 'description': 'capital and largest city of Ukraine'}, …]

sparql("SELECT ?item WHERE { ?item wdt:P31 wd:Q2095 } LIMIT 5")
```

For more than a couple, build a client. This is also how you set the User-Agent —
which the Wikimedia APIs require to identify your application, and will throttle
you for omitting:

```python
from opendata_tools import HttpClient, Wikidata, user_agent

http = HttpClient(
    user_agent=user_agent("street-atlas", "2.1", "https://example.org/street-atlas"),
    timeout=60,
)
wikidata = Wikidata(http=http, languages="uk|en")
wikidata.entities(["Q1899", "Q6436261"])
```

`HttpClient` is frozen, so it is safe to share across threads, and
`http.replace(timeout=600)` gives you a variant for one slow call without
disturbing the original.

## What is in it

### Wikidata

```python
from opendata_tools import wikidata_entities, wikidata_search, sparql

wikidata_entities(["Q1899"], languages="en")  # batched at 50 per call, transparently
sparql("SELECT ?s WHERE { wd:Q1899 rdfs:label ?s } LIMIT 1")
```

`sparql` POSTs, because a query of any size overruns a URL, and returns raw
bindings — flattening them means deciding about datatypes and language tags, and
that belongs to whoever wrote the query.

> **Look a QID up; never recall one.** `Q80895` reads like a plausible guess for
> "asphalt" and is in fact *guerrilla warfare*. Anything deriving an identifier
> from memory is producing fiction that will validate cleanly.

### Wikipedia

```python
from opendata_tools import wikipedia_article, wikipedia_search, resolve_titles

wikipedia_search("Khreshchatyk", lang="en")  # titles + snippets
article = wikipedia_article(
    "Q1076911",
    lang="uk",  # a QID or a title
    infobox_template="Вулиця України",
    sections=("Історія",),
)
article["infobox"], article["lead"], article["sections"]["Історія"], article["links"]

resolve_titles(["Kyiv", "Dnieper"], lang="en")  # titles -> QIDs
```

Search first, then read: `wikipedia_search` returns page titles and
`wikipedia_article` takes one. Web search answers the same question far worse —
short snippets off a rotating set of engines, so the same query gives different
results run to run.

### Commons

```python
from opendata_tools import Commons, commons_page_exists

commons_page_exists("File:Kyiv collage.jpg")  # a read; needs nothing

commons = Commons(http=http)  # a write; needs a bot password
commons.login(user, password)
commons.create("Data:Example.map", text, "summary")  # refuses to overwrite
```

`create` sends `createonly` unless you pass `overwrite=True`, so an existing page
comes back as a server-side refusal rather than being replaced by accident.

### OpenStreetMap and geometry

```python
from opendata_tools import osm_relation, stitch, features, midpoint, centre, endpoints

relation, ways, nodes = osm_relation("421866")
lines = features(stitch(list(ways.values())), nodes, properties={"stroke": "#f00"})

midpoint(lines)  # (lat, lon) — a point *on* the line
centre(lines)  # (lat, lon) — the bounding-box centre; a display hint only
endpoints(lines)  # the two ends, in way order
```

Coordinates go **in** as `(lon, lat)` (GeoJSON order, and what the OSM functions
return) and single points come **out** as `(lat, lon)`, which is how people write
them. Prefer `midpoint` over `centre` for anything that has to sit on the feature:
across fourteen sample streets the bounding-box centre was a median 8.7 m off the
carriageway and at worst 182 m.

### Wikitext

```python
from opendata_tools import parse_template, section, wikilinks, strip_comments
```

`parse_template` tracks brace and bracket depth, because template values contain
nested templates and piped wikilinks whose pipes are not the template's — a plain
`split("|")` shreds them quietly into fragments that look almost right.

### Web search

```python
from opendata_tools import web_search

web_search("query", provider="ddgs")  # no account needed
web_search("query", provider="serper", api_key=key)  # plain HTTP
web_search("query", provider="searlo", api_key=key)
```

All three return `{"provider": …, "results": [{"title", "url", "snippet"}]}`.
**These never raise** — a failure is `{"provider": …, "error": …}` — because web
search is the one call whose failure is routinely uninteresting, and a caller in
the middle of a long run should be able to note it and carry on.

## When things fail

Everything raised descends from `OpenDataError`:

```
OpenDataError
├─ HttpError            .url .what .status .body
│  ├─ HttpStatusError   a final, non-retryable status
│  ├─ TransientError    timeout / dropped connection / unparseable body
│  └─ RetriesExhausted
├─ ApiError             HTTP 200, and the payload reports a failure
│  ├─ MediaWikiError    .code .info
│  └─ LoginError
├─ NotFound · TagMismatch · GeometryError · ConfigurationError
```

Two deliberate exceptions to "raise on failure":

- **`fetch_page` returns** `None` for a 404, `""` for anything unreadable, and the
  text otherwise. The two failures mean opposite things about whatever cited the
  URL: a 404 says the citation points at nothing, a timeout says nothing about the
  citation at all.
- **`web_search` returns** an `error` key, as above.

## Logging

The library configures no logging and attaches a `NullHandler`. Retries are logged
at `WARNING` and one line per request at `DEBUG` — **never** headers or bodies, so
an API key or a login POST cannot reach a log file.

A retrying run looks like a hang if you never turn these on. In a CLI:

```python
import logging

logging.basicConfig(level=logging.WARNING, format="%(message)s")
```

## Keys and `.env`

```python
from opendata_tools import load_env_file, require_env

load_env_file(".env")  # returns the NAMES it set, never the values
require_env("SERPER_API_KEY", hint="Create one at https://serper.dev")
```

A real environment variable always wins over the file, so a CI secret beats a
stale local `.env` without anyone having to remember. Values are never returned
alongside their names, logged, or put in an error message.

## Tests

```bash
uv run pytest                                              # offline; no network
OPENDATA_TOOLS_INTEGRATION=1 uv run pytest -m integration   # hits the live APIs
```

Integration tests are gated by a marker *and* an environment variable, so a bare
`pytest` is green with no network and even an explicit `pytest -m integration`
stays skipped. Reaching the real APIs has to be deliberate. There is no live test
that writes to Commons — a test that edits a public wiki is not a test.

## Versioning

Semantic versioning from 0.1.0. **While the major version is 0, a breaking change
bumps the minor** — pin `mx-opendata-tools>=0.1,<0.2`. See [CHANGELOG.md](CHANGELOG.md).

## Licence

MIT.
