Metadata-Version: 2.4
Name: reapx
Version: 0.1.0
Summary: A tiny, dependency-free client over reapx's public open data: sources, entity pages and their Hugging Face mirrors.
Author-email: reapx <reapxdev@proton.me>
Maintainer-email: reapx <reapxdev@proton.me>
License: MIT
Project-URL: Homepage, https://reapx.dev
Project-URL: Documentation, https://reapx.dev/data/
Project-URL: Machine index, https://reapx.dev/llms.txt
Project-URL: Dataset mirrors, https://huggingface.co/reapxdev
Keywords: open-data,datasets,scraping,public-data,sec-edgar,arxiv,openfda,coingecko,huggingface
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Database :: Front-Ends
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# reapx

A tiny, dependency-free Python client over reapx's public open data.

[reapx.dev](https://reapx.dev) publishes one page per entity returned by a public-source
scraper - one page for `apache/airflow`, one for Apple's SEC filer record, one for every
arXiv paper that a run actually returned. Each page carries the observations behind it and
the id of the run that produced them. This package reads that surface from Python: it lists
the sources, searches the entities under one, and hands you back the rows.

Everything is standard library. No requests, no BeautifulSoup, no lxml, nothing to resolve.
Python 3.9 and up.

## Install

```bash
pip install reapx
```

## Quickstart

```python
import reapx

for source in reapx.sources():
    print(source.slug, source.pages, source.key)

hits = reapx.search("sec-edgar-scraper", "apple", limit=3)
record = reapx.get("github-repo-scraper", "airflow")
print(record.observations[0]["language"])
```

## The three functions

### `sources()`

Reads [the machine index at reapx.dev/llms.txt](https://reapx.dev/llms.txt), which is the
site's own declaration of what exists, and returns a `Source` for each one.

```python
>>> s = [x for x in reapx.sources() if x.slug == "sec-edgar-scraper"][0]
>>> s.title
'SEC EDGAR Scraper — Filings, Financials & Full-Text Search'
>>> s.key, s.pages, s.rows
('cik', 771, 4125)
>>> s.huggingface
'reapxdev/sec-edgar-scraper'
```

`Source` carries `slug`, `title`, `description`, `url`, `pages`, `rows`, `key` (the field the
entity pages are keyed on), `newest`, `actor_url`, and two derived properties, `huggingface`
and `huggingface_url`, which point at that source's mirror on the Hugging Face Hub.

### `search(source, query, limit=20)`

Case-insensitive substring match over the entity slugs and labels under one source. Entity
names are identifiers - tickers, package names, repo slugs, CIK numbers - so a substring is
the right tool; there is no ranking and none is pretended.

```python
>>> for h in reapx.search("sec-edgar-scraper", "apple", limit=3):
...     print(h.entity, "|", h.label)
0000320193 | Apple Inc.
0001938109 | Pineapple Financial Inc.
aapl | Apple Inc.
```

Hubs are paginated at 250 entities per page, and `search` walks them lazily through
`rel="next"`, stopping the moment it has `limit` matches. An empty query returns the first
`limit` entities in the source's own order. If you want to iterate the whole source rather
than search it, `reapx.entities(source)` is a generator over the same walk.

### `get(source, entity)`

Fetches one entity page and returns a `Record`: the metadata, the observations, and the run
ids behind them.

```python
>>> r = reapx.get("coingecko-scraper", "bch")
>>> r.name
'Bitcoin Cash — coingecko scraper'
>>> len(r.fields)
25
>>> r.observations[0]["currentPrice"]
'1067.98'
>>> r.runs
['0NNZkkxvV7oBS0y0a', 'EHkLvzXn94bZ4hDWd', 'Ei5mJGc17bd3fuCfU', 'P7kQr8Wm5wbEy9nIi', ...]
```

That `runs` list is the point of the whole thing. An entity page is the union of every run
that ever returned rows for that entity, so Bitcoin Cash is backed by thirteen separate runs
and says so. Nothing on the page is estimated, modelled or filled in; if a figure is there,
a run returned it.

`Record` fields: `source`, `entity`, `name`, `url`, `description`, `fields` (every measured
field the source declares), `observations` (the rows shown on the page, as dicts), `runs`,
`updated`, `license`, `keywords`, `temporal_coverage`, `actor_url`, plus `huggingface` and
`huggingface_url`. `record.to_dict()` gives you a JSON-serialisable dict.

## Command line

The package installs a `reapx` command, and `python -m reapx` does the same thing.

```console
$ reapx sources
app-store-reviews-scraper           166 pages    22,375 rows  key=appId
arbeitsagentur-scraper            3,555 pages    23,618 rows  key=city
arxiv-papers-scraper             10,594 pages    21,922 rows  key=arxivId
clinicaltrials-scraper            8,175 pages    11,191 rows  key=nctId
coingecko-scraper                 4,828 pages     9,951 rows  key=id
crossref-scraper                  2,492 pages     2,550 rows  key=doi
discogs-scraper                   3,013 pages    14,515 rows  key=handle
docker-hub-scraper                2,805 pages     2,835 rows  key=slug
federal-register-scraper          6,210 pages     9,784 rows  key=citation
github-repo-scraper               2,071 pages     2,481 rows  key=slug
...
```

```console
$ reapx search coingecko-scraper bitcoin -n 4
0xBitcoin
    0xbtc
    https://reapx.dev/data/coingecko-scraper/0xbtc/
BitcoinII
    bc2
    https://reapx.dev/data/coingecko-scraper/bc2/
Bitcoin Atom
    bca
    https://reapx.dev/data/coingecko-scraper/bca/
Bitcoin Cash
    bch
    https://reapx.dev/data/coingecko-scraper/bch/

4 match(es)
```

```console
$ reapx get github-repo-scraper airflow
apache/airflow — github repo scraper
https://reapx.dev/data/github-repo-scraper/airflow/
updated  2026-08-03T22:13:12+00:00
runs     k6WwXwZp1DPqPimjS
covers   2015-04-13/2026-08-03
mirror   https://huggingface.co/datasets/reapxdev/github-repo-scraper

1 observation(s):
  createdAt              2015-04-13T18:04:58Z
  defaultBranch          main
  description            Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
  forks                  17514
  fullName               apache/airflow
  htmlUrl                https://github.com/apache/airflow
  isArchived             False
  isDisabled             False
  isFork                 False
  language               Python
  license                apache-2.0
  licenseName            Apache License 2.0
```

Add `--json` to any command to get machine-readable output instead:

```bash
reapx --json get sec-edgar-scraper 0000320193 | jq '.observations[0]'
```

## Bulk data

This client reads one page at a time, which is the right shape for looking something up and
the wrong shape for training on a whole source. For bulk, every source is mirrored as a
dataset under [the reapxdev organisation on Hugging Face](https://huggingface.co/reapxdev),
and each `Source` tells you which one:

```python
>>> import reapx
>>> reapx.sources()[0].huggingface_url
'https://huggingface.co/datasets/reapxdev/app-store-reviews-scraper'
```

Each mirror holds `<slug>.jsonl` and `<slug>.csv`, so pulling a whole source needs nothing
beyond the standard library either:

```python
>>> import json, urllib.request, reapx
>>> s = [x for x in reapx.sources() if x.slug == "github-repo-scraper"][0]
>>> url = s.huggingface_url + "/resolve/main/" + s.slug + ".jsonl"
>>> with urllib.request.urlopen(url) as r:
...     rows = [json.loads(line) for line in r.read().decode().splitlines() if line.strip()]
>>> len(rows)
2481
>>> rows[0]["fullName"]
'tailwindlabs/tailwindcss'
```

That 2481 is the same row count `sources()` reports for the source, which is the point: the
mirror and the pages are built from the same runs. If you already use the Hugging Face
tooling, `load_dataset("reapxdev/github-repo-scraper", data_files="github-repo-scraper.jsonl")`
reads the same file.

If you would rather collect fresh rows than read published ones, each source is a scraper you
can run yourself; `Source.actor_url` and `Record.actor_url` link to it.

## Caching and politeness

reapx.dev is a static site behind a CDN, but it is still somebody else's bandwidth. Every
response is cached on disk for an hour, so a repeated `search` over the same source costs
nothing after the first walk.

- Cache location: `$REAPX_CACHE_DIR`, else `$XDG_CACHE_HOME/reapx`, else `~/.cache/reapx`.
- Disable entirely with `REAPX_NO_CACHE=1`, or per call with `cache=False`.
- Change the TTL per call with `ttl=`, or pass `ttl=-1` to keep entries forever.

Requests carry a `reapx-python/<version>` user agent, retry with backoff on 429 and 5xx, and
give up after three attempts.

## Errors

Everything raised inherits `reapx.ReapxError`.

| exception | when |
| --- | --- |
| `SourceNotFound` | no source has that slug; call `sources()` for the list |
| `EntityNotFound` | that source publishes no page for that entity |
| `NotFound` | a URL returned 404 |
| `HTTPError` | any other unsuccessful status, or the host was unreachable |
| `ParseError` | a page was fetched but did not have the expected shape |

## Data licence and provenance

The published data is CC BY 4.0. Each entity page names the runs that produced it, and
`Record.runs` gives you those ids, so any figure you take from this package can be traced
back to the run that returned it. The underlying sources are public APIs and public
websites; each source page on [reapx.dev](https://reapx.dev/data/) names which.

This package is a client, not the data. It reads the same public URLs your browser would.

## Tests

The test suite runs against the live site on purpose - a parser that passes fixtures and
fails production is worthless.

```bash
python -m unittest discover -s tests -v
REAPX_SKIP_LIVE=1 python -m unittest discover -s tests   # offline: parsers only
```

## Licence

MIT.
