Metadata-Version: 2.4
Name: fetchkit-agents
Version: 0.5.1
Summary: A YAML-configured data fetching library for agentic applications.
Author-email: mete morris <morrismete@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/metemorris/fetchkit
Project-URL: Repository, https://github.com/metemorris/fetchkit
Project-URL: Issues, https://github.com/metemorris/fetchkit/issues
Keywords: rss,hackernews,fetcher,data-collection,agents,llm,feed
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: requests>=2.28
Requires-Dist: feedparser>=6.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: responses>=0.23; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Provides-Extra: discovery
Provides-Extra: discovery-embeddings
Requires-Dist: numpy>=1.21; extra == "discovery-embeddings"
Requires-Dist: sentence-transformers>=2.2; extra == "discovery-embeddings"
Provides-Extra: discovery-external
Dynamic: license-file

# fetchkit

[![CI](https://github.com/metemorris/fetchkit/actions/workflows/ci.yml/badge.svg)](https://github.com/metemorris/fetchkit/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/fetchkit-agents.svg)](https://pypi.org/project/fetchkit-agents/)
[![Python versions](https://img.shields.io/pypi/pyversions/fetchkit-agents.svg)](https://pypi.org/project/fetchkit-agents/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

A YAML-configured data-fetching library for agentic applications.

`fetchkit` collects posts and comments from sources (Hacker News, RSS/Atom, arXiv,
GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT,
SEC EDGAR, Polymarket, Manifold) into a single canonical `Post`
model, with de-duplication,
deterministic sorting, and a shared HTTP client with retries and rate limiting. It is
designed as the data-collection layer for LLM/agent pipelines — feed it configs, get
back clean typed data.

- **YAML-first** configuration with strict validation: unknown or misspelled keys
  are rejected up front, so a bad config fails loudly instead of silently.
- **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange,
  Bluesky, Mastodon, Reddit, Google News, GDELT (global news firehose), SEC EDGAR
  (securities filings), and Polymarket/Manifold (prediction markets) — all zero-auth
  by default. For the sources that block anonymous datacenter/cloud traffic there are
  opt-in authenticated variants (`reddit-auth`, `bluesky-auth`) whose credentials come
  from environment variables — see
  [Authenticated fetcher types and secrets](#authenticated-fetcher-types-and-secrets).
- **Relative time windows**: say `window: "last 6 hours"` instead of computing
  timestamps.
- **Open `metadata`**: each `Post` carries a `metadata` dict for source-specific
  detail (arXiv categories/DOI, GitHub language/stars, tags) without bloating the
  core model.
- **Robust HTTP**: shared session pooling, exponential backoff + retries, `Retry-After`
  handling, and optional per-host rate limiting.
- **Identify your project**: set `http.user_agent` (or `$FETCHKIT_USER_AGENT`) once and
  every request carries your project's identity — see
  [Identify your project (User-Agent)](#identify-your-project-user-agent).
- **Typed end-to-end**: Pydantic v2 models throughout.
- **Deterministic output**: dedup by `(source, id)` — or across sources by normalized
  article URL with `dedup: url` — sorted descending by `(created_at, id)`.
- **Agent-friendly CLI**: `fetchkit run config.yaml` emits clean JSON on stdout —
  no Python required. `fetchkit example <type>` prints a commented, runnable
  config for any source, and `fetchkit guide` prints an agent guide covering
  field semantics and per-source gotchas.
- **Uniform text surface**: `Post.text` carries the best available body at every
  source (arXiv abstract, GitHub description/README, RSS content, HN self-post…)
  — null only when the source genuinely has no body. See
  [The `Post` model](#the-post-model).
- **RSS feed discovery**: `fetchkit discover "<use case>"` maps a natural-language
  query onto real RSS feeds an agent can fetch — closing the "which feed URL?" gap.
- **Per-source discovery**: `fetchkit suggest <source>` lists the selectable knobs
  for any source (tags, sites, arXiv categories, trending hashtags, Bluesky feeds) —
  all zero-auth — so an agent can fill a config without guessing.

## Install

```bash
pip install fetchkit-agents     # PyPI distribution name

# or install the latest from source:
pip install "git+https://github.com/metemorris/fetchkit.git"
```

The PyPI package is `fetchkit-agents` (the `fetchkit` name was taken), but you still
`import fetchkit` and the CLI command is `fetchkit`. Requires Python ≥ 3.10.

## Quick start

### 1. YAML config

`config.yaml`:

```yaml
window: "last 24 hours"   # or set explicit start_time / end_time
fetchers:
  - type: hackernews
    posts:
      max_items: 50
      order: new
  - type: arxiv
    categories: ["cs.AI", "cs.LG"]
    max_items: 40
  - type: github
    resource: releases
    repos: ["python/cpython", "pydantic/pydantic"]
  - type: lobsters
    listing: hottest
  - type: stackexchange
    site: stackoverflow
    tagged: ["python", "asyncio"]
  - type: bluesky
    resource: search
    query: "large language models"
  - type: mastodon
    instance: mastodon.social
    tag: ai
  - type: edgar
    companies: ["AAPL", "MSFT"]
    forms: ["8-K", "10-Q"]
  - type: reddit
    subreddits: ["news", "wallstreetbets"]
    listing: new
  - type: googlenews
    resource: search
    query: "nvidia earnings when:6h"
  - type: gdelt
    query: '"interest rates"'
    languages: ["english"]
  - type: polymarket
    query: "fed"
  - type: rss
    feeds:
      - url: "https://feeds.bbci.co.uk/news/rss.xml"
        name: "BBC News"
    max_items_per_feed: 40
    max_total_items: 200
http:           # optional
  user_agent: "my-news-pipeline/1.0 (contact@example.com)"   # identify YOUR project
  timeout: 15
  max_retries: 5
  rate_limit_per_host: 2.0
```

### 2. Run

```python
from fetchkit import load_config, collect_all

config = load_config("config.yaml")
result = collect_all(config)

print(f"Collected {len(result.posts)} posts")
for post in result.posts:
    print(post.created_at, post.source, post.title, post.url)

if result.has_errors:
    for source, err in result.errors:
        print(f"  {source}: {err}")
```

### 3. Command line (for agents & scripts)

`fetchkit` installs a CLI so an agent can shell out and parse JSON without writing
Python. `run` prints **only** JSON to stdout (diagnostics go to stderr), so it pipes
cleanly into `jq`:

```bash
fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "sources", "errors"}
fetchkit run config.yaml -o out.json        # write JSON to a file (stdout stays empty)
fetchkit run config.yaml --window "6h"      # override the time window at runtime
fetchkit run config.yaml --compact          # single-line JSON
fetchkit run config.yaml --fail-on-error    # exit 1 if any source failed
echo "$CONFIG_YAML" | fetchkit run -        # read the config from stdin
fetchkit validate config.yaml               # validate a config without fetching (also accepts -)
fetchkit schema                             # JSON Schema for every config/output model
fetchkit example arxiv                      # commented, runnable YAML config for a source
fetchkit example arxiv | fetchkit run -     # …which pipes straight into run
fetchkit guide                              # agent guide: field semantics + per-source gotchas
python -m fetchkit run config.yaml          # module form, identical behavior
```

`fetchkit run --help` and `fetchkit validate --help` print the YAML config format
inline (top-level keys, a minimal example, and the fetcher type list), so simple
configs need no schema round-trip.

`fetchkit schema` lets an agent discover what it can fetch — it prints the JSON
Schema (with field descriptions) for the top-level config, the shared HTTP
settings, every builtin fetcher's typed config, and the canonical `Post` output —
so a config can be written without knowing the YAML format out of band. Pass a
section name to print just that part instead of the full ~50 KB document:

```bash
fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","bluesky-auth","edgar","gdelt","github","googlenews","hackernews","lobsters","manifold","mastodon","polymarket","reddit","reddit-auth","rss","stackexchange"]
fetchkit schema config                      # just the top-level config schema
fetchkit schema rss                         # just one fetcher's config schema
fetchkit schema post                        # just the output Post model
fetchkit schema -o schema.json              # write to a file; supports --compact too
```

Output shape — `sources` reports per-fetcher success/failure (one entry per
enabled fetcher, in config order), so partial failures are visible without
diffing domains against `posts`. A source can partially fail, e.g. one dead
feed among many: it keeps its posts, records the error, and reports `ok: false`.

```json
{
  "count": 2,
  "posts": [ { "id": "...", "source": "rss", "title": "...", "url": "...", "created_at": "..." } ],
  "sources": [
    { "source": "rss", "name": null, "ok": false, "posts": 2,
      "errors": ["Failed to fetch feed https://dead.example/feed.xml: ..."] }
  ],
  "errors": [ { "source": "rss", "error": "Failed to fetch feed https://dead.example/feed.xml: ..." } ]
}
```

Exit codes: `0` success · `1` a source failed (only with `--fail-on-error`) ·
`2` configuration error.

### 4. Programmatic (no YAML)

```python
from datetime import datetime, timezone, timedelta
from fetchkit import FetchKitConfig, HackerNewsFetchConfig, PostFetchConfig, SortOrder, collect_all

config = FetchKitConfig(
    start_time=datetime.now(timezone.utc) - timedelta(days=1),
    end_time=datetime.now(timezone.utc),
    fetchers=[
        HackerNewsFetchConfig(posts=PostFetchConfig(max_items=30, order=SortOrder.NEW)),
    ],
)
result = collect_all(config)
```

## Discovering RSS feeds

RSS is the hardest source to discover for: a feed is an *arbitrary URL*, so an
agent that wants "central-bank policy" or "rust programming" has no way to know
which feeds exist. The optional `discovery` module closes that gap — give it a
use case, get back ranked feeds you can drop straight into an `rss` fetcher.
(The other sources have their own, simpler discovery surface — see
[Discoverability for every source](#discoverability-for-every-source) below.)

```python
from fetchkit.discovery import discover, to_rss_config

matches = discover("news and topics regarding AI safety research", top_k=5)
for m in matches:
    print(m.score, m.name, m.url)

config = to_rss_config(matches)   # an RSSFetchConfig, ready for collect_all
```

```bash
fetchkit discover "AI safety research news" --top-k 5            # ranked feeds as JSON
fetchkit discover "central bank policy" --as-config -o rss.json  # emit a runnable config…
fetchkit run rss.json                                            # …then fetch (discover → config → run)
fetchkit find-feeds https://example.com                          # autodiscover a site's feeds
```

### How it works (and what it can't do)

Feeds are **matched, not searched**. You can't embed feeds you've never seen, so
embeddings only ever *rank* a set of candidates you already hold. Discovery
assembles that candidate set from three sources, then ranks it against your query:

1. **Curated catalog** — a shipped, versioned directory of ~50 high-quality,
   long-lived feeds across news, research, programming, finance, and more. The
   offline, deterministic floor.
2. **Autodiscovery** (`find_feeds(url)` / `--from-urls`) — given a site, it reads
   the `<link rel="alternate">` RSS-autodiscovery tags, probes common feed paths,
   and validates each candidate. This reaches the open-web long tail. The
   *topic → which sites* step is left to **your** web search: an agent already has
   one, so it finds the sites and hands them to fetchkit, which extracts the feeds.
   fetchkit never bundles a search engine.
3. **External index** (`--external`, opt-in) — queries a third-party feed-search
   service for recall over millions of feeds. Off by default (network + ToS).

Ranking has two backends behind one interface:

- **lexical** (default) — pure-Python BM25 over each feed's metadata. Zero extra
  dependencies, fully deterministic, works offline and in CI.
- **embedding** — a local sentence-transformers model for stronger semantic
  matching. Opt in with the extra:

  ```bash
  pip install "fetchkit-agents[discovery-embeddings]"
  fetchkit discover "papers on diffusion models" --backend embedding
  ```

  `--backend auto` (the default) uses the embedding ranker when the extra is
  installed and falls back to lexical otherwise.

Discovery matches a feed's **description/metadata** (what the feed is *about*),
not the live article stream — for the latter, fetch the feed with the `rss`
fetcher and match individual posts. Catalog quality is the main lever on result
quality; see `src/fetchkit/discovery/data/CATALOG.md` to extend it.

Because discovery only *ranks* candidates it already holds, a niche query (a
specific region, product, or community) can come back filled with generic feeds
that matched only the query's broad terms. The CLI flags this instead of letting
it pass: the JSON payload carries `unmatched_terms` (query terms no returned
feed mentions) plus a `hint`, and the same hint is printed to stderr. When you
see it, feed discovery better candidates — `--from-urls` with sites from your
own web search, `fetchkit find-feeds <site-url>`, or `--external`.

## Discoverability for every source

`discover` solves RSS's "which feed URL?" problem. The other sources have a
*different* discoverability gap — *which tag / site / instance / feed do I put in
the config?* — answered by `fetchkit suggest <source>` (and `run_suggester()` in
Python). Each suggester is **no-auth** and returns JSON-ready rows:

```bash
fetchkit suggest lobsters                                  # all Lobsters tags
fetchkit suggest stackexchange --site stackoverflow        # popular SO tags
fetchkit suggest stackexchange --what sites                # available SE sites
fetchkit suggest arxiv --query vision                      # matching arXiv categories
fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to watch
fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
fetchkit suggest bluesky                                   # popular custom feeds
fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
fetchkit suggest edgar --query apple                       # tickers/CIKs for an edgar config
fetchkit suggest reddit --query finance                    # subreddits for a topic
fetchkit suggest gdelt --query country                     # GDELT languages/countries/operators
fetchkit suggest googlenews                                # Google News topics + query operators
fetchkit suggest polymarket --query fed                    # most active prediction markets
fetchkit suggest manifold --query "rate cut"               # Manifold markets for a topic
fetchkit suggest rss --query "AI safety news"              # delegates to discover()
```

| Source | `suggest` returns | Fills config field |
|--------|-------------------|--------------------|
| `hackernews`    | sort orders (static — HN has no tags)         | `posts.order` |
| `rss`           | ranked feeds (delegates to `discover`)        | `feeds` |
| `arxiv`         | category codes + names (static taxonomy)      | `categories` |
| `github`        | popular `owner/name` repos for a query        | `repos` |
| `lobsters`      | all tags (from `/tags.json`)                  | `tag` |
| `stackexchange` | popular tags, or sites (`--what sites`)       | `tagged` / `site` |
| `bluesky`       | popular feeds, or actors (`--what actors`)    | `actor` |
| `mastodon`      | trending hashtags on an instance              | `tag` / `instance` |
| `edgar`         | company tickers/CIKs matching a query         | `companies` |
| `reddit`        | subreddits matching a query                   | `subreddits` |
| `gdelt`         | languages, country codes, query operators (static) | `languages` / `countries` / `query` |
| `googlenews`    | topic sections + query operators (static)     | `topic` / `query` |
| `polymarket`    | most active live markets (by 24h volume)      | `query` |
| `manifold`      | live markets matching a query                 | `query` |

> Most suggesters call the source's live API (all no-auth); `hackernews`/`arxiv`
> are static, and `rss` reuses the offline catalog ranker. Network suggesters fail
> gracefully (the CLI reports the error on stderr and exits non-zero).

## Configuration reference

### Top-level (`FetchKitConfig`)

| Field         | Type                | Required | Description                                |
|---------------|---------------------|----------|--------------------------------------------|
| `window`      | str                 | no       | Relative window (resolves to start/end).   |
| `start_time`  | datetime            | no       | Global window start (inclusive).           |
| `end_time`    | datetime            | no       | Global window end (inclusive).             |
| `fetchers`    | list[FetcherConfig] | no       | Fetcher instances to run.                  |
| `http`        | HttpConfig          | no       | Shared HTTP client settings.               |
| `dedup`       | str                 | no       | `exact` (default) or `url` — see below.    |

Set the time window with **either** a relative `window` **or** both
`start_time`/`end_time` — the two are mutually exclusive. **If you specify none of
them, the window defaults to the last 24 hours.** `start_time <= end_time` is
enforced, and unknown top-level keys are rejected. Per-fetcher
`start_time`/`end_time` default to `None` and **inherit** the run window from these
top-level keys at runtime (there is no `global:` key — the top level *is* the
global scope).

#### Deduplication (`dedup`)

Posts are always deduplicated by `(source, id)`. With `dedup: url` the collector
*additionally* drops posts whose normalized article URL was already seen —
tracking parameters (`utm_*`, `fbclid`, …), fragments, and trailing slashes are
stripped before comparing — which collapses the same story fetched via
different sources (e.g. a publisher's RSS feed and GDELT). First occurrence in
config order wins, so put the richest source first. Note that Google News links
are `news.google.com` redirect URLs, so they never URL-match the publisher's
own links.

#### Authenticated fetcher types and secrets

The base fetchers are strictly no-auth. Sources whose anonymous endpoints are
commonly blocked from datacenter/cloud IPs have a separate authenticated
variant — `reddit-auth` and `bluesky-auth` — whose credential fields *default
to environment-variable references*, so the minimal config is just the type:

```yaml
fetchers:
  - type: reddit-auth            # reads ${REDDIT_CLIENT_ID} / ${REDDIT_CLIENT_SECRET}
    subreddits: ["news"]
  - type: bluesky-auth           # reads ${BLUESKY_IDENTIFIER} / ${BLUESKY_APP_PASSWORD}
    query: "cyprus"
```

Validation fails immediately (exit 2, naming the missing variable) when the
environment isn't set — auth problems surface at load time, never as a
mid-run 401. Any credential field also accepts a literal value or an explicit
`${OTHER_VAR}` reference (`mastodon.access_token` works the same way inline).

#### Relative windows

`window` accepts (case-insensitive): `"last 6 hours"`, `"past 30 minutes"`,
`"last 7 days"`, `"last week"`, `"last month"`, `"today"`, `"yesterday"`, or a bare
duration like `"6h"`, `"2d"`, `"90m"`. It resolves **once** to a concrete
`(start_time, end_time)` pair (end = now), so collection stays deterministic for that
resolved pair. The same parsing is available programmatically via
`fetchkit.resolve_window(spec)` and `fetchkit.parse_duration(text)`.

### Hacker News (`type: hackernews`)

| Field                | Default | Notes                                  |
|----------------------|---------|----------------------------------------|
| `query`              | null    | Full-text topic filter (Algolia search) — e.g. `"LLM"` |
| `min_score`          | null    | Only stories with at least this many points |
| `posts.max_items`    | 10      | 1–500                                  |
| `posts.order`        | top     | top / new / controversial / asc / desc |
| `comments.fetch`     | false   | Fetch comment threads for each post    |
| `comments.max_items` | 10      | 1–100 roots per post                   |
| `comments.max_depth` | 1       | 0 = roots only                         |
| `comments.order`     | top     | Sort order for comment roots           |

### RSS / Atom (`type: rss`)

| Field                  | Default | Notes                          |
|------------------------|---------|--------------------------------|
| `feeds`                | —       | List of `{url, name?, headers?}` |
| `max_items_per_feed`   | 50      | 1–500                          |
| `max_total_items`      | 200     | 1–2000                         |
| `include_content`      | true    | Include full entry content     |
| `allow_local_files`    | false   | Permit local-path / `file://` feeds |

`feeds[].headers` sends extra HTTP headers with that feed's request — typically
a custom `User-Agent` for sites that block non-browser clients:

```yaml
feeds:
  - url: "https://www.havadiskibris.com/rss"
    headers: { User-Agent: "Mozilla/5.0 (compatible; my-pipeline)" }
```

`feeds[].url` is restricted to HTTP(S) URLs by default, and each URL's host must
resolve to a **public** address — feeds pointing at loopback, private (RFC-1918),
link-local, or cloud-metadata (`169.254.169.254`) hosts are refused to prevent
SSRF. Setting `allow_local_files: true` additionally permits local file paths and
`file://` URLs *and* opts out of the SSRF guard (it declares the config trusted) —
see the security note below before enabling it.

> **⚠️ Security — untrusted configs.** fetchkit is built to run YAML that may be
> produced by an LLM/agent. By default it blocks two abuse vectors in RSS feed
> URLs: local file reads (e.g. `file:///etc/passwd`) and SSRF to internal
> addresses (e.g. `http://169.254.169.254/...` or `http://127.0.0.1`). Both
> protections are **on by default**. `allow_local_files: true` turns *both* off, so
> enable it only for configs and fixtures you trust. The SSRF guard is reasonable,
> not perfect — it checks the host at request time and does not defend against DNS
> rebinding or redirects to private hosts.

### arXiv (`type: arxiv`)

Uses the arXiv export API (Atom, parsed with feedparser). The full abstract is
in `post.text`; authors, categories, DOI, PDF link, and primary category are
preserved in `post.metadata`. `post.url` is the human-readable abstract page
by default (`link_format: pdf` for direct PDFs — the PDF link is always in
`metadata.pdf_url` either way).

| Field         | Default  | Notes                                            |
|---------------|----------|--------------------------------------------------|
| `categories`  | `[]`     | arXiv categories, e.g. `["cs.AI", "cs.LG"]` (empty = all) |
| `query`       | null     | Free-text query (combined with categories via AND) |
| `link_format` | abstract | What `url` points at: `abstract` page or direct `pdf` |
| `max_items`   | 50       | 1–500                                            |

### GitHub (`type: github`)

Public GitHub REST API (no auth; anonymous quota is 60 requests/hour). Repo,
language, stars, forks, topics, description, `pushed_at`, and `updated_at` are
preserved in `post.metadata`.

The `query` string is passed **verbatim** to the GitHub Search API, so the
[full repository-search syntax](https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories)
works: `language:python topic:llm stars:>100`. For an "active this week"
digest use `sort: updated` + `window_qualifier: pushed` — the default
(`created`) windows on repo *creation* date, which can surface repos untouched
since day one.

| Field              | Default  | Notes                                                        |
|--------------------|----------|--------------------------------------------------------------|
| `resource`         | releases | `releases` (per-repo) or `search_repos`                      |
| `repos`            | `[]`     | `owner/name` list — required for `resource: releases`        |
| `query`            | null     | Search query — required for `resource: search_repos`; passed verbatim |
| `sort`             | stars    | `stars`, `updated` (recent pushes first), or `best_match`    |
| `window_qualifier` | created  | Map the run window to a `created:`/`pushed:` search qualifier, or `none` |
| `include_readme`   | false    | Fetch each repo's README into `post.text` (+1 request/repo)  |
| `readme_max_chars` | 2000     | 100–20000; READMEs are truncated to this length              |
| `max_items`        | 50       | 1–300                                                        |

### Lobsters (`type: lobsters`)

The lobste.rs public JSON endpoints (no auth). Tags are preserved in `post.metadata`.

| Field      | Default | Notes                                           |
|------------|---------|-------------------------------------------------|
| `listing`  | hottest | `hottest` or `newest`                           |
| `tag`      | null    | Restrict to a single tag (uses `/t/<tag>.json`) |
| `max_items`| 50      | 1–200                                           |

### Stack Exchange (`type: stackexchange`)

The Stack Exchange API (no auth; anonymous access is capped at 300 requests/day/IP).
Questions become `Post`s; with `comments.fetch: true`, top answers are attached as
`Comment`s. Tags, answer count, and the accepted-answer id are kept in `post.metadata`.

| Field      | Default        | Notes                                                       |
|------------|----------------|-------------------------------------------------------------|
| `site`     | stackoverflow  | API site, e.g. `serverfault`, `superuser`, `askubuntu`      |
| `tagged`   | `[]`           | Questions carrying ALL of these tags (joined with `;`)      |
| `query`    | null           | Free-text search (uses `/search/advanced`)                  |
| `posts`    | —              | `max_items` (1–500) and `order` (`top`→votes, `new`→creation) |
| `comments` | —              | `fetch`/`max_items` — answers attached as comments          |

### Bluesky (`type: bluesky`, `type: bluesky-auth`)

`bluesky` is the public AppView (`public.api.bsky.app`), unauthenticated by
design — but it blocks some datacenter/cloud IP ranges (403). `bluesky-auth`
takes the same `resource`/`query`/`actor` fields and calls the same XRPC
endpoints, but through a session created on `service` (default `bsky.social`)
with an [app password](https://bsky.app/settings/app-passwords) — never the
account password. (Bluesky is migrating to full AT Protocol OAuth and calls
app passwords legacy, but they remain the supported path for headless
scripts.) `uri`, `cid`, and `langs` are kept in `post.metadata`; likes map to
`score` and replies to `comment_count`.

| Field          | Default                    | Notes                                             |
|----------------|----------------------------|---------------------------------------------------|
| `resource`     | search                     | `search` (full-text) or `author_feed`             |
| `query`        | null                       | Search query — required for `resource: search`    |
| `actor`        | null                       | Handle/DID — required for `resource: author_feed` |
| `max_items`    | 50                         | 1–500 (paginated, 100/page)                       |
| `identifier`   | `${BLUESKY_IDENTIFIER}`    | *auth only* — handle or email                     |
| `app_password` | `${BLUESKY_APP_PASSWORD}`  | *auth only* — app password, not account password  |
| `service`      | https://bsky.social        | *auth only* — PDS/entryway for the session        |

### Mastodon (`type: mastodon`)

Public and hashtag timelines on any instance. Anonymous access works while the
instance keeps public preview enabled, but it is the first to be rate-limited
(429 under load) or disabled (401). Setting `access_token` sends authenticated
requests, which lifts both restrictions — create a token on the instance under
Settings → Development → New application (read scope is enough). HTML content
is reduced to plain text; tags, instance, and visibility are kept in
`post.metadata`.

| Field          | Default          | Notes                                                  |
|----------------|------------------|--------------------------------------------------------|
| `instance`     | mastodon.social  | Instance host without scheme, e.g. `fosstodon.org`     |
| `resource`     | tag              | `tag` (`/timelines/tag/<tag>`) or `public`             |
| `tag`          | null             | Hashtag without `#` — required for `resource: tag`     |
| `local`        | false            | Restrict to statuses originating on this instance      |
| `access_token` | null             | OAuth token sent as Bearer (`${ENV}` ok)               |
| `max_items`    | 50               | 1–200 (paginated, 40/page)                             |

### SEC EDGAR (`type: edgar`)

Securities filings from the SEC's free, keyless EDGAR APIs. `resource: filings`
pulls recent filings per company from `data.sec.gov/submissions` (acceptance
timestamps are near-real-time — new 8-Ks appear within minutes); `resource: search`
runs EDGAR full-text search across all filers (date granularity only; the run
window is passed to the API as a date range). Form type, CIK, accession number,
and 8-K item codes are kept in `post.metadata`.

The SEC's [fair-access policy](https://www.sec.gov/os/accessing-edgar-data) asks
clients to identify themselves with contact info and stay under 10 req/s — set
`user_agent` and consider `http.rate_limit_per_host`.

| Field        | Default  | Notes                                                        |
|--------------|----------|--------------------------------------------------------------|
| `resource`   | filings  | `filings` (per-company recent) or `search` (full-text)       |
| `companies`  | `[]`     | Tickers or CIKs — required for `resource: filings`           |
| `forms`      | `[]`     | Form types to include, e.g. `["8-K", "10-Q"]` (empty = all)  |
| `query`      | null     | Full-text query — required for `resource: search`            |
| `user_agent` | null     | Descriptive UA with contact info, e.g. `myapp me@example.com`|
| `max_items`  | 50       | 1–500                                                        |

### Reddit (`type: reddit`, `type: reddit-auth`)

Subreddit listings or search. `listing: new` surfaces posts seconds after
submission (engagement is near zero at that point by definition); `hot`/`top`
carry engagement at the cost of recency. Score, comment count, and upvote
ratio are captured as of fetch time. Subreddit, flair, and upvote ratio are
kept in `post.metadata`.

- **`reddit`** — anonymous, via the public `.json` endpoints. Tolerated at low
  volume (~10 requests/minute per IP) but formally a gray area of Reddit's
  Data API terms, and Reddit blocks many datacenter/cloud IPs outright (403
  regardless of User-Agent) — expect this to fail from CI or a cloud host.
- **`reddit-auth`** — same fields plus OAuth through `oauth.reddit.com` with a
  token from a registered app
  ([reddit.com/prefs/apps](https://www.reddit.com/prefs/apps)) — the
  sanctioned path, and the one that works from cloud infrastructure. Without
  `username`/`password` an application-only token is used (fine for reading
  public listings); set both to use the password grant of a *script*-type app
  (2FA accounts append the code as `password:123456`). Reddit asks OAuth
  clients for a descriptive `user_agent` like `linux:my-pipeline:v1 (by /u/you)`.

| Field           | Default                    | Notes                                                  |
|-----------------|----------------------------|--------------------------------------------------------|
| `subreddits`    | `[]`                       | Without `r/` prefix; combined into one listing request |
| `listing`       | new                        | `new`, `hot`, `top`, or `rising` (when `query` unset)  |
| `query`         | null                       | Search; restricted to `subreddits` when both are set   |
| `time_filter`   | day                        | For `listing: top` and searches: `hour`…`all`          |
| `user_agent`    | null                       | Custom UA; Reddit wants a descriptive one              |
| `max_items`     | 50                         | 1–100 (single request)                                 |
| `client_id`     | `${REDDIT_CLIENT_ID}`      | *auth only* — registered app client id                 |
| `client_secret` | `${REDDIT_CLIENT_SECRET}`  | *auth only* — registered app secret                    |
| `username`      | null                       | *auth only* — script-app password grant                |
| `password`      | null                       | *auth only* — set together with `username`             |

### GDELT (`type: gdelt`)

The GDELT Project's DOC 2.0 API — machine-reads worldwide news media in 65
languages, re-indexed every 15 minutes. A zero-auth global news firehose. Hits
carry URL, title, timestamp, source domain/country/language (kept in
`post.metadata`); there is no author, body text, or engagement, so those fields
are `None`. The run window maps to the API's `startdatetime`/`enddatetime`.

| Field       | Default | Notes                                                          |
|-------------|---------|----------------------------------------------------------------|
| `query`     | —       | Required; keywords/"phrases" plus operators (`domain:`, `tone<-5`) |
| `languages` | `[]`    | Source languages, e.g. `["english", "spanish"]`                |
| `countries` | `[]`    | Source countries (GDELT/FIPS codes), e.g. `["US", "UK"]`       |
| `max_items` | 75      | 1–250                                                          |

### Google News (`type: googlenews`)

Google News' public RSS endpoints: top stories, search, or a topic section.
Minutes-fresh headlines aggregated from thousands of publishers. The query
language supports `when:2h` (recency), `site:`, quoted phrases, `-exclusions`,
and `intitle:`. Links are Google News redirect URLs (they resolve to the
publisher); the publisher name maps to `author` and a headline snippet to
`text`.

> **Boolean queries:** Google News ANDs space-separated terms, and long
> `X OR Y OR Z` chains are parsed unreliably — results silently degrade into
> generic headlines (fetchkit warns on stderr when it spots the pattern). To
> OR several topics, use `queries`: each entry is fetched as its own RSS
> request and the results are merged and deduplicated. `queries` ×
> `required_domains` is capped at 8 requests per fetcher; validation rejects
> configs beyond that.

| Field              | Default | Notes                                                          |
|--------------------|---------|----------------------------------------------------------------|
| `resource`         | top     | `top`, `search` (needs `query`/`queries`), or `topic` (needs `topic`) |
| `query`            | null    | e.g. `'nvidia earnings when:2h'`                               |
| `queries`          | `[]`    | Multiple searches merged + deduped — the reliable OR (search only) |
| `required_domains` | `[]`    | Only these publishers; becomes `site:` variants per query (search only) |
| `exclude_domains`  | `[]`    | Drop these publishers: `-site:` terms + client-side filter     |
| `topic`            | null    | `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE`, `HEALTH` |
| `language`         | en-US   | UI language (`hl=`)                                            |
| `country`          | US      | Edition (`gl=`/`ceid=`)                                        |
| `max_items`        | 50      | 1–100                                                          |

### Polymarket (`type: polymarket`) and Manifold (`type: manifold`)

Prediction markets as posts: the market question is the `title`, and live odds,
volume, and close date land in `post.metadata` (`odds`/`probability`,
`volume_24h`, …). Odds moving is often a signal about breaking news before
articles are written. Both APIs are public read-only, no auth.

> **Snapshot semantics:** markets are live state, not timestamped events, so
> these two fetchers deliberately ignore the run window — an active market
> created months ago is still current. `created_at` is the market's creation
> time, which also means snapshot posts sort below newer article-shaped posts
> in a mixed feed.

| Field (`polymarket`) | Default | Notes                                            |
|----------------------|---------|--------------------------------------------------|
| `query`              | null    | Substring filter on the question, applied client-side over the top ~100 markets for the chosen `order` (not a full search) |
| `order`              | volume  | `volume` (24h), `liquidity`, or `new`            |
| `closed`             | false   | Include closed/resolved markets                  |
| `max_items`          | 50      | 1–100                                            |

| Field (`manifold`) | Default | Notes                                              |
|--------------------|---------|----------------------------------------------------|
| `query`            | null    | Full-text search; omit for most recent markets     |
| `market_filter`    | open    | `open`, `closed`, `resolved`, `all` (searches)     |
| `sort`             | score   | `score`, `newest`, `liquidity`, `24-hour-vol`      |
| `max_items`        | 50      | 1–100                                              |

### HTTP (`http:`)

| Field                 | Default                  | Notes                                              |
|-----------------------|--------------------------|----------------------------------------------------|
| `timeout`             | 10.0                     | Per-request timeout in seconds                     |
| `user_agent`          | null                     | Your project's identity — see below                |
| `max_retries`         | 3                        | Retries on transient errors (0–10)                 |
| `backoff_factor`      | 0.5                      | `wait = backoff_factor * 2^attempt` seconds        |
| `rate_limit_per_host` | null                     | Max requests/sec per host (null = disabled)        |
| `retry_statuses`      | 429,500,502,503,504      | Status codes that trigger a retry                  |

Timeouts and connection errors are retried `max_retries` times with exponential
backoff (429/503 honor `Retry-After`), and errors that exhaust their retries say
so (`... (after 4 attempts with backoff)`). If a host throttles parallel runs
(e.g. GDELT from one IP), set `rate_limit_per_host` to smooth requests out
instead of racing them.

#### Identify your project (User-Agent)

fetchkit is a library — traffic it generates belongs to *your* project, so
**set a User-Agent that describes your use case**, ideally with contact info.
Site operators use it to attribute (and tolerate) automated traffic; anonymous
generic clients are the first thing they block. Two equivalent ways:

```yaml
http:
  user_agent: "cyprus-news-pipeline/1.0 (ops@example.com)"
```

```bash
export FETCHKIT_USER_AGENT="cyprus-news-pipeline/1.0 (ops@example.com)"   # no config change needed
```

The YAML field wins over the environment variable. Your identity is sent
first, with the library's product token appended per HTTP convention:
`cyprus-news-pipeline/1.0 (ops@example.com) fetchkit-agents/0.4.0`. If neither
is set, a generic `fetchkit-agents (+https://pypi.org/project/fetchkit-agents/)`
identifier is used — fine for experiments, too anonymous for anything running
on a schedule. Escape hatches that send exactly what you write (no token
appended): per-feed rss `headers` and the per-fetcher `user_agent` fields on
`reddit` and `edgar`.

## Adding a fetcher

Fetchers live in the library itself, so each one ships with a typed config,
validation, and tests. Add a new source via a PR or a local fork in four steps:

1. Add a typed config to `src/fetchkit/schemas/fetcher.py` (subclass `FetcherBase`,
   give it a `Literal` `type`), and register it in `_BUILTIN_TYPES` + `FetcherConfig`.
2. Write the fetcher module in `src/fetchkit/fetchers/`, returning a `FetcherResult`:

   ```python
   from fetchkit.fetchers.base import FetcherResult
   from fetchkit.fetchers.registry import register_fetcher
   from fetchkit.schemas.post import Post

   @register_fetcher("mysource")
   def fetch(config) -> FetcherResult:
       posts = [Post(id="1", source="mysource", title="…", source_url="https://…")]
       return FetcherResult(posts=posts, errors=[])
   ```

3. Import the module in `src/fetchkit/fetchers/__init__.py` so it registers on import.
4. Add tests (mock HTTP with the `responses` library — see `tests/fetchers/`).
5. *(Optional)* register a discovery helper so agents can find your source's knobs:

   ```python
   from fetchkit.fetchers.suggest_registry import register_suggester

   @register_suggester("mysource")
   def suggest(*, query=None, limit=20, **kwargs) -> list[dict]:
       return [{"tag": "…"}]   # JSON-ready rows; surfaced via `fetchkit suggest mysource`
   ```

Use `Post.metadata` for any source-specific fields that don't map to the canonical
columns.

## The `Post` model

```python
class Post(BaseModel):
    id: str                       # unique within source
    source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
                                  #   | "stackexchange" | "bluesky" | "mastodon" | "edgar"
                                  #   | "reddit" | "gdelt" | "googlenews" | "polymarket" | "manifold"
    title: str | None
    text: str | None              # body / content
    url: str | None               # external link
    author: str | None
    score: int | None             # source-relative; NOT comparable across sources
    comment_count: int | None
    created_at: datetime | None   # UTC-aware
    source_url: str               # direct link on the source platform
    comments: list[Comment]       # nested threads (HN)
    metadata: dict[str, Any]      # source-specific extras (categories, stars, tags, …)
```

All datetimes are normalized to UTC. Posts are deduplicated by `(source, id)` and
sorted descending by `(created_at, id)` for deterministic output.

### The `text` contract

`text` carries the **best available body text at the source**, so a pipeline
can summarize/keyword-extract every post without a second HTTP request and
without special-casing fetchers. It is null only when the source genuinely
has no body — never because fetchkit skipped it.

| Source | `text` contains |
|--------|-----------------|
| `arxiv` | The full abstract |
| `rss` | Entry content or summary (feed-dependent) |
| `hackernews` | Self-post body (Ask/Show HN); link posts have no body → null |
| `github` | Repo description — or the README with `include_readme: true`; release notes for `releases` |
| `lobsters` | Story description (often empty for pure links) |
| `stackexchange` | Question body (answers attach as comments) |
| `reddit` | Selftext (link posts → null) |
| `bluesky` / `mastodon` | Post text (HTML stripped) |
| `googlenews` | Headline snippet from the RSS description (HTML stripped) |
| `edgar` | Filing description |
| `polymarket` / `manifold` | Market description when present |
| `gdelt` | Always null — GDELT indexes headlines only |

### Window semantics per source

The run window filters on `created_at`, but each source's clock means
something different — a "past week" window is repo-creation time on GitHub
(`window_qualifier: pushed` switches it to push activity), submission time on
HN/Lobsters, publication time on news sources, and is deliberately ignored by
the prediction-market fetchers (live state). Run `fetchkit guide` for the
full per-source table of what `created_at` timestamps and whether the window
is applied server-side or client-side.

> **`score` is source-relative.** Each source defines it differently — Hacker News
> points, Lobsters score, GitHub stars (for `search_repos`), Stack Exchange question
> score, Bluesky likes, Mastodon favourites, Reddit score (fuzzed by Reddit, and
> near zero on brand-new posts), and `None` for arXiv, RSS, Google News, GDELT,
> EDGAR, and the prediction-market sources. The values
> are **not comparable across sources**, so don't rank a mixed
> feed by `score` directly. Compare within a single `source`, or use a source-aware
> ranking of your own. Output is ordered by recency (`created_at`), not by `score`.

## Collector invariants

`collect_all` preserves three guarantees:

1. **Window inheritance** — per-fetcher `start_time`/`end_time` fall back to the
   global window when omitted.
2. **Dedup** — by `(source, id)`; first occurrence wins.
3. **Sort** — descending by `(created_at or UTC_MIN, id)`.

Partial failures are aggregated, not fatal: a failing source yields an entry in
`result.errors` while other sources still collect. Check `result.has_errors`.

## Development

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# tests (skip live/networked)
pytest -m "not live"

# live network smoke (opt-in)
pytest -m live

# strict typecheck
mypy src
```

## License

MIT — see [LICENSE](LICENSE). MIT is intentional: as a small, dependency-light
utility meant to be embedded freely in agent pipelines (including commercial and
closed-source ones), a permissive license maximizes adoption with no copyleft
obligations. A weak-copyleft license (e.g. MPL-2.0) or strong copyleft (GPL/AGPL)
would force redistribution terms on downstream users and discourage exactly the
embedded use fetchkit is built for.
