Metadata-Version: 2.3
Name: mediaq
Version: 0.1.0
Summary: Hybrid CLI + REST tool that searches and scrapes media content from multiple sites via their native search.
Author: Mahamudul Hasan
Author-email: Mahamudul Hasan <919.hasan@gmail.com>
Requires-Dist: typer>=0.15
Requires-Dist: rich>=13.9
Requires-Dist: fastapi>=0.115
Requires-Dist: uvicorn[standard]>=0.34
Requires-Dist: pydantic>=2.10
Requires-Dist: pydantic-settings>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: wreq>=0.12
Requires-Dist: selectolax>=0.3.28
Requires-Dist: aiosqlite>=0.20
Requires-Dist: structlog>=24.4
Requires-Dist: aiofiles>=25.1.0
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# mediaq

Hybrid **CLI + REST API** that searches media sites through their *native* search, sort and
filter capabilities and resolves results into direct media URLs — optionally downloading them.
Both halves share one core: the CLI runs in-process by default, can manage the server
(`mediaq server start|stop|status`), and can drive a running server with `--remote`.

Stack: Python 3.12 · [uv] · Typer (CLI) · FastAPI/uvicorn (REST) · [wreq] (browser-emulated
HTTP) · selectolax/Lexbor (HTML) · Pydantic v2 · SQLite (jobs & results) · YAML config.

> Sites covered are adult sites. Respect their terms of use and rate limits; defaults are
> deliberately conservative (1 request/s, 2 concurrent per host).

## Install

```bash
uv sync                      # development
uv run mediaq --help

uv tool install .            # or: install the `mediaq` command globally
```

## Quick start

```bash
mediaq sites list                                   # capability matrix
mediaq sites test xvideos -q blonde                 # live smoke test of one adapter
mediaq --json sites list                            # global --json: JSON output everywhere
mediaq search "blonde" -s xvideos -s pornhub --sort newest --quality hd
mediaq search "blonde" --all --pages 2 -f jsonl -o results.jsonl --save
mediaq fetch --from results.jsonl                   # resolve direct media URLs
mediaq fetch "https://www.xvideos.com/video.abc/x" --download --dest ./dl --quality 720p
mediaq jobs list && mediaq jobs results <job-id> -f csv
```

Search options map onto each site's own parameters (`--sort`, `--quality`, `--period`,
`--orientation`, `--min-duration`, `--max-duration`); anything a site cannot honour is
dropped with a warning (`--strict` fails instead). Raw native parameters pass through with
`--extra key=value`, and native query syntax (booru tags, nhentai filters, `site:` on NudeVista)
goes straight into the query string.

`fetch` accepts page URLs, `site:id` / `site:kind:id` refs, `--from` a search output file, or
`--job` a saved search. Aggregator results (PornMD, TubeGalore, iXXX, NudeVista) are resolved
through the adapter of the site they point at.

## Server

```bash
mediaq server start                 # detached uvicorn on 127.0.0.1:8765, pidfile + log file
mediaq server status | logs -f | restart | stop
mediaq server start --foreground    # run in the terminal

curl -s localhost:8765/health
curl -s -X POST localhost:8765/jobs/search -H 'content-type: application/json' \
     -d '{"query":"blonde","sites":["nhentai","rule34"],"pages":1}'
mediaq --remote http://127.0.0.1:8765 jobs list
```

Endpoints: `GET /health`, `GET /sites[/{id}]`, `POST /search` (sync, ≤5 pages), `POST /fetch`,
`POST /jobs/search|fetch|download` (async), `GET /jobs[/{id}[/results|/downloads]]`,
`POST /jobs/{id}/cancel`, `DELETE /jobs/{id}`, `POST /jobs/purge`, `GET /downloads/{id}`,
`POST /admin/shutdown` (loopback-only; bearer token when `server.admin_token` is set).
OpenAPI docs are served at `/docs`.

## Configuration

`~/.config/mediaq/config.yaml` (or `./mediaq.yaml`, `--config PATH`, `MEDIAQ_CONFIG`).
Environment variables override the file: `MEDIAQ_SERVER__PORT=9000`. See
[`config/mediaq.example.yaml`](config/mediaq.example.yaml) for every key, and use
`mediaq config init|show|set` to manage it.

Per-site settings live under `sites.<id>`: `enabled`, `cookies` (e.g. `cf_clearance`),
`user_agent`, `api_key`/`user_id` (Rule34, Gelbooru, nhentai).

## Sites

Generated by `uv run python scripts/site_matrix.py`.

| id | site | kinds | default | sorts | filters | notes |
|---|---|---|---|---|---|---|
| `pornhub` | Pornhub | gallery, gif, video | on | duration, newest, rating, views | period, min-duration, max-duration |  |
| `xvideos` | XVideos | video | on | duration, newest, random, rating, views | quality, period, min-duration, max-duration |  |
| `xnxx` | XNXX | video | on | random, views | quality, period, min-duration, max-duration |  |
| `xhamster` | xHamster | gallery, video | on | duration, newest, rating, views | quality, period, orientation, min-duration, max-duration |  |
| `youporn` | YouPorn | video | on | - | - |  |
| `redtube` | RedTube | video | on | duration, rating, views | period |  |
| `spankbang` | SpankBang | video | on | newest, views | quality, period, min-duration |  |
| `eporner` | Eporner | video | on | duration, newest, rating, views | quality, period, orientation |  |
| `noodlemagazine` | NoodleMagazine | video | on | duration, newest | quality, min-duration, max-duration |  |
| `hqporner` | HQPorner | video | on | - | - |  |
| `redgifs` | RedGIFs | gif, image | on | newest, rating, views | period |  |
| `pornmd` | PornMD | video | on | - | - | aggregator (resolves via source site) |
| `tubegalore` | TubeGalore | video | on | - | - | aggregator (resolves via source site) |
| `ixxx` | iXXX | video | on | - | - | aggregator (resolves via source site) |
| `nudevista` | NudeVista | gallery, video | on | newest, random, views | - | aggregator (resolves via source site); native query syntax: `site:<domain>` restricts to one source site |
| `imagefap` | ImageFap | gallery | on | newest | - |  |
| `pornpics` | PornPics | gallery | on | - | - |  |
| `sexcom` | Sex.com | gif, image, video | on | - | orientation |  |
| `rule34` | Rule34.xxx | gif, image, video | on | newest, random, rating | - | native query syntax: space-separated tags; `-tag` excludes; `rating:explicit|questionable|general`; `score:>100`; `sort:score|id|random`; `video`/`animated` for media types |
| `gelbooru` | Gelbooru | gif, image, video | on | newest, random, rating | - | native query syntax: space-separated tags; `-tag` excludes; `rating:explicit|questionable|general`; `score:>100`; `sort:score|id|random`; `video`/`animated` for media types |
| `nhentai` | nhentai | gallery | on | newest, rating, views | period | native query syntax: `"exact phrase"`, `-word`, `tag:"big breasts"`, `artist:x`, `language:english`, `pages:>10`, `favorites:>=100`, `uploaded:<7d` |

Notes on what could be verified during development:

- Pornhub serves HLS renditions to guests; progressive MP4 (`get_media`) is usually empty.
  GIFs and album photos resolve to direct files.
- xHamster and HQPorner sign progressive URLs against the client IP; downloads fail with
  `403 Wrong key` / `404` from networks whose egress IP differs per destination. HLS masters
  work. HLS-only sources are recorded but not downloaded in v1.
- XVideos/XNXX offer 240p/360p MP4 directly; higher qualities are HLS.
- Sex.com videos resolve to an HLS master plus a 240p MP4 preview (the page shows a regional
  age-verification overlay, but the media URLs are still exposed); GIFs are .webp animations.
- Interrupted downloads leave a `.part` file that the next attempt resumes with a Range request.
- Cloudflare challenges are detected and retried with backoff; a persistent challenge marks
  that site as failed for the request while other sites still return results.

## Output formats

`--format table|json|jsonl|csv`, `-o FILE`. JSON Lines output is accepted back by
`fetch --from`. `SearchResult`/`FetchResult` JSON includes per-site `errors` and `warnings`.

## Development

```bash
scripts/check.sh                       # ruff format --check, ruff check, mypy, pytest
uv run pytest -m network               # opt-in live checks against every site
uv run python scripts/capture_fixtures.py [site ...]   # raw captures for parser drift
uv run python scripts/trim_fixture.py IN.html OUT.html 'css selector' ...
```

Adapters live in `src/mediaq/sites/` (one module per site, shared mixins prefixed with `_`).
Each adapter is a pure parser (`parse_search`, `parse_detail`) plus URL building, tested
against trimmed HTML/JSON fixtures in `tests/fixtures/<site>/`. `PLAN.md` documents the
research findings and the implementation log.

## Hermes plugin

[`plugins/mediaq/`](plugins/mediaq/) wraps this CLI as a [Hermes](https://claude-code.nousresearch.com/docs)
plugin: `mediaq_search`, `mediaq_fetch`, `mediaq_sites`, `mediaq_jobs` and
`mediaq_config` tools, three bundled skills, and a `/mediaq` slash command. It
shells out to the installed `mediaq` binary, so Hermes' venv stays independent of
this project's dependencies.

```bash
uv tool install .
hermes plugins install ./plugins/mediaq && hermes plugins enable mediaq
```

See [`plugins/mediaq/README.md`](plugins/mediaq/README.md).

[uv]: https://docs.astral.sh/uv/
[wreq]: https://github.com/0x676e67/wreq-python
