Metadata-Version: 2.4
Name: vidurl
Version: 0.3.0
Summary: Extract video URLs from web pages and generate curl download commands
Author: vidurl contributors
Maintainer: vidurl contributors
License: AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/gfrmin/vidurl
Project-URL: Repository, https://github.com/gfrmin/vidurl.git
Project-URL: Issues, https://github.com/gfrmin/vidurl/issues
Keywords: video,url,extractor,yt-dlp,playwright,curl,download,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: beautifulsoup4>=4.13.5
Requires-Dist: requests>=2.32.5
Requires-Dist: playwright>=1.50.0
Requires-Dist: yt-dlp>=2025.1.1
Provides-Extra: llm
Requires-Dist: scrapegraphai>=1.30.0; extra == "llm"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: isort>=5.12; extra == "dev"
Requires-Dist: flake8>=6.0; extra == "dev"
Dynamic: license-file

# vidurl

**A more powerful, cleverer yt-dlp.** Given any web page — a single-video page, a thumbnail-grid listing, or something obscure yt-dlp doesn't know about — `vidurl` downloads the videos.

[![License: AGPL v3](https://img.shields.io/badge/License-AGPLv3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)

## How it works

Three-tier escalation per URL, cheapest first:

1. **yt-dlp** handles whatever it can natively (thousands of sites, including playlists/channels).
2. **Playwright** loads the page and sniffs the rendered DOM, JS, network responses, and embedded iframes for a video URL — then downloads with `curl`. For listing pages, vidurl scrapes the DOM for video-page links and recurses.
3. **LLM** (optional, opt-in) — when the heuristics fail, vidurl can ask a large language model (via [scrapegraphai](https://github.com/ScrapeGraphAI/Scrapegraph-ai)) to identify the video URL or video-page links on the rendered DOM.

## Installation

```bash
uv sync
uv run playwright install chromium
```

`yt-dlp` and `playwright` are installed automatically. The LLM tier is optional:

```bash
uv sync --extra llm           # pulls in scrapegraphai
```

Also requires `curl` in `PATH`.

## Usage

```bash
# yt-dlp does the work
vidurl https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Site yt-dlp doesn't know — Playwright finds the video
vidurl https://example.com/embedded-video-page

# Listing of videos — vidurl visits each link
vidurl https://example.com/gallery
vidurl https://example.com/gallery --listing --link-selector "a.video-card"

# Don't download, just print the commands
vidurl https://example.com/page --dry-run

# Enable LLM fallback
vidurl https://example.com/weird-page \
    --llm-provider anthropic --llm-model claude-haiku-4-5
```

## Flags

| Flag | Purpose |
|---|---|
| `--output-dir / -o DIR` | Where to save files (default: cwd) |
| `--dry-run` | Print the yt-dlp / curl commands instead of executing |
| `--no-ytdlp` | Skip the yt-dlp tier; go straight to Playwright |
| `--ytdlp-args 'STR'` | Extra args appended to the yt-dlp invocation |
| `--listing` | Force listing mode (skip per-page video extraction) |
| `--no-listing` | Never recurse into links |
| `--link-selector CSS` | Use this CSS selector to find video-page links |
| `--link-pattern REGEX` | Only follow links whose absolute URL matches |
| `--min-links N` | Minimum links for listing auto-detect (default 3) |
| `--max-pages N` | Max listing pages to walk (default 10) |
| `--no-paginate` | Disable pagination — process only the first listing page |
| `--next-selector CSS` | CSS selector for the next-page link |
| `--next-pattern REGEX` | Regex; treat as next-page link only if URL matches |
| `--page-url-template URL` | URL template with `{n}`; vidurl walks 2..max-pages |
| `--llm-provider P` | `anthropic`, `openai`, `groq`, `google`, `ollama`, ... |
| `--llm-model M` | Model id |
| `--no-llm` | Disable the LLM tier even if provider/model are set |
| `--no-headless` | Show the browser window |
| `--timeout S` | Page load timeout (default 15s) |
| `--verbose / --quiet` | Logging |

## LLM tier

The LLM tier is **off by default**. To enable, pass both `--llm-provider` and `--llm-model` (or set them in `config.json`). API keys are read from environment first, then from gnome-keyring via `secret-tool` under `service=env, key=<PROVIDER_KEY>`. If a key is present but provider/model are not set, vidurl logs a hint and stays off — no silent spend.

## Pagination

When a listing page is detected, vidurl walks subsequent pages automatically (capped by `--max-pages`, default 10). Next-page discovery tries, in order:

1. `<link rel="next">` or `<a rel="next">`.
2. Anchor text or `aria-label` matching `Next`, `›`, `→`, `»`, `More` (filtered to avoid "next video" navigation).
3. URL-template inference (`?page=N`, `/page/N/`, `&offset=N`, trailing `/N`), validated with a HEAD/GET probe.
4. LLM fallback (if the LLM tier is enabled).

Override with `--next-selector`, `--next-pattern`, or `--page-url-template URL` (e.g. `https://example.com/list?page={n}`). Disable with `--no-paginate`.

## What's intentionally not (yet) supported

- **"Load more" buttons** and **infinite scroll** — these need click-and-wait logic without a URL change.
- **Parallel downloads** of multiple listing-page videos — sequential for now.

## Configuration

Pass a JSON file via `--config path/to/config.json`. See `config.example.json` for all fields.

## Development

```bash
uv sync --extra dev
uv run pytest        # no tests yet
```

## License

AGPL-3.0-or-later. See `LICENSE`.
