Metadata-Version: 2.4
Name: product-dl
Version: 0.0.0.dev3
Summary: Youtube-DL but for product pages.
Author-email: Lucky Lucy <wrk2026._.code@luckydonald.de>
License-File: LICENSE
Requires-Python: >=3.14
Requires-Dist: beautifulsoup4>=4.14.3
Requires-Dist: fastapi[standard]>=0.135.1
Requires-Dist: httpx-curl-cffi>=0.1.5
Requires-Dist: httpx>=0.28.1
Requires-Dist: lxml>=6.0.2
Requires-Dist: natsort>=8.4.0
Requires-Dist: playwright-stealth>=2.0.3
Requires-Dist: playwright>=1.58.0
Requires-Dist: pydantic>=2.13.0b2
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: typer>=0.24.1
Description-Content-Type: text/markdown

# product-dl / product-ui / list-dl

This repository contains three packages:

- **`product-dl`** — "youtube-dl but for product pages". Given a URL it detects the right provider, scrapes the product page, and returns a normalised `ParsedProduct` as JSON. Exposes a CLI and a FastAPI JSON API (default port **7001**).
- **`product-ui`** — UI companion. Calls the `product-dl` API, persists crawls to disk, and serves a Bootstrap web interface for browsing and triggering crawls (default port **7002**).
- **`list-dl`** — Scrapes order histories and wishlists on demand. Like `product-dl` but for lists (Amazon orders, public wishlists, etc.). Returns structured `Order` objects. Exposes a CLI and a FastAPI JSON API (default port **7003**).

---

## product-dl

### Supported sites

| Provider | Domains | ID format |
|---|---|---|
| `kleinanzeigen` | kleinanzeigen.de | `/s-anzeige/…/<id>` |
| `ebay` | ebay.de, ebay.com, ebay.co.uk, … | `/itm/…/<id>` |
| `amazon` | amazon.de, amazon.com, amazon.co.uk, … | `/dp/<ASIN>`, `/gp/product/<ASIN>` |
| `aliexpress` | aliexpress.com, aliexpress.us | `/item/….html` |
| `thomann` | thomann.de, thomannmusic.com | `/<slug>.htm` |

### Install

```bash
pip install product-dl        # or: uv add product-dl
```

### CLI

```bash
# Fetch a product page and print JSON
product-dl fetch <url>

# Extra options
product-dl fetch <url> \
  --headers '{"Accept-Language": "de-DE"}' \
  --cookies path/to/cookies.json \
  --data-dir ./data \
  --save-example my-label \
  --indent 4

# Parse a locally saved HTML file
product-dl parse amazon path/to/product.html --url https://www.amazon.de/dp/B08N5WRWNW

# Start the API server
product-dl serve --host 0.0.0.0 --port 7001

# Export the OpenAPI schema (default: YAML)
product-dl openapi
product-dl openapi --format json --output ./openapi.json

# List discovered providers
product-dl providers
product-dl providers --format json
```

### `--data-dir` / `PRODUCT_DL_DATA_DIR`

When set, the CLI (and API server via env var) will:

- Auto-load `<data-dir>/cookies/<provider>.json` and
  `<data-dir>/headers/<provider>.json` before each request.
- Detect **Akamai CDN bot-detection** and switch to
  `httpx-curl-cffi` (Chrome TLS impersonation) automatically:
  - If any Akamai cookie key (`ak_bmsc`, `_abck`, `bm_sz`, `bm_sv`, `RT`)
    is present in the merged cookies.
  - Or if a 503 response body contains Akamai signatures
    (`errors.edgesuite.net` / `Zero size object`).
- Save Akamai cookies received in curl_cffi responses back to
  `<data-dir>/cookies/<provider>.json` for reuse.

```bash
# CLI
product-dl fetch https://www.ebay.de/itm/123 --data-dir ./data

# env var (also used by the API server)
export PRODUCT_DL_DATA_DIR=./data
product-dl fetch https://www.ebay.de/itm/123
```

### API

```bash
product-dl serve          # http://127.0.0.1:7001
# or
uvicorn product_dl.api.app:app
```

```
GET  /fetch?url=<url>
POST /fetch  {"url": "<url>"}
```

Both return a `ParsedProduct` JSON object. `PRODUCT_DL_DATA_DIR` is read
server-side from the environment — it is not an API parameter.

Latest generated API descriptions are published to the repository `gh-pages`
branch as:

- `product_dl/openapi.json`
- `product_dl/openapi.yml`
- `product_dl/providers.json`
- `list_dl/openapi.json`
- `list_dl/openapi.yml`
- `list_dl/providers.json`
- `list_dl/login_providers.json`
- `catalog/commit_map.json`

### Output schema (`ParsedProduct`)

```
ParsedProduct
├── id: str | int | None
├── title: str
├── description: str | None
├── gallery: list[Image | Video | PDF] | None
│     Image / Video ── url, mime, thumb, width, height
│     Video         ── length (seconds / frames)
│     PDF           ── url, name, page_count
├── pricing: Pricing | None
│     ├── price: Price | None        (value, unit, raw, symbol, suffix)
│     ├── shipping: list[Shipping]   (name, price)
│     ├── original: Price | None     (pre-discount list price)
│     └── reduction: list[Reduction] (percentage or price discount)
├── canonical_url: str | None
├── fetch_url: str
├── parser: str                      e.g. "kleinanzeigen", "amazon"
├── website_variant: str | None       e.g. "de", "co.uk", "gb"
├── version: int                     schema version (starts at 1)
├── available: bool | None
├── seller: Seller | None            (name, id, url, picture)
├── brand: Brand | None              (name, url, logo)
├── category: Category | None        (name, id, url, parent)
├── variation_types: list[VariationType] | None
├── tables: dict[str, dict[str, str]] | None
├── meta: PageMeta | None            (meta tags, LD+JSON, tracking)
└── extra: <ProviderExtra>           provider-specific fields
```

### Adding a provider

**Built-in** — create `src/product_dl/providers/<name>/__init__.py` exposing:

```python
PROVIDER_NAME: str
def can_provide(url: str, parsed_url: ParseResult) -> T | None: ...
async def provide(
    url: str,
    parsed_url: ParseResult,
    provide_check: T,
    *,
    save_files: SaveFiles | None = None,
    local_html: str | None = None,
    extra_headers: dict[str, str] | None = None,
    extra_cookies: dict[str, str] | None = None,
    data_dir: Path | None = None,
) -> ParsedProduct: ...
```

Auto-discovered via `pkgutil` — no `pyproject.toml` changes needed.
Use `product_dl.utils.http.fetch_html` for HTTP requests so Akamai handling
and cookie persistence work automatically.

**Third-party** — publish a package with a `product_dl.providers` entry point.
No changes to this project needed.

---

## product-ui

### Install

```bash
pip install product-dl        # product-ui is bundled in the same package
```

### Quick start

```bash
# 1. Start the product-dl API backend
product-dl serve

# 2. Start the UI (in another terminal)
product-ui serve

# 3. Open http://127.0.0.1:7002
```

### CLI

```bash
# Start the web UI
product-ui serve [--host 127.0.0.1] [--port 7002] [--reload] \
                 [--data-dir ./crawl_data/product_dl] [--product-dl-api http://localhost:7001]

# Fetch a URL and save to disk (product-dl must be running)
product-ui fetch <url> [--data-dir ./crawl_data/product_dl] [--product-dl-api http://localhost:7001]
```

### Environment variables

| Variable | Default | Description |
|---|---|---|
| `PRODUCT_UI_DATA_DIR` | `./crawl_data/product_dl` | Directory where crawls are stored |
| `PRODUCT_DL_API` | `http://localhost:7001` | Base URL of the `product-dl` JSON API |

### Disk layout

```
{data_dir}/{provider}/{YYYY}/{MM}/{DD}/{HH_MM_SS}/{id}/
    crawl.json                  ← {"information": {url, provider, variant, time}, "crawl": <ParsedProduct>}
    files/full/{nnn}.{ext}      ← base64 gallery files decoded to disk
    files/thumb/{nnn}.{ext}     ← thumbnails (same index number as matching full)
{data_dir}/index.jsonl          ← append-only JSON-Lines: {provider, url, id, title, date, path}
```

### Web UI routes

| Route | Description |
|---|---|
| `GET /` | Filterable list of crawls; includes new-crawl form (filters via GET → shareable URLs) |
| `GET /crawl?url=<path>` | Detail view: gallery, pricing, variations, tables, provider extras |
| `POST /fetch` | Trigger a new crawl (form field `url`); saves to disk, redirects to detail |
| `GET /data/…` | Static serving of saved crawl files (images, PDFs, etc.) |

---

## list-dl

Scrapes order histories and wishlists from e-commerce sites on demand. Mirror of `product-dl` architecture but for lists instead of individual products.

### Supported providers

| Provider | Status | Notes |
|---|---|---|
| `amazon_orders` | Active | Amazon order history scraper (pagination, delivery states, items). Requires authentication. |

### Install

```bash
# Bundled with product-dl
pip install product-dl
```

### Login & Credentials

`list-dl` uses a pluggable login provider system. Each provider handles site-specific authentication and stores credentials locally for reuse.

#### Login Command

```bash
# Interactive login (prompts for password)
list-dl login amazon --username user@example.com

# Non-interactive login (provide all credentials)
list-dl login amazon --username user@example.com --password mypassword

# Login to a specific flavor (e.g., amazon.de)
list-dl login amazon --flavor de --username user@example.com --password mypassword
```

Credentials are stored in `./crawl_data/list_dl/login/{site}/login.json`.

#### Environment Variables

Credentials can be loaded from environment variables. Precedence (highest to lowest):
1. CLI arguments (`--username`, `--password`, `--code`)
2. `LOGIN_{SITE}_{FLAVOR}_{FIELD}` (e.g., `LOGIN_AMAZON_DE_USERNAME`)
3. `LOGIN_{SITE}_{FIELD}` (e.g., `LOGIN_AMAZON_USERNAME`)
4. Generic field name (e.g., `USERNAME`, `PASSWORD`, `CODE`) — **Warning:** generic `USERNAME` defaults to the Linux user if not explicitly set

```bash
# Load credentials from environment
export LOGIN_AMAZON_USERNAME=user@example.com
export LOGIN_AMAZON_PASSWORD=mypassword
list-dl login amazon

# Or with flavor
export LOGIN_AMAZON_DE_USERNAME=user@example.com
export LOGIN_AMAZON_DE_PASSWORD=mypassword
list-dl login amazon --flavor de

# Non-interactive (no TTY)
echo mypassword | list-dl login amazon --username user@example.com
```

#### .env File Support

The login command reads from `.env.local` (highest priority) and `.env` files:

```bash
# .env.local (excluded from git)
LOGIN_AMAZON_USERNAME=user@example.com
LOGIN_AMAZON_PASSWORD=mypassword
```

Then:
```bash
list-dl login amazon
```

#### Two-Factor Authentication (2FA / TOTP)

When a site requires 2FA, `list-dl` supports three workflows:

**Workflow 1: Single command (all credentials at once)**
```bash
list-dl login amazon --username user@example.com --password mypass --code 123456
```
Automatically chains login → 2FA → stores final credentials.

**Workflow 2: Two separate commands**
```bash
# First run: login, save partial credentials with TOTP form
list-dl login amazon --username user@example.com --password mypass
# Output: ⚠ TOTP code required. Run: list-dl login amazon --code <code> --flavor com

# Second run: submit 2FA code using stored credentials
list-dl login amazon --code 123456
```

**Workflow 3: Interactive mode (detects TTY)**
```bash
# Prompts for missing credentials, including 2FA code when required
list-dl login amazon
# Email address: user@example.com
# Password: (hidden input)
# 2FA Code (6 digits): 123456
```

2FA codes can also be provided via the `CODE` environment variable:
```bash
export CODE=123456
list-dl login amazon
```

### CLI

```bash
# Fetch orders from Amazon (requires prior login)
list-dl fetch amazon orders --mode my_orders

# Fetch a public wishlist (no login required)
list-dl fetch amazon wishlist --url https://www.amazon.de/hz/wishlist/ls/ABC123

# Fetch from a different flavor (amazon.de)
list-dl fetch amazon orders --flavor de --mode my_orders

# Start the API server
list-dl serve --host 0.0.0.0 --port 7003

# Export the OpenAPI schema (default: YAML)
list-dl openapi
list-dl openapi --format json --output ./openapi.json

# List discovered providers
list-dl providers
list-dl providers --format json
list-dl login-providers --format json
```

### API

```bash
list-dl serve          # http://127.0.0.1:7003
```

Credentials can be supplied in the request:

```
POST /fetch
{
  "site": "amazon",
  "category": "orders",
  "mode": "my_orders",
  "credentials": {"username": "...", "password": "...", "cookies": {...}},
  "fetch_details": true
}
```

Or, if already logged in, credentials are auto-loaded from disk:

```bash
curl -X POST http://localhost:7003/fetch \
  -H "Content-Type: application/json" \
  -d '{"site": "amazon", "category": "orders", "mode": "my_orders"}'
```

### Output schema (`Order`)

```
Order
├── order_number: str                    unique order ID
├── provider_name: str                   e.g. "amazon_orders"
├── url: str | None                      link to order detail page
├── order_date: date | None              when order was placed
├── total: Price | None                  {value, unit, raw, symbol, suffix}
├── dispatch_to: Address | None          shipping address
├── delivery: DeliveryStatus | None      {state, substate}
│     state: "delivered" | "on_the_way_late" | "cancelled" | ...
│     substate: Optional freeform substate (e.g. "Parcel left in letterbox")
├── delivered_at: datetime | None
├── return_until: date | None
├── documents: list[Attachment] | None   invoices, receipts, etc.
├── summary: OrderSummary | None         price breakdown
│     ├── items_subtotal, postage, total_before_vat, estimated_vat, total, grand_total
├── payment: PaymentMethod | None        how order was paid
│     ├── type, bank, masked_number
├── items: list[OrderItem] | None        products in order
│     ├── link: URL
│     ├── product: ParsedProduct | None  (fetched via product-dl if available)
│     └── extra: <ProviderItemExtra>
└── extra: <ProviderOrderExtra>          provider-specific fields
```

### Adding a provider

**Built-in** — create `src/list_dl/providers/<name>/__init__.py` with:

```python
from list_dl.providers.base import ListProviderBase, register

@register
class MyOrdersProvider(ListProviderBase[CHECK_TYPE, MyOrderExtra, MyItemExtra]):
    PROVIDER_NAME = "my_orders"

    @classmethod
    def can(cls, provider_name, mode, url) -> CHECK_TYPE | None:
        """Return truthy if this provider can handle the request."""
        ...

    async def init(self) -> None:
        """Initialize (e.g., open HTTP client)."""
        ...

    async def fetch_orders(self) -> AsyncIterator[Order]:
        """Yield Order objects."""
        ...

    async def fetch_order_detail(self, order: Order) -> Order:
        """Enrich order with detail page data (items, summary, payment)."""
        return order  # or enriched order
```

Auto-discovered via `pkgutil` — no `pyproject.toml` changes needed. Use `httpx` with optional Akamai bypass via curl_cffi (same pattern as `product-dl`).

# Deploy
1. Remember to bump the version in pyproject.toml:3
2. ```shell
   export PYPI_TOKEN=''
   ```
3. ```shell
   uv build
   uv publish --token "$PYPI_TOKEN"
   ```
