Metadata-Version: 2.4
Name: qrawlix
Version: 1.0.0
Summary: High-performance async web scraping and asset retrieval with TLS fingerprint emulation and concurrent processing.
Project-URL: Homepage, https://github.com/YOUR_USER/qrawlix
Author: Qrawlix Contributors
License: MIT
License-File: LICENSE
Keywords: async,asyncio,concurrent,crawler,data-extraction,http-client,parallel,scraping,tls-fingerprint,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Requires-Dist: beautifulsoup4>=4.12.3
Requires-Dist: curl-cffi>=0.5.10
Requires-Dist: selectolax>=0.3.17
Description-Content-Type: text/markdown

"""
Qrawlix — High-Performance Asynchronous Web Scraping & Asset Retrieval
======================================================================

Qrawlix is a research-oriented Python library for fetching web pages,
extracting structured data, and downloading binary assets (images, documents,
archives) at scale. It is built on top of ``curl_cffi``, which provides
browser-grade TLS fingerprint emulation — allowing requests to appear as
though they originate from standard browsers (Chrome, Firefox, Safari, Edge).

Key capabilities
----------------
- **TLS fingerprint rotation** — automatic per-request browser identity
  cycling to improve compatibility with diverse server configurations.
- **Concurrent scraping** — fetch dozens of pages simultaneously with
  configurable parallelism and built-in rate-limiting.
- **Concurrent asset downloading** — download hundreds of images or files
  in parallel, with per-directory organisation.
- **Structured data extraction** — declarative CSS-selector-based rules for
  pulling titles, prices, links, and other fields from HTML.
- **Multi-strategy request routing** — automatic fallback through secondary
  channels when a direct connection is unavailable.
- **Parallel engine** — high-level orchestrator that handles the full
  pipeline (scrape → extract → download → organise) in one call.

Installation
------------
::

    pip install qrawlix

Or from source::

    git clone https://github.com/YOUR_USER/qrawlix.git
    cd qrawlix
    pip install .

Quick start
-----------
Scrape a single page and extract the title::

    import asyncio
    from qrawlix import QrawlixClient

    async def main():
        client = QrawlixClient()
        result = await client.scrape("https://example.com")
        if result["success"]:
            title = result["parser"].text("h1")
            print(f"Page title: {title}")

    asyncio.run(main())

Scrape multiple pages and download all discovered images in one call::

    from qrawlix.parallel import scrape_pages

    results = asyncio.run(scrape_pages([
        "https://site-a.com/gallery",
        "https://site-b.com/chapter/42",
    ]))

    print(f"{results['succeeded']}/{results['targets']} sites processed")
    print(f"{results['images_downloaded']} images downloaded")

Output is automatically organised::

    qrawlix_output/
    ├── summary.json
    ├── site-a.com/
    │   ├── page.html
    │   ├── data.json
    │   └── images/
    │       ├── photo_001.jpg
    │       └── photo_002.png
    └── site-b.com/
        ├── page.html
        ├── data.json
        └── images/
            └── ...

Core API
--------

**QrawlixClient** — async HTTP client
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    from qrawlix import QrawlixClient

    client = QrawlixClient(
        max_concurrency=10,   # max simultaneous requests
        timeout=15.0,         # per-request timeout (seconds)
        retries=1,            # retry attempts on failure
    )

    # Single URL
    result = await client.scrape("https://example.com")
    # result["success"]         — bool
    # result["html"]            — raw response text
    # result["parser"]          — ContentParser instance
    # result["elapsed_ms"]      — request latency
    # result["status_code"]     — HTTP status

    # Structured extraction
    result = await client.scrape("https://example.com", rules={
        "title": {"selector": "h1", "type": "text"},
        "links": {"selector": "a", "type": "attr_all", "attr": "href"},
    })
    # result["extracted_data"]  — dict with extracted fields

    # Multiple URLs concurrently
    batch = await client.scrape_many([
        "https://site-a.com",
        "https://site-b.com",
    ])
    # batch["successful"]       — count of successful fetches
    # batch["results"]          — list of per-URL result dicts


**ContentParser** — HTML / markdown parser
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    parser = result["parser"]          # obtained from QrawlixClient.scrape()

    parser.text("h1")                  # first match inner text
    parser.text_all("p")               # all match inner texts
    parser.attribute("a", "href")      # first match attribute
    parser.attribute_all("img", "src") # all match attributes

    # Declarative extraction
    data = parser.extract_by_rules({
        "title":   {"selector": "h1",              "type": "text"},
        "price":   {"selector": ".price",          "type": "text"},
        "image":   {"selector": "img.product",     "type": "attr",  "attr": "src"},
        "gallery": {"selector": ".gallery img",    "type": "attr_all", "attr": "src"},
    })

    # List extraction (repeating containers)
    items = parser.extract_list("div.product-card", {
        "name":  {"selector": "h2",   "type": "text"},
        "link":  {"selector": "a",    "type": "attr", "attr": "href"},
    })


**AssetPipeline** — concurrent binary downloader
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    from qrawlix import AssetPipeline

    pipeline = AssetPipeline(
        output_dir="downloads",
        concurrency=20,          # max simultaneous downloads
    )
    dl = await pipeline.download_all([
        "https://example.com/photo1.jpg",
        "https://example.com/photo2.png",
    ])
    # dl["total_downloaded"]   — success count
    # dl["total_bytes"]        — total size downloaded
    # dl["results"]            — per-asset status dicts


**ParallelEngine** — full pipeline orchestrator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    from qrawlix import ParallelEngine

    engine = ParallelEngine(
        scrape_concurrency=8,    # concurrent page fetches
        download_concurrency=24, # concurrent asset downloads
    )
    summary = await engine.run([
        "https://site-a.com/gallery",
        "https://site-b.com/chapter/1",
    ], output_root="my_output")

    # One-liner equivalent:
    from qrawlix.parallel import scrape_pages
    summary = await scrape_pages(urls, output="my_output")


Configuration
-------------
Concurrency levels are automatically derived from system CPU count
when not explicitly set.  The default heuristic is ``max(4, min(cpu*2, 32))``.

Custom image selectors can be passed to the parallel engine for
domain-specific extraction::

    engine = ParallelEngine(image_selectors={
        "my-site.com": [".custom-gallery img", "img.content"],
    })

Important notes
---------------
- This library is provided for **research and educational purposes**.
  Users are responsible for complying with the terms of service of any
  website they interact with.
- Qrawlix uses public content-reader proxies and translation services
  as secondary request channels when a direct connection is unavailable.
  These are best-effort fallbacks and may not work for all endpoints.
- The library does **not** execute JavaScript.  Pages that rely entirely
  on client-side rendering (e.g., Google Images, React SPAs) will return
  limited content from the initial HTML payload.

License
-------
MIT License — see the LICENSE file for details.
"""
