Metadata-Version: 2.5
Name: web2md-client
Version: 0.1.0
Summary: Python client for the Web2MD API: web pages to clean Markdown and heading-aware RAG chunks, with LangChain and LlamaIndex loaders.
Project-URL: Homepage, https://rapidapi.com/kanavsingla28/api/web2md-webpage-to-markdown-rag-chunks
Project-URL: Source, https://github.com/kanavsingla28/web2md-clients
Project-URL: Issues, https://github.com/kanavsingla28/web2md-clients/issues
Project-URL: Documentation, https://github.com/kanavsingla28/web2md-clients/tree/main/python#readme
Author: Kanav Singla
License-Expression: MIT
License-File: LICENSE
Keywords: chunking,langchain,llamaindex,llm,markdown,rag,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.12; extra == 'llamaindex'
Description-Content-Type: text/markdown

# web2md-client (Python)

Turn any web page into clean Markdown and heading-aware RAG chunks with the [Web2MD API](https://rapidapi.com/kanavsingla28/api/web2md-webpage-to-markdown-rag-chunks). The free BASIC plan allows 500 requests/month; PRO is $9/month for 10,000.

## Install

```bash
pip install web2md-client
pip install "web2md-client[langchain]"    # LangChain document loader
pip install "web2md-client[llamaindex]"   # LlamaIndex reader
```

Requires Python 3.10+. Get a key from the RapidAPI listing above and export it as `RAPIDAPI_KEY`.

## Quickstart

```python
from web2md_client import Web2MD

with Web2MD() as w:  # reads RAPIDAPI_KEY
    doc = w.read("https://example.com")
    print(doc.metadata.title, doc.tokens)
    print(doc.markdown)
    for chunk in w.chunks("https://example.com", max_tokens=512).chunks:
        print(chunk.heading_path, chunk.tokens)
```

Other methods: `convert(html, url=None)` for HTML you already have, and `metadata(url)` for page metadata only.

Async:

```py
import asyncio
from web2md_client import AsyncWeb2MD


async def main():
    async with AsyncWeb2MD() as w:
        doc = await w.read("https://example.com")
        print(doc.markdown)


asyncio.run(main())
```

## RAG over any website in 10 lines

No LLM key needed; embeddings run locally with FastEmbed.

```python
# pip install "web2md-client[langchain]" langchain-community fastembed
# export RAPIDAPI_KEY=...  (free key: https://rapidapi.com/kanavsingla28/api/web2md-webpage-to-markdown-rag-chunks)
from langchain_community.embeddings import FastEmbedEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

from web2md_client.langchain import Web2MDLoader

docs = Web2MDLoader(["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"], max_tokens=400).load()
store = InMemoryVectorStore.from_documents(docs, FastEmbedEmbeddings())
for hit in store.similarity_search("What problem does RAG solve?", k=3):
    print(f"[{hit.metadata['heading_path'] or hit.metadata.get('title', '')}]", hit.page_content[:300], "\n")
```

## LlamaIndex

No LLM key needed: this uses a local FastEmbed embedding and a retriever (no query engine).

```py
# pip install "web2md-client[llamaindex]" llama-index-embeddings-fastembed
from llama_index.core import Settings, VectorStoreIndex
from llama_index.embeddings.fastembed import FastEmbedEmbedding
from web2md_client.llamaindex import Web2MDReader

Settings.embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")
docs = Web2MDReader(max_tokens=400).load_data(["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"])
index = VectorStoreIndex.from_documents(docs)
for hit in index.as_retriever(similarity_top_k=3).retrieve("What problem does RAG solve?"):
    print(hit.node.metadata.get("heading_path"), hit.node.get_content()[:200])
```

`VectorStoreIndex.from_documents` may re-split our chunks with its default `SentenceSplitter`; that is harmless.

## Loader options

Both loaders accept keyword-only `include_links`, `include_images` and `include_tables` (each `True`, `False` or `None`). They are forwarded to the API only when set; `None` leaves the API default. `include_links=False` gives cleaner embeddings.

```py
Web2MDLoader(urls, include_links=False, include_tables=True)
```

A missing API key is reported when loading starts, not when the loader is constructed.

## Loader metadata

In `chunks` mode (the default), every LangChain `Document` / LlamaIndex node carries:

| Key | Meaning |
| --- | --- |
| `source` | The page URL |
| `title` | Page title |
| `heading_path` | Heading trail of the chunk, joined with `" > "` |
| `chunk_id` | Chunk identifier |
| `tokens` | Token count of the chunk |

In `read` mode the keys are `source`, `title`, `tokens`, `word_count`, `author`, `published`, `description`, `language` and `site_name`. Keys with no value are omitted.

## Errors

API, gateway and network failures raise `Web2MDError`; invalid arguments, such as passing both `url` and `html`, raise `ValueError`. `Web2MDError` has `.code`, `.message` and `.status` (HTTP status, or `None` for client-side failures).

```py
from web2md_client import Web2MD, Web2MDError

try:
    Web2MD().read("https://example.com")
except Web2MDError as e:
    print(e.code, e.status, e.message)
```

| Code | Status | Meaning |
| --- | --- | --- |
| `invalid_url` | 400 | The URL is malformed or not http(s) |
| `blocked_destination` | 400 | The URL points to a disallowed destination |
| `dns_error` | 400 | The host could not be resolved |
| `robots_disallowed` | 403 | The site's robots.txt forbids fetching |
| `page_too_large` | 413 | The page exceeds the size limit |
| `unsupported_content_type` | 415 | The URL is not an HTML page |
| `invalid_request` | 422 | The request parameters are invalid |
| `no_content` | 422 | No readable content could be extracted |
| `host_rate_limited` | 429 | The target host is rate limiting us |
| `target_blocked` | 502 | The target site refused the fetch |
| `target_error` | 502 | The target site returned an error |
| `fetch_failed` | 502 | The page could not be fetched |
| `too_many_redirects` | 502 | Redirect limit exceeded |
| `fetch_timeout` | 504 | The target site took too long |
| `rapidapi_auth` | 401/403 | Gateway rejected the key or you are not subscribed |
| `rapidapi_rate_limited` | 429 | Your plan's quota or rate limit is used up |
| `missing_api_key` | none | No `api_key` argument and `RAPIDAPI_KEY` is not set |
| `timeout` | none | The request exceeded the client timeout |
| `network_error` | none | Connection failed |
| `http_error` | any | Unexpected HTTP error |
| `invalid_response` | any | The response could not be parsed |

Loaders (`Web2MDLoader`, `Web2MDReader`) raise on the first failure by default. Pass `continue_on_failure=True` to skip failing URLs and keep loading the rest.

## Licence

MIT
