Metadata-Version: 2.5
Name: llamaindex-webmetadata-extractor
Version: 0.1.0
Summary: LlamaIndex reader and tools for the Web Metadata, OpenGraph & Contact Extractor API — ingest any URL as a clean-Markdown Document for RAG, plus agent tools for contacts, tech stack, and SEO auditing.
Project-URL: Homepage, https://github.com/JosejuX/rapidapi-metadata-extractor
Project-URL: RapidAPI Listing, https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor
Project-URL: PyPI SDK, https://pypi.org/project/webmetadata-extractor/
Author-email: Juanjo <josejuanjo.coding@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agent,llama-index,llamaindex,metadata-extractor,rag,reader,seo,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Requires-Dist: llama-index-core>=0.10
Requires-Dist: pydantic>=2
Requires-Dist: webmetadata-extractor>=0.1.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: responses>=0.23; extra == 'dev'
Description-Content-Type: text/markdown

# llamaindex-webmetadata-extractor

LlamaIndex integration for the [Web Metadata, OpenGraph & Contact Extractor API](https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor) — a `Reader` that ingests any URL as a clean-Markdown `Document` for RAG, plus agent tools for contact discovery and SEO auditing.

```bash
pip install llamaindex-webmetadata-extractor
```

## Quick start

Get a free API key on [RapidAPI](https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor) (1,000 requests/month, no credit card required).

```bash
export WEBMETADATA_API_KEY=your-rapidapi-key
```

### As a Reader (RAG ingestion)

```python
from llamaindex_webmetadata_extractor import WebMarkdownReader
from llama_index.core import VectorStoreIndex

reader = WebMarkdownReader()  # reads WEBMETADATA_API_KEY, or pass api_key=...
documents = reader.load_data(["https://example.com", "https://another-site.com"])

index = VectorStoreIndex.from_documents(documents)
```

Each page's boilerplate (nav, ads, cookie banners) is stripped server-side before it reaches you, so `Document.text` is close to ready for chunking/embedding as-is. `Document.metadata` also carries the page's title, word count, reading time, and a short summary snippet. A URL that fails to fetch is skipped by default (`skip_errors=True`); pass `skip_errors=False` to raise instead.

### As agent tools

```python
from llamaindex_webmetadata_extractor import get_tools
from llama_index.core.agent import ReActAgent

tools = get_tools(api_key="YOUR_RAPIDAPI_KEY")
agent = ReActAgent.from_tools(tools, llm=llm)
```

| Tool | Use it when the agent needs to... |
|---|---|
| `web_metadata_extract` | Get a broad, all-in-one picture of a page (SEO, contacts, tech stack, security, links) |
| `web_contacts_extract` | Find public emails/phones/social links for outreach or lead-gen workflows |
| `web_seo_audit` | Grade a page's SEO with a score, warnings, and readability metrics |

Each tool returns a JSON string; API errors come back as `{"error": true, "status_code": ..., "detail": ...}` instead of raising, so an agent loop can react to them without crashing. For content ingestion (as opposed to a one-off agent tool call), use `WebMarkdownReader` above instead.

## Configuration

Pass `api_key` explicitly to the reader/tool factories, or set it once via the `WEBMETADATA_API_KEY` environment variable and omit it everywhere.

## Links

- [Python SDK this wraps](https://pypi.org/project/webmetadata-extractor/) (`webmetadata-extractor`)
- [Also available for LangChain](https://pypi.org/project/langchain-webmetadata-extractor/) and [CrewAI](https://pypi.org/project/crewai-webmetadata-extractor/)
- [Full API documentation](https://github.com/JosejuX/rapidapi-metadata-extractor)
- [Live interactive demo](https://rapidapi-metadata-extractor.onrender.com) — no signup needed

## License

MIT
