Metadata-Version: 2.4
Name: webdata-extractor
Version: 1.0.0
Summary: Pipeline-based web content extraction library
Home-page: 
Author: Dmytro Chernousan
Author-email: Dmytro Chernousan <chernousan@gmail.com>
License: MIT
Keywords: web,scraping,extraction,pipeline,metadata
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Requires-Dist: beautifulsoup4>=4.9.0
Requires-Dist: urllib3>=1.26.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.10; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.9; extra == "dev"
Requires-Dist: mypy>=0.900; extra == "dev"
Dynamic: author
Dynamic: license-file
Dynamic: requires-python

# WebData Extractor

Pipeline-based web content extraction library for extracting company information and metadata from web pages.

## 🚀 Features

- **Pipeline Architecture**: Modular architecture with independent processors
- **Extensibility**: Easy to add new processors without modifying existing code
- **Auto-Discovery**: Automatic discovery of processors in the folder
- **Prioritization**: Control processor execution order
- **Conditional Execution**: Processors can execute conditionally
- **Error Handling**: Pipeline continues working even if individual processors fail

## 📦 Installation

```bash
pip install -e .
```

Or for development:

```bash
pip install -e ".[dev]"
```

## 🔧 Usage

### Simple Usage

```python
from web_extractor import extract_company_info

# Extract company information
result = extract_company_info("example.com")

print(result["domain"])                    # Domain
print(result["metadata"]["og:site_name"]) # Company name (if available)
print(result["metadata"])                  # All metadata
```

### With Processor Selection

```python
from web_extractor import extract_company_info

# Use only standard processors
result = extract_company_info("example.com", use_all_processors=False)

# Or automatically discover all processors (default)
result = extract_company_info("example.com", use_all_processors=True)
```

### Advanced Usage

```python
from web_extractor import fetch_url, Pipeline
from web_extractor.processors import (
    MetaTagsProcessor,
    TitleProcessor,
    CustomProcessor  # Your custom processor
)

# Fetch HTML
data = fetch_url("example.com")

# Configure pipeline
pipeline = Pipeline()
pipeline.add_processors([
    MetaTagsProcessor(),
    TitleProcessor(),
    CustomProcessor(),
])

# Execute processing
result = pipeline.execute(data, data["soup"], data["html"])
```

### Auto-Discovery

```python
from web_extractor import discover_processors, extract_with_auto_discovery

# Show all discovered processors
processors = discover_processors()
for proc in sorted(processors, key=lambda p: p.priority):
    print(f"[{proc.priority:3d}] {proc.name}")

# Use all discovered processors
result = extract_with_auto_discovery("example.com")
```

## 🔌 Creating a Custom Processor

1. Create a file in `web_extractor/processors/my_processor.py`:

```python
from typing import Any, Dict
from bs4 import BeautifulSoup
from web_extractor.pipeline import DataProcessor


class MyProcessor(DataProcessor):
    @property
    def name(self) -> str:
        return "my_processor"
    
    @property
    def priority(self) -> int:
        return 60  # Execution priority
    
    def process(self, data: Dict[str, Any], soup: BeautifulSoup, html: str) -> Dict[str, Any]:
        # Your processing logic
        meta_tag = soup.find("meta", attrs={"name": "custom-tag"})
        if meta_tag:
            content = meta_tag.get("content")
            if content:
                data["metadata"]["custom_data"] = str(content)
        
        return data
```

2. Add to `web_extractor/processors/__init__.py`:

```python
from .my_processor import MyProcessor

__all__ = [
    # ... others
    'MyProcessor',
]
```

3. Done! The processor will execute automatically.

## 📊 Result Structure

```python
{
    "url": "https://example.com",
    "domain": "example.com",
    "metadata": {
        "og:site_name": "Example Company",
        "og:title": "Example - Home",
        "og:description": "...",
        "twitter:site": "examplecom",
        "title_raw": "Example Company - Official Site",
        "title": "Example Company",
        "description": "...",
        "copyright": {...},
        # ... other metadata
    },
    "processors_executed": [
        "meta_tags",
        "title",
        "copyright",
        "domain_fallback"
    ]
}
```

## 🎯 Existing Processors

| Priority | Processor | Description |
|----------|-----------|-------------|
| 10 | meta_tags | All meta tags (OpenGraph, Twitter, Dublin Core, Apple, Microsoft, Facebook, Article, etc.) |
| 30 | title | HTML title, application-name, title variants, title parts |
| 40 | copyright | Copyright from text, meta tags, schema.org, footer, link rel |
| 999 | domain_fallback | Fallback to domain name |

## 📝 Examples

See the `examples/` folder for complete examples:

- `examples/basic_usage.py` - Basic usage
- `examples/custom_processor.py` - Creating a custom processor
- `examples/advanced_pipeline.py` - Advanced pipeline configuration

## 🧪 Testing

```bash
# Run tests
pytest

# With coverage
pytest --cov=web_extractor --cov-report=html
```

## 📄 License

MIT License

## 🤝 Contributing

Contributions are welcome! Please create an issue or pull request.
