Metadata-Version: 2.4
Name: dictionary-crawler
Version: 0.1.1
Summary: A Python package for crawling Cambridge and OxfordLearners dictionaries
Project-URL: Homepage, https://github.com/ArshinMQ/dictionary-crawler
Project-URL: Issues, https://github.com/ArshinMQ/dictionary-crawler/issues
Author-email: Arshin <fatemmaqu@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: beautifulsoup4
Requires-Dist: httpx
Requires-Dist: pydantic>=2.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.22; extra == 'dev'
Description-Content-Type: text/markdown

# 🦄 Dictionary Crawler

A Python package for crawling dictionary websites. Currently supports Cambridge Dictionary and Oxford Learners Dictionary.

## Installation

```bash
pip install dictionary-crawler
```

Or install from source:

```bash
git clone https://github.com/arshinmq/dictionary-crawler.git
cd dictionary-crawler
pip install -e .
```

## Features

- Async HTTP client for non-blocking requests
- Parse Cambridge and OxfordLearners Dictionary pages
- Extract:
  - Headwords and parts of speech
  - Pronunciations (IPA + audio URLs)
  - Definitions with CEFR levels
  - Examples and sources
  - Phrases, idioms, and phrasal verbs
  - Collocations
  - Verb forms
  - Topics and semantic domains (OxfordLearners)
- Structured Pydantic models for type-safe data access
- JSON serialization support

## Quick Start

```python
import asyncio
from dictionary_crawler import CambridgeCrawler, OxfordLearnersCrawler

async def main():
    # Cambridge Dictionary
    async with CambridgeCrawler() as crawler:
        page = await crawler.crawl("run")
        
        print(f"Search term: {page.search_term}")
        
        for section in page.sections:
            for entry in section.entries:
                print(f"\n{entry.headword} ({entry.part_of_speech})")
                
                if entry.pronunciations:
                    for p in entry.pronunciations:
                        print(f"  [{p.region}] {p.pronunciation}")
                
                for sense in entry.senses:
                    for defn in sense.definitions:
                        print(f"  - {defn.definition}")
                        if defn.examples:
                            for ex in defn.examples:
                                if ex.example:
                                    print(f"    e.g. {ex.example}")
    
    # OxfordLearners Dictionary
    async with OxfordLearnersCrawler() as crawler:
        page = await crawler.crawl("run")
        
        print(f"\nOxfordLearners - Search term: {page.search_term}")
        
        for definition in page.definitions:
            print(f"\n{definition.headword} ({definition.part_of_speech})")
            
            if definition.pronunciations:
                for p in definition.pronunciations:
                    print(f"  [{p.phonetic}]")
            
            for sense in definition.senses:
                print(f"  - {sense.definition}")
                if sense.examples:
                    for ex in sense.examples:
                        print(f"    e.g. {ex.example}")

asyncio.run(main())
```

## API Reference

### CambridgeCrawler

The main crawler class for Cambridge Dictionary.

```python
from dictionary_crawler import CambridgeCrawler

async with CambridgeCrawler() as crawler:
    page = await crawler.crawl(word: str) -> CambridgePage
```

#### Parameters

- `client` (optional): Custom `httpx.AsyncClient` instance

### OxfordLearnersCrawler

The main crawler class for OxfordLearners Dictionary.

```python
from dictionary_crawler import OxfordLearnersCrawler

async with OxfordLearnersCrawler() as crawler:
    page = await crawler.crawl(word: str, direct_call: bool = False) -> OxfordLearnersPage
```

#### Parameters

- `client` (optional): Custom `httpx.AsyncClient` instance
- `direct_call` (optional): If using the exact endpoint, pass this as true so the package doesn't search the term through OxfordLearners API to find a match. Default is False.

### Data Models

#### OxfordLearnersPage

The root model for OxfordLearners Dictionary data:

| Field | Type | Description |
|-------|------|-------------|
| `search_term` | `str` | The word that was searched |
| `definitions` | `list[OxfordLearnersDefinition]` | List of definitions |

#### OxfordLearnersDefinition

| Field | Type | Description |
|-------|------|-------------|
| `headword` | `str` | The word/phrase |
| `part_of_speech` | `str` | noun, verb, adjective, etc. |
| `pronunciations` | `list[OxfordLearnersPronunciation]` | Pronunciations |
| `cefr_level` | `CEFRLevel \| None` | CEFR level (A1-C2) |
| `senses` | `list[OxfordLearnersSense]` | Word senses with definitions |
| `verb_forms` | `list[OxfordLearnersDefinitionVerbForm] \| None` | Verb conjugations |
| `idioms` | `list[OxfordLearnersIdiom] \| None` | Related idioms |
| `phrasal_verbs` | `list[str] \| None` | Related phrasal verbs |

#### OxfordLearnersPronunciation

| Field | Type | Description |
|-------|------|-------------|
| `phonetic` | `str` | IPA transcription |
| `pronunciation_url` | `str` | Audio file URL |

#### OxfordLearnersSense

| Field | Type | Description |
|-------|------|-------------|
| `semantic_domain` | `str` | Domain category |
| `definition` | `str` | The definition text |
| `synonyms` | `list[str] \| None` | Synonyms |
| `topics` | `list[OxfordLearnersSenseTopic] \| None` | Topics with CEFR levels |
| `cefr_level` | `CEFRLevel \| None` | CEFR level (A1-C2) |
| `grammar_labels` | `list[str] \| None` | Grammar labels |
| `labels` | `list[str] \| None` | Usage labels |
| `examples` | `list[OxfordLearnersExample] \| None` | Example sentences |
| `collocations` | `list[OxfordLearnersSenseCollocation] \| None` | Collocations |

#### OxfordLearnersIdiom

| Field | Type | Description |
|-------|------|-------------|
| `idiom` | `str` | The idiom text |
| `senses` | `list[OxfordLearnersIdiomSense]` | Idiom senses |

#### CambridgePage

The root model containing all parsed data:

| Field | Type | Description |
|-------|------|-------------|
| `search_term` | `str` | The word that was searched |
| `sections` | `list[CambridgeSection]` | Dictionary sections (British, American, Business) |
| `examples` | `list[CambridgeExample] \| None` | Example sentences |
| `collocations` | `list[CambridgeCollocation] \| None` | Collocations |

#### CambridgeSection

| Field | Type | Description |
|-------|------|-------------|
| `type` | `CambridgeSectionType` | British, American, or Business |
| `entries` | `list[CambridgeEntry]` | Dictionary entries |

#### CambridgeEntry

| Field | Type | Description |
|-------|------|-------------|
| `headword` | `str` | The word/phrase |
| `part_of_speech` | `str` | noun, verb, adjective, etc. |
| `pronunciations` | `list[CambridgePronunciation] \| None` | Pronunciations |
| `senses` | `list[CambridgeSense]` | Word senses with definitions |
| `phrases` | `list[CambridgeSectionPhrase] \| None` | Related phrases |
| `idioms` | `list[str] \| None` | Related idioms |
| `phrasal_verbs` | `list[str] \| None` | Related phrasal verbs |
| `verb_forms` | `list[CambridgeVerbForm] \| None` | Verb conjugations |

#### CambridgePronunciation

| Field | Type | Description |
|-------|------|-------------|
| `pronunciation` | `str` | IPA transcription |
| `audio_urls` | `list[str]` | Audio file URLs |
| `region` | `str` | e.g., "UK", "US" |

#### CambridgeSense

| Field | Type | Description |
|-------|------|-------------|
| `cefr` | `CEFRLevel \| None` | CEFR level (A1-C2) |
| `definitions` | `list[CambridgeDefinition]` | Definitions |

#### CambridgeDefinition

| Field | Type | Description |
|-------|------|-------------|
| `definition` | `str` | The definition text |
| `examples` | `list[CambridgeDefinitionExample] \| None` | Example sentences |
| `image_urls` | `list[str] \| None` | Illustration URLs |

#### CEFRLevel

Enum with values: `A1`, `A2`, `B1`, `B2`, `C1`, `C2`

### Exceptions

| Exception | Description |
|-----------|-------------|
| `DictionaryException` | Base exception |
| `DictionaryFetchException` | HTTP request failed |
| `DictionaryWordNotFoundException` | Word not in dictionary |
| `DictionaryParseException` | Failed to parse page |
| `CambridgeException` | Cambridge-specific base |
| `CambridgeFetchException` | Cambridge fetch error |
| `CambridgeWordNotFoundException` | Word not in Cambridge |
| `CambridgeParseException` | Cambridge parse error |
| `OxfordLearnersException` | OxfordLearners-specific base |
| `OxfordLearnersFetchException` | OxfordLearners fetch error |
| `OxfordLearnersWordNotFoundException` | Word not in OxfordLearners |
| `OxfordLearnersParseException` | OxfordLearners parse error |
| `OxfordLearnersUnexpectedContent` | Unexpected OxfordLearners page content |

## JSON Export

Models can be serialized to JSON:

```python
page = await crawler.crawl("example")
json_data = page.model_dump_json(indent=2, ensure_ascii=False)
print(json_data)
```

## Development

### Setup

```bash
python -m venv venv
source venv/bin/activate
pip install -e ".[dev]"
```

### Running Tests

```bash
pytest
```

### Project Structure

```
dictionary-crawler/
├── src/
│   └── dictionary_crawler/
│       ├── __init__.py
│       ├── base.py          # BaseCrawler ABC
│       ├── http.py          # HTTP client setup
│       ├── utils.py         # URL utilities
│       ├── exceptions.py    # Custom exceptions
│       ├── crawlers/
│       │   ├── cambridge.py # Cambridge parser
│       │   └── oxford_learners.py    # OxfordLearners parser
│       └── models/
│           ├── common.py    # Shared models (CEFRLevel)
│           ├── cambridge.py # Cambridge data models
│           └── oxford_learners.py    # OxfordLearners data models
└── tests/ # Will add soon
```

## Roadmap (Future)
- Support for additional dictionaries (Merriam-Webster, Collins, etc.)
- Multi-language support (Spanish, French, German, etc.)
- Add more test coverage for OxfordLearners support

## License

MIT
