Metadata-Version: 2.4
Name: nltk-extratokenizers
Version: 0.1.1
Summary: Core NLTK model/data files (punkt, tagger, stopwords, wordnet, etc.) packaged for pip installation
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Julien-ser/nltk_Group1_ENGG4450
Project-URL: Repository, https://github.com/Julien-ser/nltk_Group1_ENGG4450
Project-URL: Issues, https://github.com/Julien-ser/nltk_Group1_ENGG4450/issues
Keywords: nltk,nlp,tokenizer,punkt,natural-language-processing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# nltk-extratokenizers

**Core NLTK model/data files packaged for pip installation**

`nltk-extratokenizers` provides essential NLTK data files (tokenizers, taggers, corpora) as a pip-installable package, eliminating the need for `nltk.download()`. This package implements [NLTK Issue #3413](https://github.com/nltk/nltk/issues/3413) by using Python entry points to make NLTK data available directly from pip.

## Installation

```bash
pip install nltk-extratokenizers
```

**Important:** This package requires a patched version of NLTK that supports entry-point-based data loading. You'll need to use the modified NLTK from this repository:

```bash
git clone https://github.com/Julien-ser/nltk_Group1_ENGG4450.git
cd nltk_Group1_ENGG4450
pip install -e .
```

Then install the data package:

```bash
pip install nltk-extratokenizers
```

## What's Included

This package provides the following NLTK data resources:

### Tokenizers
- **punkt** - Sentence tokenizer models

### Taggers
- **averaged_perceptron_tagger** - Part-of-speech tagger

### Corpora & Lexical Resources
- **stopwords** - Stopword lists for multiple languages
- **wordnet** - WordNet lexical database
- **omw-1.4** - Open Multilingual Wordnet
- **snowball_data** - Snowball stemmer data
- **names** - Name corpus
- **brown** - Brown corpus
- **movie_reviews** - Movie reviews corpus

## Usage

Once installed, you can use NLTK tokenizers and other resources **without calling `nltk.download()`**:

### Example: Using Punkt Sentence Tokenizer

```python
from nltk.tokenize import PunktSentenceTokenizer

# No nltk.download() needed!
tok = PunktSentenceTokenizer()
sentences = tok.tokenize("Hello world. This is a test. Another sentence here.")
print(sentences)
# Output: ['Hello world.', 'This is a test.', 'Another sentence here.']
```

### Example: Using Word Tokenizer

```python
from nltk.tokenize import word_tokenize

# Works without nltk.download()
tokens = word_tokenize("Natural language processing is fun!")
print(tokens)
# Output: ['Natural', 'language', 'processing', 'is', 'fun', '!']
```

### Example: Using Stopwords

```python
from nltk.corpus import stopwords

# No download required
english_stops = set(stopwords.words('english'))
print('the' in english_stops)  # True
```

### Example: Using POS Tagger

```python
from nltk.tag import pos_tag
from nltk.tokenize import word_tokenize

# Works without nltk.download()
text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text)
tags = pos_tag(tokens)
print(tags)
# Output: [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ...]
```

## Verification

To verify that the package is working correctly and data is being loaded from the pip package:

### 1. Check Package Installation

```bash
pip show nltk-extratokenizers
```

### 2. Verify Entry Points

```python
import importlib.metadata as m

print("nltk_data entry points:")
for ep in m.entry_points(group="nltk_data"):
    print(f"  {ep.name} -> {ep.value}")
```

Expected output should include:
```
punkt -> nltk_punkt.data
averaged_perceptron_tagger -> nltk_punkt.data
stopwords -> nltk_punkt.data
wordnet -> nltk_punkt.data
...
```

### 3. Verify Data Loading from Pip Package

```python
import nltk

# Check that punkt tokenizer is loaded from pip package
path = nltk.data.find("tokenizers/punkt/english.pickle")
print(f"Punkt tokenizer location: {path}")
# Should show a path containing 'nltk_punkt' or 'site-packages'
```

### 4. Test Tokenizer Functionality

```python
from nltk.tokenize import PunktSentenceTokenizer

try:
    tok = PunktSentenceTokenizer()
    result = tok.tokenize("Test sentence. Another one!")
    print("✓ Punkt tokenizer works correctly!")
    print(f"  Result: {result}")
except LookupError as e:
    print(f"✗ Error: {e}")
    print("  Make sure nltk-extratokenizers is installed and NLTK is patched.")
```

## Requirements

- Python 3.8 or higher
- NLTK (patched version from [this repository](https://github.com/Julien-ser/nltk_Group1_ENGG4450))
- setuptools >= 61.0

## How It Works

This package uses Python's entry point system to register NLTK data resources. The patched NLTK's `nltk.data.find()` function:

1. First checks for pip-installed packages via `nltk_data` entry points
2. Falls back to the standard NLTK data search paths if not found

This allows NLTK data to be distributed and installed via pip, making it easier to manage dependencies and deploy applications.

## Troubleshooting

### "LookupError: Resource not found"

If you get a `LookupError` when using NLTK resources:

1. **Verify the package is installed:**
   ```bash
   pip show nltk-extratokenizers
   ```

2. **Check that you're using the patched NLTK:**
   ```python
   import nltk
   print(nltk.__file__)
   # Should point to your local NLTK clone, not site-packages
   ```

3. **Verify entry points are registered:**
   ```python
   import importlib.metadata as m
   eps = list(m.entry_points(group="nltk_data"))
   print(f"Found {len(eps)} entry points")
   ```

### "Module not found: nltk_punkt"

If you see import errors, make sure:
- The package is installed: `pip install nltk-extratokenizers`
- You're using the correct Python environment

## Contributing

Contributions are welcome! Please visit the [GitHub repository](https://github.com/Julien-ser/nltk_Group1_ENGG4450) to report issues or submit pull requests.

## License

This package is distributed under the Apache License 2.0, same as NLTK.

## Related Projects

- [NLTK](https://www.nltk.org/) - Natural Language Toolkit
- [NLTK GitHub](https://github.com/nltk/nltk) - Official NLTK repository

## Citation

If you use this package in your research, please cite:

```
Bird, Steven, Edward Loper and Ewan Klein (2009).
Natural Language Processing with Python. O'Reilly Media Inc.
```

---

**Package Name:** `nltk-extratokenizers`  
**Version:** 0.1.0  
**Repository:** https://github.com/Julien-ser/nltk_Group1_ENGG4450
