Metadata-Version: 2.4
Name: news-fetch
Version: 1.0.0
Summary: Python news scraper & article extractor — fetch news URLs, extract title/text/authors/date/image with confidence scores. Bulk scraping, proxies, RSS/sitemap discovery. No API key.
Author-email: M Santhosh Kumar <santhoshse7en@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/santhoshse7en/news-fetch
Project-URL: Repository, https://github.com/santhoshse7en/news-fetch
Project-URL: Issues, https://github.com/santhoshse7en/news-fetch/issues
Project-URL: Changelog, https://github.com/santhoshse7en/news-fetch/blob/master/CHANGELOG.md
Project-URL: Documentation, https://santhoshse7en.github.io/newsfetch_doc/
Keywords: news scraper,news extractor,article extractor,news-fetch,python news scraper,web scraping,news crawler,json-ld,opengraph,rss,sitemap,bulk scraping,proxy,trafilatura alternative,newspaper3k alternative,newspaper4k alternative,news-please alternative,readability,without api,news api free
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: lxml>=5.0
Requires-Dist: cssselect>=1.2
Requires-Dist: requests>=2.32
Requires-Dist: python-dateutil>=2.9
Provides-Extra: async
Requires-Dist: httpx>=0.27; extra == "async"
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == "browser"
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: setuptools; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

[![PyPI version](https://img.shields.io/pypi/v/news-fetch.svg?style=flat-square)](https://pypi.org/project/news-fetch)
[![Downloads](https://pepy.tech/badge/news-fetch/month)](https://pepy.tech/project/news-fetch)
[![Python versions](https://img.shields.io/pypi/pyversions/news-fetch.svg?style=flat-square)](https://pypi.org/project/news-fetch)
[![License](https://img.shields.io/pypi/l/news-fetch.svg?style=flat-square)](https://pypi.org/project/news-fetch/)
[![CI](https://img.shields.io/github/actions/workflow/status/santhoshse7en/news-fetch/ci.yml?style=flat-square)](https://github.com/santhoshse7en/news-fetch/actions)

# news-fetch

**Python news scraper & article extractor** — extract title, text, authors, date, image, and publisher from any news URL. No API key. Confidence scores included.

> **Fetch news. Know why it worked.**

```bash
pip install news-fetch
```

```python
from newsfetch import fetch

article = fetch("https://www.thehindu.com/...")
print(article.title)
print(article.text)
print(article.authors)
print(article.published_at)
print(article.image)
print(article.confidence.overall)   # 0.0–1.0
print(article.content_source)       # e.g. json-ld.articleBody
```

```bash
news-fetch https://example.com/article
news-fetch https://example.com/article --json
news-fetch batch urls.txt -o articles.jsonl
```

---

## Why news-fetch?

A **lightweight alternative** to newspaper3k / newspaper4k / trafilatura wrappers — with its **own extraction engine**, **confidence scores**, and **bulk + proxy** support.

| Feature | news-fetch |
| --- | :---: |
| News article extraction (title, body, authors, date, image) | ✅ |
| Confidence scores + extraction provenance | ✅ |
| Bulk scraping (`fetch_many` / `fetch_iter` / CLI JSONL) | ✅ |
| Proxy + proxy rotation for thousands of URLs | ✅ |
| RSS / sitemap article discovery | ✅ |
| Async (`pip install news-fetch[async]`) | ✅ |
| Optional browser render (`pip install news-fetch[browser]`) | ✅ |
| Disk cache + robots.txt respect | ✅ |
| No API key / no account | ✅ |
| Small deps (`lxml`, `requests`, `python-dateutil`, `cssselect`) | ✅ |

---

## Install

```bash
pip install news-fetch
pip install news-fetch[async]     # httpx async fetch
pip install news-fetch[browser]   # Playwright fallback (then: playwright install chromium)
```

**Requirements:** Python 3.10+

---

## Quick start

### Single URL

```python
from newsfetch import fetch

article = fetch(url)
print(article.title, article.text, article.confidence.overall)
```

### From HTML (no network)

```python
from newsfetch import extract

article = extract(html_bytes, url="https://example.com/story")
```

### Bulk scraping + proxies

```python
from newsfetch import fetch_many, fetch_iter

results = fetch_many(
    urls,
    max_workers=20,
    proxies=["http://user:pass@p1:8080", "http://user:pass@p2:8080"],
    request_delay=0.05,
)

for url, article in fetch_iter(urls, max_workers=16):
    if article:
        print(article.title)
```

### Strict mode (production pipelines)

```python
from newsfetch import fetch, LowConfidenceExtractionError

try:
    article = fetch(url, strict=True)
except LowConfidenceExtractionError as e:
    print(e.failed_fields, e.confidence.overall)
```

### Discovery (RSS / sitemaps)

```python
from newsfetch import discover

for item in discover("https://www.bbc.com", limit=10):
    print(item["url"], item.get("title"))
```

### Async

```python
from newsfetch import fetch_async, fetch_many_async

article = await fetch_async(url)
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)
```

### CLI

```bash
news-fetch https://example.com/article
news-fetch get URL --json
news-fetch batch urls.txt -o out.jsonl --workers 20
news-fetch discover https://www.theguardian.com --limit 10
```

### Cache / robots / browser

```python
from newsfetch import fetch, Config, NewsFetcher

fetch(url, cache=True, respect_robots=True)
fetch(url, render=True)                      # needs news-fetch[browser]
fetch(url, browser_fallback=True)            # retry with Playwright if confidence is low
```

### Custom strategy plugin

```python
from newsfetch import NewsFetcher, CallableStrategy
from newsfetch.strategies.base import Candidate

def my_strategy(doc):
    return {"title": [Candidate("Custom", "plugin.custom", 0.99)]}

fetcher = NewsFetcher()
fetcher.register_strategy(CallableStrategy("custom", my_strategy))
```

---

## Article fields

`url` · `canonical_url` · `title` · `description` · `text` · `authors` · `published_at` · `modified_at` · `publisher` · `language` · `image` · `keywords` · `section` · `summary` · `word_count` · `reading_time_minutes` · `page_type` · `is_article` · `confidence` · `extraction` · `sources`

```python
article.to_dict()
article.to_json()
```

---

## Links

- **PyPI:** https://pypi.org/project/news-fetch/
- **Docs:** https://santhoshse7en.github.io/newsfetch_doc/
- **GitHub:** https://github.com/santhoshse7en/news-fetch
- **Issues:** https://github.com/santhoshse7en/news-fetch/issues
- **Changelog:** [CHANGELOG.md](CHANGELOG.md)

MIT License · Built for developers who need reliable Python news scraping without an API.
