Metadata-Version: 2.4
Name: cnbctv18-scraper-safe
Version: 0.1.0
Summary: Polite, robots.txt-respecting CNBC-TV18 news collector for Indian-market sentiment and swing-trading research.
Author: BharatSwingAI Research
License: MIT
Project-URL: Homepage, https://github.com/bharatswingai/cnbctv18-scraper-safe
Project-URL: Issues, https://github.com/bharatswingai/cnbctv18-scraper-safe/issues
Keywords: cnbctv18,scraper,news,finance,indian-stock-market,sentiment,nlp,swing-trading,research
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: pandas>=2.0
Requires-Dist: python-dateutil>=2.8
Requires-Dist: lxml>=4.9
Requires-Dist: urllib3>=2.0
Dynamic: license-file

# cnbctv18_scraper_safe

A polite, **robots.txt-respecting** Python library for collecting public CNBC-TV18
news/financial content for **research and NLP sentiment training only**.

## What it does NOT do

- Does **not** bypass logins, paywalls, captchas, Cloudflare, or anti-bot systems.
- Does **not** ignore `robots.txt` or rate limits.
- Does **not** scrape personal data.

If `robots.txt` cannot be loaded or disallows a URL, the fetcher refuses (fail-closed).

## Install

```bash
pip install -r requirements.txt
```

## Quick start

```python
from cnbctv18_scraper import (
    collect_all_categories,
    parse_articles_dataframe,
    filter_valid_articles,
    add_content_hash, deduplicate,
    add_sentiment_column, add_impact_column,
    add_stock_symbols_column,
    save_dataframe, save_unique_to_sqlite,
)
import pandas as pd

urls     = collect_all_categories(["markets", "economy", "earnings"])
articles = parse_articles_dataframe(urls["url"].head(20).tolist())
articles = filter_valid_articles(articles)              # quality filter
articles = deduplicate(add_content_hash(articles))      # content-hash dedup
articles = add_sentiment_column(articles)               # +sentiment_*
articles = add_impact_column(articles)                  # +impact_score
# Optional ticker mapping (CSV with columns: symbol,name)
articles = add_stock_symbols_column(articles, pd.read_csv("stocks.csv"))

articles["news_id"] = articles["content_hash"]
save_dataframe(articles, "articles", fmt="csv")
save_unique_to_sqlite(articles)                         # upsert into SQLite
```

### CLI

```bash
python main.py --category markets --limit-per-category 20 --max-articles 100
python main.py --category all --max-articles 50 --stocks stocks.csv
python main.py --category economy --no-sqlite
```

CLI flags: `--category`, `--limit-per-category`, `--max-articles`, `--stocks`, `--db`, `--no-sqlite`.

### Pipeline

```
Category + RSS + Sitemap URLs
   → robots check (built into fetcher)
   → fetch allowed pages (delay + retry + timeout)
   → parse article
   → quality filter      (is_valid_article)
   → deduplicate         (content_hash)
   → stock mapping       (map_stock_symbols)
   → sentiment score     (add_sentiment_column)
   → impact score        (add_impact_column)
   → store CSV + SQLite  (save_unique_to_sqlite)
```

## Layout

```
cnbctv18_scraper_safe/
├── requirements.txt
├── main.py
├── README.md
├── cnbctv18_scraper/
│   ├── __init__.py
│   ├── config.py          # base URL, category map, headers, delays
│   ├── http_client.py     # requests session w/ retries + backoff
│   ├── robots_checker.py  # cached robots.txt enforcement
│   ├── html_collector.py  # polite single-page fetcher (BeautifulSoup)
│   ├── url_collector.py   # discover article URLs from category pages
│   ├── parser.py          # parse + quality-filter articles
│   ├── dedup.py           # content_hash + deduplicate
│   ├── sentiment.py       # sentiment + impact + stock-symbol mapping
│   └── storage.py         # CSV / Parquet / JSON + SQLite upsert
└── data/
    ├── raw/               # raw HTML dumps
    └── clean/             # CSV/Parquet/JSON outputs
```

## Categories

Configured in `cnbctv18_scraper/config.py` — includes `latest_news`, `markets`,
`stocks`, `economy`, `finance`, `business`, `earnings`, `technology`,
`real_estate`, `auto`, `telecom`, `aviation`, `infrastructure`, `healthcare`,
`agriculture`, `commodities`, `currency`, `expert_views`, `personal_finance`,
`videos`, and more.

## Settings (in `config.py`)

| Setting                   | Default                          |
| ------------------------- | -------------------------------- |
| `REQUEST_DELAY_SECONDS`   | `2`                              |
| `REQUEST_TIMEOUT_SECONDS` | `20`                             |
| `MAX_RETRIES`             | `3`                              |
| `RETRY_BACKOFF`           | `2`                              |
| `HEADERS["User-Agent"]`   | `BharatSwingAIResearchBot/1.0`   |

## Sentiment

`sentiment.py` ships a small finance lexicon (gain/loss/beat/miss/upgrade/...).
It's a **baseline only** — swap in FinBERT or a fine-tuned model for serious work.

## License / use

Research and educational use only. You are responsible for complying with
CNBC-TV18's Terms of Service and applicable law in your jurisdiction.
