Metadata-Version: 2.4
Name: business-standard-scraper-safe
Version: 0.1.1
Summary: Polite, robots.txt-respecting scraper for public Business Standard news, with sentiment + stock mapping for Indian-market research.
Author: BharatSwingAI
License: MIT License
        
        Copyright (c) 2026 BharatSwingAI
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/your-org/business-standard-scraper-safe
Project-URL: Issues, https://github.com/your-org/business-standard-scraper-safe/issues
Project-URL: Source, https://github.com/your-org/business-standard-scraper-safe
Keywords: business-standard,news-scraper,indian-stock-market,sentiment-analysis,swing-trading,nse,finance,rss,sitemap
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Text Processing :: Markup :: XML
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: pandas>=2.0
Requires-Dist: python-dateutil>=2.8
Requires-Dist: lxml>=4.9
Requires-Dist: urllib3>=2.0
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# business-standard-scraper-safe

[![PyPI version](https://img.shields.io/pypi/v/business-standard-scraper-safe.svg)](https://pypi.org/project/business-standard-scraper-safe/)
[![Python versions](https://img.shields.io/pypi/pyversions/business-standard-scraper-safe.svg)](https://pypi.org/project/business-standard-scraper-safe/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

A polite, **robots.txt-respecting** Python library that collects public Business Standard news for Indian market sentiment and swing-trading research.

This package is a **data collector only**. Prediction logic belongs in a separate package (e.g. `swing_prediction_engine`).

---

## Features

- Sitemap **+ RSS** ingestion, deduped by URL
- `robots.txt` enforcement on every request
- Polite client: configurable delay, retry/backoff, timeout, structured logging
- Article parsing: title, summary, body, published date
- **SHA256 content-hash deduplication**
- **Stock-symbol mapping** against an NSE master list
- **Structured sentiment** output: `sentiment`, `impact`, `confidence`, `reason`
- Storage: CSV snapshots + **SQLite upsert** (PostgreSQL-ready schema)
- CLI (`bs-scraper`) and importable Python API

## Install

```bash
pip install business-standard-scraper-safe
```

## Quick start

### CLI

```bash
bs-scraper --limit 100 --category markets
bs-scraper --limit 50  --category all --db data/business_standard.db
```

CLI flags:

| Flag         | Default                              | Description                                               |
|--------------|--------------------------------------|-----------------------------------------------------------|
| `--limit`    | `50`                                 | Max URLs to process                                       |
| `--category` | `all`                                | `markets`, `companies`, `economy`, `finance`, `ipo`, `commodities`, `industry`, `technology`, `india`, `world`, `opinion`, or `all` |
| `--db`       | `data/business_standard.db`          | SQLite path                                               |
| `--log-file` | `business_standard_scraper.log`      | Log path                                                  |

### Python API

```python
from bs_scraper import run, fetch_all_urls, enrich_articles, analyze, setup_logging

setup_logging()

# Full pipeline (sitemap+RSS → enrich → dedup → sentiment → stocks → CSV+SQLite)
df = run(limit=100, category="markets")

# Or use the building blocks
urls_df = fetch_all_urls(limit=200)
articles_df = enrich_articles(urls_df.head(50))
print(analyze("Reliance posts record profit and beats estimates"))
```

## Pipeline

```
Sitemap + RSS
   ↓
Filter by robots.txt
   ↓
Fetch public article pages (delay + retry)
   ↓
Parse title / summary / body / date
   ↓
Deduplicate (sha256 content hash)
   ↓
Map stock symbols (NSE)
   ↓
Sentiment + impact + confidence + reason
   ↓
SQLite upsert + CSV snapshot
   ↓
Consumed by your prediction engine
```

## Output schema

CSV files land in `data/clean/business_standard_news_<timestamp>.csv`.
SQLite table `news_clean` (primary key `news_id`, upserted on conflict):

| Column | Description |
|---|---|
| `news_id` | sha256(`title` + `url`) |
| `source` | `business_standard` |
| `category` | `markets`, `companies`, … |
| `title`, `summary`, `article_text` | parsed text |
| `url`, `published_at` | source + ISO 8601 date |
| `sentiment` | `Bullish` / `Bearish` / `Neutral` |
| `impact` | `High` / `Medium` / `Low` |
| `confidence` | 0–95 |
| `reason` | short explanation of detected cues |
| `sentiment_score` | raw integer (positive − negative cues) |
| `matched_stocks` | comma-separated NSE symbols |
| `inserted_at` | ISO 8601 |

## Sentiment payload

```json
{
  "sentiment": "Bullish",
  "impact": "High",
  "confidence": 82,
  "reason": "Positive cues: order, growth, wins"
}
```

Currently Level 1 (keyword-based). Upgrade path: FinBERT → LLM sentiment → market-impact classifier. Swap the body of `bs_scraper/sentiment.py::analyze` to upgrade.

## Stock mapping

Ships with a small NSE sample. Drop a CSV at `data/stocks_master.csv` with columns `symbol,name` to use a full universe:

```csv
symbol,name
RELIANCE.NS,Reliance Industries
TCS.NS,Tata Consultancy Services
```

## Rules of engagement

This library deliberately **does not**:

- ignore `robots.txt`
- scrape disallowed paths (`/api/`, `/_next/`, `/static/`, `/assets/`, `/search?`)
- bypass paywalls, login, captcha, Cloudflare, or rate limits
- spoof browsers or rotate identities

If a path or feed is blocked, it is logged and skipped.

## Project layout

```
business_standard_scraper_safe/
├── pyproject.toml
├── LICENSE
├── MANIFEST.in
├── README.md
├── PUBLISHING.md
├── main.py
└── bs_scraper/
    ├── __init__.py
    ├── cli.py
    ├── pipeline.py
    ├── config.py
    ├── http_client.py
    ├── robots_checker.py
    ├── sitemap_collector.py
    ├── rss_collector.py
    ├── html_collector.py
    ├── enrichment.py
    ├── parser.py
    ├── sentiment.py
    ├── stocks.py
    ├── storage.py
    └── logging_setup.py
```

## Publishing

See [PUBLISHING.md](PUBLISHING.md). **Never paste API tokens into chat, commits, or logs.** Use environment variables (`TWINE_USERNAME=__token__`, `TWINE_PASSWORD=pypi-…`) or a chmod-600 `~/.pypirc`.

## Roadmap

- Full NSE master list at `data/stocks_master.csv`
- FinBERT / LLM sentiment
- Source reliability score
- FastAPI endpoints
- Scheduler (cron / APScheduler)
- Combine with Moneycontrol, ET, LiveMint
- PostgreSQL storage with `UNIQUE(news_id) ON CONFLICT DO UPDATE`

## License

[MIT](LICENSE)
