Metadata-Version: 2.4
Name: livemint-scraper-safe
Version: 0.1.1
Summary: Ethical LiveMint news collector (RSS-first, robots-aware) that produces a clean NLP dataset for sentiment training and swing-trading research.
Author: BharatSwingAI
License: MIT License
        
        Copyright (c) 2026 BharatSwingAI
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/your-org/livemint-scraper-safe
Project-URL: Issues, https://github.com/your-org/livemint-scraper-safe/issues
Keywords: livemint,news,rss,nlp,sentiment,finance,swing-trading,india
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: pandas>=2.0
Requires-Dist: feedparser>=6.0
Requires-Dist: python-dateutil>=2.8
Requires-Dist: lxml>=4.9
Requires-Dist: urllib3>=1.26
Dynamic: license-file

# livemint-scraper-safe

[![PyPI version](https://img.shields.io/pypi/v/livemint-scraper-safe.svg)](https://pypi.org/project/livemint-scraper-safe/)
[![Python versions](https://img.shields.io/pypi/pyversions/livemint-scraper-safe.svg)](https://pypi.org/project/livemint-scraper-safe/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

A safe, ethical Python library for collecting public LiveMint financial and
news data — designed as a **clean NLP dataset builder** for sentiment training
and swing-trading research.

**Live on PyPI:** https://pypi.org/project/livemint-scraper-safe/

## Quick start

```bash
pip install livemint-scraper-safe

# CLI
livemint-scraper --limit 10

# Python
python -c "from livemint_scraper import run_pipeline; print(run_pipeline(article_limit=5).head())"
```

> This library does not produce trading signals on its own. Combine its
> sentiment output with technicals, volume, sector strength, and risk/reward
> in your trading engine.
> Suggested mix: **Technical 40% | News Sentiment 25% | Volume 15% | Sector 10% | R:R 10%**.

## Principles

- Prefers official RSS feeds.
- Respects `robots.txt`.
- Does not bypass login, paywall, captcha, Cloudflare, or rate limits.
- Adds delay, retry, timeout, and logging.
- Saves raw and clean data to CSV and SQLite.
- Returns pandas DataFrames.

## Pipeline

```
LiveMint RSS
   ↓
Deduplicate (md5 of title+link)
   ↓
Fetch Article Text (robots-aware)
   ↓
Normalize Date (dateutil)
   ↓
Map Stock Symbols (NSE/BSE tickers)
   ↓
Sentiment + Impact + Confidence
   ↓
Store in CSV + SQLite
```

## Install

From PyPI:

```bash
pip install livemint-scraper-safe
```

From source (for development):

```bash
git clone <your-repo-url>
cd livemint_scraper_safe
pip install -r requirements.txt
```

## Run

After install:

```bash
livemint-scraper --limit 15
```

From source:

```bash
python main.py
```

Outputs:
- `data/clean/livemint_news_<timestamp>.csv`
- `data/livemint.db` (SQLite, table `news_clean`)
- `livemint_scraper.log`

## Programmatic use

```python
import pandas as pd
from livemint_scraper import run_pipeline

stocks = pd.DataFrame([
    {"symbol": "RELIANCE.NS", "name": "Reliance Industries"},
    {"symbol": "TCS.NS", "name": "Tata Consultancy Services"},
])

df = run_pipeline(stocks_df=stocks, fetch_articles=True, article_limit=20)
print(df.head())
```

## Sentiment engine

`sentiment.analyze(text)` returns:

```json
{
  "sentiment": "Bullish",
  "sentiment_score": 3,
  "impact": "High",
  "confidence": 82,
  "reason": "Detected bullish cues with high impact words."
}
```

Upgrade path (kept behind the same `analyze()` interface so callers don't change):
1. Keyword sentiment (current)
2. FinBERT
3. LLM-based sentiment
4. Market impact classifier

## Output schema

`news_id`, `source`, `category`, `title`, `summary`, `article_text`, `link`,
`published`, `published_at`, `fetched_at`, `company_symbols`,
`text_for_sentiment`, `sentiment_label`, `sentiment_score`, `impact_score`,
`confidence_score`, `sentiment_reason`.

## Folder structure

```
livemint_scraper_safe/
├── requirements.txt
├── main.py
├── README.md
├── livemint_scraper/
│   ├── __init__.py
│   ├── config.py
│   ├── http_client.py
│   ├── robots_checker.py
│   ├── rss_collector.py
│   ├── html_collector.py
│   ├── parser.py
│   ├── enrich.py
│   ├── dedup.py
│   ├── dates.py
│   ├── stocks.py
│   ├── sentiment.py
│   ├── storage.py
│   ├── logger.py
│   └── pipeline.py
└── data/
    ├── raw/
    └── clean/
```
