Metadata-Version: 2.4
Name: moneycontrol-datascraper
Version: 0.1.1
Summary: Polite Moneycontrol data collector for trading research. Respects robots.txt; no anti-bot bypass.
Author: moneycontrol_scraper_safe contributors
License: MIT
Project-URL: Homepage, https://github.com/your-org/moneycontrol_scraper_safe
Project-URL: Issues, https://github.com/your-org/moneycontrol_scraper_safe/issues
Keywords: moneycontrol,scraper,rss,finance,stocks,ipo,mutual-funds,news,research,robots-txt
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Requires-Dist: beautifulsoup4>=4.11
Requires-Dist: pandas>=1.5
Requires-Dist: feedparser>=6.0
Requires-Dist: python-dateutil>=2.8
Requires-Dist: lxml>=4.9
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4.0; extra == "dev"
Dynamic: license-file

# moneycontrol_scraper_safe

A **polite** Moneycontrol data collector intended for trading research.

## Principles

- **Respects `robots.txt`** for every request via `urllib.robotparser`.
- **No bypassing** of Cloudflare, login walls, paywalls, captchas, or rate limits.
- **Prefers official RSS feeds** over HTML scraping.
- **Adds delay** between requests (default 3s, honors any declared `Crawl-delay`).
- **Stores both raw and clean** data so research is reproducible.
- Outputs **pandas DataFrames** plus **JSON / CSV** files.

If a page is disallowed by `robots.txt`, or the server returns 401 / 403 / 429,
the scraper **stops** and does not retry with tricks.

## Install

```bash
pip install -r requirements.txt
```

## Usage

```bash
# Fetch all configured RSS feeds and write clean output
python main.py rss --sentiment

# Limit to specific categories
python main.py rss --category latest_news ipo mutual_funds

# Fetch the public homepage / markets page (only if robots.txt allows)
python main.py html

# Both
python main.py all --sentiment
```

Set a contact email so site operators can reach you:

```bash
export MC_CONTACT="you@example.com"
```

## Modules

| Module | Purpose |
| --- | --- |
| `config.py` | URLs, feed list, delays, paths |
| `robots_checker.py` | `is_allowed()` / `crawl_delay()` via `urllib.robotparser` |
| `rss_collector.py` | Polite feed fetcher using `requests` + `feedparser` |
| `html_collector.py` | Polite public-page fetcher (no auth, no bypass) |
| `parser.py` | RSS + HTML → clean pandas DataFrames |
| `sentiment.py` | Lightweight finance-flavoured sentiment scoring |
| `storage.py` | Save raw payloads (`data/raw/`) and clean output (`data/clean/`) |
| `cli.py` | `argparse` CLI: `rss`, `html`, `all` |

## Data layout

```
data/
├── raw/      # JSON payloads of feeds, raw HTML pages
└── clean/    # CSV + JSON of parsed/clean DataFrames
```

## Categories collected

Latest market news, stock news, IPO news, economy news, business news, commodity
news, personal finance news, mutual fund news, results / company headlines, and
a best-effort market overview from the public homepage / markets page (only if
`robots.txt` permits).

## Disclaimer

This project is for personal research. Use the data in compliance with
Moneycontrol's terms of service. Do not redistribute scraped content.
