Metadata-Version: 2.4
Name: crypto_exchange_news_crawler
Version: 0.1.8
Summary: Cryptocurrency exchange announcement news crawler for major crypto exchanges
Home-page: https://github.com/lowweihong/crypto_exchange_news_crawler
Author: lowweihong
Author-email: lowweihong14@gmail.com
Project-URL: Bug Reports, https://github.com/lowweihong/crypto-exchange-news-crawler/issues
Project-URL: Source, https://github.com/lowweihong/crypto-exchange-news-crawler
Project-URL: Documentation, https://github.com/lowweihong/crypto-exchange-news-crawler#readme
Keywords: cryptocurrency,crypto,exchange,news,crawler,scraper,bybit,binance,bitget,bitfinex,xt,okx,bingx,kraken,crypto.com,mexc,deepcoin,kucoin,announcement,trading,bot,api,scrapy,web-scraping,fintech,blockchain,defi,trading-bot,crypto-news
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Office/Business :: Financial
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: Scrapy>=2.11.0
Requires-Dist: scrapy-playwright>=0.0.25
Requires-Dist: playwright>=1.40.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Crypto Exchange News Crawler 🚀

A powerful and easy-to-use Python package for scraping cryptocurrency exchange announcements from major exchanges.

## 🎯 Features

- **Multi-Exchange Support**: Scrape from 12 major crypto exchanges
- **Multiple Output Formats**: JSON, CSV, and XML support
- **Structured Data**: Clean, standardized output format
- **Rate Limiting**: Built-in delays to respect exchange servers
- **Extensible**: Easy to add new exchanges

## 📦 Installation Options

### Option 1: Direct Usage

```bash
git clone https://github.com/lowweihong/crypto-exchange-news-crawler.git
cd crypto-exchange-news-crawler
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
playwright install

scrapy crawl bybit -o output.json
```

### Option 2: Install from PyPI

```bash
pip install crypto-exchange-news-crawler
playwright install

## directly use proxy and uncomment DOWNLOADER_MIDDLEWARES
crypto-news crawl binance -o binance.json

crypto-news crawl bybit -s DOWNLOADER_MIDDLEWARES='{"crypto_exchange_news.middlewares.MyProxyMiddleware": 610}' -s PROXY_LIST="http://proxy1:port,http://proxy2:port"
```

### Supported Exchanges

| Exchange  | Status |
|-----------|--------|
| Bybit     | ✅ |
| Binance   | ✅ |
| OKX       | ✅ |
| Bitget    | ✅ |
| BingX     | ✅ |
| Kraken    | ✅ |
| Bitfinex  | ✅ |
| XT        | ✅ |
| Crypto.com| ✅ |
| MEXC      | ✅ |
| Deepcoin  | ✅ |
| Kucoin    | ✅ |

```
Available options : ["bybit", "binance", "okx", "bitget", "bitfinex", "xt", "bingx", 'kraken', 'cryptocom', 'mexc', 'deepcoin', 'kucoin']
```
#

## 📊 Output Format

Each scraped announcement includes:

```json
{
    "news_id": "unique_identifier",
    "title": "Announcement title",
    "desc": "Announcement description",
    "url": "Full URL to announcement",
    "category_str": "Category (e.g., latest_activities, new_crypto)",
    "exchange": "Exchange name",
    "announced_at_timestamp": 1749235200,
    "timestamp": 1749232733
}
```

## ⚙️ Configuration

Key settings in `settings.py`:

- `MAX_PAGE`: Maximum number of pages to crawl (default: 2)
- `DOWNLOAD_DELAY`: Delay between requests in seconds (default: 3)
- `CONCURRENT_REQUESTS`: Number of concurrent requests (default: 8)
- `USER_AGENT`: List of user agents for rotation
- `PROXY_LIST`: Fill the list with your proxy list and remember also to open uncomment the DOWNLOADER_MIDDLEWARES part to use the proxy middleware
- `PLAYWRIGHT_LAUNCH_OPTIONS`: Browser configuration for Playwright spiders

### Custom Settings

You can override settings from the command line:

```bash
scrapy crawl bitget -s MAX_PAGE=5 -s DOWNLOAD_DELAY=2
```

## 🔧 Technical Requirements

- Python 3.7+
- Scrapy 2.11.0+
- Playwright (for Bitget spider)
- Chromium browser (automatically installed with Playwright)

## 🌐 Exchange URLs

Direct links to announcement pages:

| Exchange | Announcement URL |
|----------|------------------|
| **Binance** | https://www.binance.com/en/support/announcement |
| **OKX** | https://www.okx.com/help/category/announcements |
| **Bybit** | https://announcements.bybit.com/en/?category=&page=1 |
| **Bitget** | https://www.bitget.com/support/sections/12508313443483 |
| **BingX** | https://bingx.com/en/support/notice-center/ |
| **Kraken** | https://blog.kraken.com/category/product |
| **XT** | https://xtsupport.zendesk.com/hc/en-us/categories/10304894611993-Important-Announcements |
| **Bitfinex** | https://www.bitfinex.com/posts/ |
| **Crypto.com** | https://crypto.com/exchange/announcements |
| **MEXC** | https://www.mexc.com/support/categories/360000254192 |
| **Deepcoin** | https://support.deepcoin.online/hc/en-001/categories/360003875752-Important-Announcements |
| **Kucoin** | https://www.kucoin.com/announcement |



## ⚖️ Legal & Ethical Usage

This crawler is designed for educational and research purposes. Please ensure you comply with:

- Applicable data protection laws
- Fair use guidelines

Always use the crawler responsibly and consider the impact on the target servers.

## 🤝 Contributing

Contributions welcome! Areas for improvement:
- Add support for more exchanges (Huobi, KuCoin, Gateio, etc.)
- Implement real-time WebSocket feeds
- Add telegram/discord notification integrations
- Improve data parsing and categorization

## Support

For issues, questions, or contributions, please create an issue in the repository.
