Metadata-Version: 2.4
Name: apcloudy-pipeline
Version: 0.1.7
Summary: Scrapy pipeline & extensions for AP Cloudy (logs, stats, requests, items)
Home-page: https://github.com/fawadss1/apcloudy-pipeline
Author: Fawad
Author-email: fawadstar6@email.com
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Framework :: Scrapy
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENCE
Requires-Dist: requests>=2.30.0
Requires-Dist: w3lib<3.0.0,>=1.22.0
Requires-Dist: itemadapter>=0.8.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# APCloudy Pipeline

A Scrapy integration that sends **items, requests, logs, and spider statistics** to your AP Cloudy backend using **unified batch processing** and **HMAC-SHA256 authentication**.

---

## ✨ Features

- 📦 **Unified Data Sending** - All data (items, requests, logs, stats) sent in a single API call
- 🚀 **Batch Processing** - Automatic batching every 10 items or on spider close
- 🌐 **Complete Request Tracking** - Logs both successful and failed requests with detailed error information
- 📊 **Spider Statistics** - Comprehensive spider performance metrics
- 🧾 **Log Forwarding** - Captures spider, user, and Scrapy internal logs
- 🔐 **Secure Authentication** - HMAC-SHA256 signature-based API communication
- ⚡ **High Performance** - Thread-safe data collection with minimal overhead
- 🎯 **Zero Configuration** - Works out of the box with sensible defaults

---

## 📦 Installation

```bash
pip install apcloudy-pipeline
```

---

## ⚙️ Configuration

Add these settings to your Scrapy `settings.py` (or spider `custom_settings`):

```python
# Required: API credentials
APCLOUDY_URL = "https://your-api.com"  # Base URL (webhook path added automatically)
APCLOUDY_API_KEY = "your_public_api_key"
APCLOUDY_SECRET_KEY = "your_secret_key"
JOB_ID = 123  # Can also be passed via spider args: -a JOB_ID=456

# Optional: Batch size (default: 50)
APCLOUDY_BATCH_SIZE = 50

# Required: Item Pipeline
ITEM_PIPELINES = {
    "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
}

# Required: Error Middleware (failed requests)
DOWNLOADER_MIDDLEWARES = {
    "apcloudy_pipeline.middleware.APCloudyErrorMiddleware": 50,
}

# Required: Extensions (requests, logs, stats)
EXTENSIONS = {
    "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 100,
    "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 100,
    "apcloudy_pipeline.extensions.APCloudyStatsExtension": 100,
}
```

---

## 🏗️ Architecture

### Data Flow

```
Spider execution
       │
       ├── Requests ──┐
       ├── Items ─────┼──► DataCollector (thread-safe)
       ├── Logs ──────┤           │
       └── Stats ─────┘           ▼
                          Batch trigger?
                          • size >= APCLOUDY_BATCH_SIZE
                          • every 10 seconds
                          • spider closes
                                  │
                                  ▼
                          APCloudyClient (HMAC)
                                  │
                                  ▼
                     POST /api/webhook/consume
```

---

## 📡 API Payload Structure

All data is sent in a single unified payload:

```json
{
  "job_id": "123",
  "data": {
    "requests": [
      {
        "url": "https://example.com/product",
        "method": "GET",
        "status_code": 200,
        "response_time": 1.23,
        "fingerprint": "f3045685b89f920b3faefc7d3df2d3c88bdab393",
        "error": null,
        "success": true
      }
    ],
    "items": [
      {
        "title": "Product Name",
        "price": "99.99",
        "url": "https://example.com/product",
        "_ts": 1753358220
      }
    ],
    "logs": [
      {
        "level": "INFO",
        "message": "Spider started",
        "exception": null
      }
    ],
    "stats": {
      "item_scraped_count": 1,
      "response_received_count": 1,
      "finish_time": "2026-07-24T13:08:29.571736+00:00",
      "finish_reason": "finished"
    }
  }
}
```

Each scraped item includes `_ts` — the Unix timestamp (seconds) when the pipeline collected it.

---

## Components

| Component                  | Role                                                |
|----------------------------|-----------------------------------------------------|
| `APCloudyItemPipeline`     | Collects items (adds `_ts`), batches and sends data |
| `APCloudyRequestLogger`    | Logs successful HTTP responses                      |
| `APCloudyErrorMiddleware`  | Logs failed requests / exceptions                   |
| `APCloudyLoggingExtension` | Forwards Python/Scrapy logs                         |
| `APCloudyStatsExtension`   | Collects spider stats on close                      |
| `DataCollector`            | Thread-safe shared buffer                           |
| `APCloudyClient`           | HMAC-signed HTTP client                             |

---

## Authentication

```
X-API-KEY: {your_public_key}
X-TIMESTAMP: {unix_timestamp}
X-SIGNATURE: {hmac_sha256(secret_key, timestamp + "." + json_body)}
Content-Type: application/json
```

```python
message = f"{timestamp}.{json_body}"
signature = HMAC_SHA256(secret_key, message)
```

---

## Requirements

- Python 3.8+
- Scrapy 2.0+
- `requests`, `w3lib`, `itemadapter`

---

## Advanced Configuration

### Batch size

```python
APCLOUDY_BATCH_SIZE = 50   # default
APCLOUDY_BATCH_SIZE = 1    # send as soon as possible
```

Data is also flushed every **10 seconds** and always on **spider close**.

### Job ID via spider args

```bash
scrapy crawl myspider -a JOB_ID=456
```

### Backend endpoint

```
POST {APCLOUDY_URL}/api/webhook/consume
```

Example: `https://your-api.com` → `https://your-api.com/api/webhook/consume`

---

## Example Spider

**Important:** use `yield` for items. If the callback also `yield`s Requests, a trailing `return item` is ignored by Python/Scrapy.

```python
import scrapy


class MySpider(scrapy.Spider):
    name = "myspider"

    custom_settings = {
        "APCLOUDY_URL": "https://your-api.com",
        "APCLOUDY_API_KEY": "your_public_api_key",
        "APCLOUDY_SECRET_KEY": "your_secret_key",
        "JOB_ID": 123,
        "ITEM_PIPELINES": {
            "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
        },
        "DOWNLOADER_MIDDLEWARES": {
            "apcloudy_pipeline.middleware.APCloudyErrorMiddleware": 50,
        },
        "EXTENSIONS": {
            "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 100,
            "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 100,
            "apcloudy_pipeline.extensions.APCloudyStatsExtension": 100,
        },
    }

    def start_requests(self):
        for url in ["https://example.com/page1", "https://example.com/page2"]:
            yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "price": response.css(".price::text").get(),
            "url": response.url,
        }
```

No extra spider logic is required for AP Cloudy — enable the pipeline/extensions and yield items.

---

## Troubleshooting

### Items are always empty / `item_scraped_count` missing

1. Confirm the callback **`yield`s** the item (not only `return` inside a generator).
2. Confirm `APCloudyItemPipeline` is in `ITEM_PIPELINES`.
3. Check spider logs for `Failed to send APCloudy batch`.

### Data not being sent

1. Verify `APCLOUDY_URL`, `APCLOUDY_API_KEY`, `APCLOUDY_SECRET_KEY`, and `JOB_ID`.
2. Ensure pipeline, middleware, and extensions are enabled.
3. Check network access to `{APCLOUDY_URL}/api/webhook/consume`.

### Failed requests not logged

1. Add `APCloudyErrorMiddleware` to `DOWNLOADER_MIDDLEWARES`.
2. Use a low priority (e.g. `50`) so it sees exceptions early.

### Stats missing

1. Enable `APCloudyStatsExtension`.
2. Stats are attached on spider close.

---

## Changelog

### 0.1.7

- Add `_ts` (Unix timestamp) to every item before send
- Use `ItemAdapter` for dict / Item / dataclass / attrs items
- Safer JSON serialization for item fields (`datetime`, `Decimal`, etc.)
- Requeue batches on send failure instead of silently dropping them
- Periodic flush every 10 seconds
- Docs: correct default batch size (50), `yield` guidance, troubleshooting

### 0.1.6

- Previous stable release

---

## 🤝 Contributing

MIT

---

## 📧 Support

Pull requests are welcome.

---

## Support

Open an issue on [GitHub](https://github.com/fawadss1/apcloudy-pipeline).
