Metadata-Version: 2.4
Name: apcloudy-pipeline
Version: 0.1.9
Summary: Scrapy pipeline & extensions for AP Cloudy (logs, stats, requests, items)
Home-page: https://github.com/fawadss1/apcloudy-pipeline
Author: Fawad Ali
Author-email: fawadstar6@gmail.com
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Framework :: Scrapy
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENCE
Requires-Dist: requests>=2.30.0
Requires-Dist: w3lib<3.0.0,>=1.22.0
Requires-Dist: itemadapter>=0.8.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# APCloudy Pipeline

Scrapy integration that sends **items, requests, logs, and stats** to AP Cloudy with HMAC auth and full job log capture.

## Install

```bash
pip install apcloudy-pipeline
```

## Settings

```python
APCLOUDY_URL = "https://your-api.com"
APCLOUDY_API_KEY = "your_public_api_key"
APCLOUDY_SECRET_KEY = "your_secret_key"
JOB_ID = 123
APCLOUDY_BATCH_SIZE = 50  # optional

ITEM_PIPELINES = {
    "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
}
DOWNLOADER_MIDDLEWARES = {
    "apcloudy_pipeline.middleware.APCloudyErrorMiddleware": 50,
}
EXTENSIONS = {
    "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 100,
    "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 100,
    "apcloudy_pipeline.extensions.APCloudyStatsExtension": 100,
}
```

Log level follows Scrapy `LOG_LEVEL`.

## Payload

```json
{
  "job_id": "123",
  "data": {
    "requests": [{ "url": "...", "status_code": 200, "success": true, "...": "..." }],
    "items": [{ "title": "...", "_ts": 1753358220 }],
    "logs": [{
      "index": 0,
      "time": "2026-07-01 13:02:55",
      "timestamp": 1751374975.12,
      "level": "INFO",
      "logger": "scrapy.core.engine",
      "message": "Spider opened",
      "exception": null
    }],
    "stats": { "item_scraped_count": 1, "finish_reason": "finished" }
  }
}
```

Logs cover startup → crawl → stats dump → closed (plus `print()` spider lines).

## Auth

```
X-API-KEY / X-TIMESTAMP / X-SIGNATURE
POST {APCLOUDY_URL}/api/webhook/consume
signature = HMAC-SHA256(secret, timestamp + "." + body)
```

## Notes

- **Yield items** from callbacks (not only `return` inside a generator).
- Data flushes on batch size, every 10s, and on engine/process stop.
- Requires Python 3.8+, Scrapy 2.0+, `requests`, `w3lib`, `itemadapter`.

## License

MIT
