Metadata-Version: 2.4
Name: argos-news
Version: 0.3.3
Summary: Automated Realtime Global Observer System - Python Client & API for Middle East News
Author-email: EGen Labs <contact@egenlabs.com>
License: MIT
Project-URL: Homepage, https://github.com/egenlabs/argos
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content :: News/Diary
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.24.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: server
Requires-Dist: fastapi>=0.100.0; extra == "server"
Requires-Dist: uvicorn>=0.23.0; extra == "server"
Requires-Dist: aiosqlite>=0.19.0; extra == "server"
Requires-Dist: scrapy>=2.11.0; extra == "server"
Requires-Dist: pydantic-settings>=2.0.0; extra == "server"
Requires-Dist: scrapling[fetchers]>=0.2; extra == "server"

# ARGOS — Automated Realtime Global Observer System

<p align="center">
  <img src="https://i.ibb.co/84zFPd18/logo.png" alt="ARGOS Logo" width="300" />
</p>

<p align="center">
  <img src="https://img.shields.io/badge/version-0.1.0-blue" alt="Version" />
  <img src="https://img.shields.io/badge/python-3.11+-3776AB?logo=python&logoColor=white" alt="Python" />
  <img src="https://img.shields.io/badge/framework-FastAPI-009688?logo=fastapi" alt="FastAPI" />
  <img src="https://img.shields.io/badge/scraper-Scrapy-60a839?logo=scrapy" alt="Scrapy" />
  <img src="https://img.shields.io/badge/storage-SQLite-003B57?logo=sqlite" alt="SQLite" />
  <img src="https://img.shields.io/badge/license-MIT-green" alt="License" />
</p>

<p align="center">
  A 3-layer automated news pipeline that continuously scrapes, enriches, and serves articles from 4 major Middle Eastern news networks — via a REST API and Python client library.
</p>

---

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [News Sources](#news-sources)
- [Quick Start](#quick-start)
- [Running Spiders](#running-spiders)
- [API Reference](#api-reference)
- [Scheduling](#scheduling)
- [Data Model](#data-model)
- [Configuration](#configuration)
- [Project Structure](#project-structure)

---

## Overview

ARGOS (*Argus Panoptes sees everything*) automates the full lifecycle of news collection from the Middle East:

- 🕷️ **Scrapes** articles from 4 sources using Scrapy spiders
- 🧠 **Enriches** them with editorial metadata (orientation, trust score, influence flags)
- 🗄️ **Stores** everything locally in SQLite — no cloud dependency
- 🌐 **Exposes** a FastAPI REST API with filtering, pagination, and sorting
- 📦 **Ships** a ready-to-use Python client library

---

## Architecture

```
[ News Websites ]
       │  HTTP scraping (Scrapy)
       ▼
[ Home Server — Scrapy Engine ]   ← Continuous crawl with dedup & validation
       │  Direct write
       ▼
[ SQLite Database ]               ← Local, unlimited storage
       │  Query layer
       ▼
[ FastAPI REST API ]              ← Exposed via argos.news.ydns.eu
       │  Python calls
       ▼
[ ArgosClient Library ]           ← Import and use in any Python project
```

**Pipeline stages per article:**

| Stage | Component | Priority |
|-------|-----------|----------|
| Field validation | `pipelines/validation.py` | 100 |
| URL deduplication | `pipelines/dedup.py` | 200 |
| SQLite storage | `pipelines/storage.py` | 800 |

---

## News Sources

| Source | Country | Orientation | Trust Score | Israeli-Influenced |
|--------|---------|-------------|:-----------:|--------------------|
| [Al Jazeera](https://www.aljazeera.com) | Qatar | Center-Left | 0.55 | No |
| [Al Arabiya](https://www.alarabiya.net) | Saudi Arabia | Right | 0.40 | Yes |
| [Sky News Arabia](https://www.skynewsarabia.com) | UAE | Center-Right | 0.48 | Yes |
| [Middle East Eye](https://www.middleeasteye.net) | UK (ME Focus) | Left | 0.48 | No |

> Trust scores and influence flags are defined in `scraper/argos/registry.py` and are attached to every article at scrape time.

---

## Quick Start

### Prerequisites

- Python **3.11+**
- `pip`
- Linux or macOS (for the home server component)

### 1. Install

```bash
# Clone the repository
git clone <repo-url> && cd argos

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment (defaults work for local development)
cp .env.example .env
```

### 2. Launch

```bash
# Start the API server + all 4 spiders
python launch.py

# Or start components independently:
python launch.py --api-only                                   # API server only
python launch.py --spiders-only                               # All spiders
python launch.py --spiders aljazeera middleeasteye            # Specific spiders
```

### 3. Verify

| Resource | URL |
|----------|-----|
| Swagger UI | http://localhost:8000/docs |
| Health check | http://localhost:8000/v1/health |
| Latest articles | http://localhost:8000/v1/articles/latest |
| Sources | http://localhost:8000/v1/sources |

### 4. Use the Python Client

```python
from client import ArgosClient

# Connect to the public instance
client = ArgosClient()

# Or connect to your local instance
# client = ArgosClient(base_url="http://localhost:8000")

# Fetch the 10 most recent articles
latest = client.get_latest(count=10)
for article in latest:
    print(f"[{article['source_name']}] {article['title']}")

# Search with filters
results = client.get_articles(
    source_name="Al Jazeera",
    category="war",
    date_from="2024-01-01",
    page_size=50,
)
print(f"Found {results['pagination']['total_results']} articles")

# List all sources
for source in client.get_sources():
    print(f"{source['source_name']} ({source['country']}) — trust: {source['trust_score']}")
```

---

## Running Spiders

```bash
source venv/bin/activate

# Run a single spider
scrapy crawl aljazeera
scrapy crawl alarabiya
scrapy crawl skynewsarabia
scrapy crawl middleeasteye

# Limit output for testing
scrapy crawl aljazeera -s CLOSESPIDER_ITEMCOUNT=10

# Resume an interrupted crawl
scrapy crawl aljazeera -s JOBDIR=./data/crawl_jobs/aljazeera
```

---

## API Reference

### Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/v1/health` | System status |
| `GET` | `/v1/sources` | List all news sources |
| `GET` | `/v1/articles` | Search articles with filters and pagination |
| `GET` | `/v1/articles/latest` | Fetch N most recent articles |
| `GET` | `/v1/articles/{id}` | Fetch a single article by ID |

Full interactive documentation is available at `/docs` (Swagger UI) when the server is running.

### Filter Parameters — `GET /v1/articles`

| Parameter | Type | Description |
|-----------|------|-------------|
| `source_name` | string | Filter by network name |
| `country` | string | Filter by source country |
| `category` | string | `politics`, `economy`, `tech`, `war`, `health`, `science`, `culture`, `sports`, `environment`, `other` |
| `topic` | string | Alias for `category` |
| `orientation` | string | Editorial leaning |
| `is_israeli_influenced` | bool | Israeli influence flag |
| `date_from` | ISO 8601 | Earliest publication date |
| `date_to` | ISO 8601 | Latest publication date |
| `title` | string | Partial title match |
| `tags` | string | Comma-separated tags (AND logic) |
| `page` | int | Page number (default: `1`) |
| `page_size` | int | Results per page (default: `20`, max: `100`) |
| `sort_by` | string | Column to sort by (default: `published_at`) |
| `order` | string | `asc` or `desc` (default: `desc`) |

---

## Scheduling

ARGOS is designed for continuous, unattended operation. The recommended setup uses **cron** for spiders and **systemd** or **screen** for the API.

```bash
# Edit crontab to scrape every hour
crontab -e
0 * * * * cd /path/to/argos && /path/to/venv/bin/python launch.py --spiders-only
```

```bash
# Keep the API alive with screen
screen -dmS argos-api python launch.py --api-only

# Or create a systemd service for production
# → /etc/systemd/system/argos-api.service
```

---

## Data Model

Every article is stored with exactly **13 fields**:

| Field | Type | Origin |
|-------|------|--------|
| `title` | `string` | Scraped from article HTML |
| `content` | `string` | Scraped from article HTML |
| `summary` | `string` | Meta tags or first paragraph |
| `category` | `enum` | Spider classification |
| `tags` | `list[str]` | Article metadata |
| `source_name` | `string` | `registry.py` |
| `source_url` | `string` | Spider URL |
| `published_at` | ISO 8601 | Article HTML |
| `scraped_at` | ISO 8601 | Auto-generated at crawl time |
| `country` | `string` | `registry.py` |
| `orientation` | `enum` | `registry.py` |
| `trust_score` | `float [0–1]` | `registry.py` |
| `is_israeli_influenced` | `bool` | `registry.py` |

---

## Configuration

Copy `.env.example` to `.env` and adjust as needed. Defaults work out of the box for local development.

| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_PATH` | `./data/argos.db` | SQLite database path |
| `API_HOST` | `0.0.0.0` | API bind address |
| `API_PORT` | `8000` | API port |
| `API_BASE_URL` | `http://argos.news.ydns.eu:8000` | Public API base URL |
| `PROXY_URL` | *(empty)* | Optional HTTP proxy for geo-blocked sources |
| `LOG_LEVEL` | `INFO` | Logging verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
| `JOBDIR` | `./data/crawl_jobs` | Scrapy job persistence directory |

---

## Project Structure

```
argos/
├── launch.py                      ← Entry point — start API and/or spiders
├── requirements.txt
├── .env.example
├── scrapy.cfg
│
├── scraper/argos/                 ← Scrapy engine
│   ├── items.py                   ← ArticleItem (13 fields)
│   ├── registry.py                ← Source metadata + apply_registry()
│   ├── settings.py                ← Scrapy settings (documented)
│   ├── middlewares.py             ← User-agent rotation + proxy support
│   ├── pipelines/
│   │   ├── validation.py          ← Field validation        (priority 100)
│   │   ├── dedup.py               ← URL deduplication       (priority 200)
│   │   └── storage.py             ← SQLite write            (priority 800)
│   └── spiders/
│       ├── aljazeera.py
│       ├── alarabiya.py
│       ├── skynewsarabia.py
│       └── middleeasteye.py
│
├── api/                           ← FastAPI REST layer
│   ├── main.py                    ← App entry point
│   ├── config.py                  ← Environment settings
│   ├── database.py                ← SQLite query layer
│   ├── models.py                  ← Pydantic response models
│   ├── openapi.yaml               ← OpenAPI 3.1 spec
│   └── routes/
│       ├── articles.py            ← /v1/articles
│       ├── sources.py             ← /v1/sources
│       └── health.py              ← /v1/health
│
└── client/                        ← Python client library
    ├── argos_client.py            ← ArgosClient class
    └── exceptions.py              ← ArgosAPIError
```

---

## Developer

**Zayani Mouhamed Mouheb / EGen Labs** — ARGOS v0.3.x

> *Argus Panoptes sees everything.*
