Metadata-Version: 2.4
Name: argos-news
Version: 0.1.0
Summary: Automated Realtime Global Observer System - Python Client & API for Middle East News
Author-email: EGen Labs <contact@egenlabs.com>
License: MIT
Project-URL: Homepage, https://github.com/egenlabs/argos
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content :: News/Diary
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.24.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: server
Requires-Dist: fastapi>=0.100.0; extra == "server"
Requires-Dist: uvicorn>=0.23.0; extra == "server"
Requires-Dist: aiosqlite>=0.19.0; extra == "server"
Requires-Dist: scrapy>=2.11.0; extra == "server"
Requires-Dist: pydantic-settings>=2.0.0; extra == "server"

# ARGOS — Automated Realtime Global Observer System

<p align="center">
  <img src="static/logo.png" alt="ARGOS Logo" width="300" />
</p>

**ARGOS** is a 3-layer automated news pipeline that continuously scrapes articles from 4 Middle Eastern news networks, enriches them with editorial metadata, stores them locally, and exposes them via a REST API and Python client library.

---

## Architecture

```
[ News Websites ]               ← 4 Middle East sources
      │  HTTP scraping (Scrapy)
      ▼
[ Home Server — Scrapy Engine ]  ← Runs on your machine
      │  Direct write to local DB
      ▼
[ SQLite Database ]              ← Local, unlimited storage
      │  REST API queries
      ▼
[ FastAPI API Layer ]            ← Exposed via argos.news.ydns.eu
      │  Python function calls
      ▼
[ ArgosClient Library ]          ← Import and use
```

## Source Networks

| Source | Country | Orientation | Trust | Israeli-Influenced |
|--------|---------|-------------|-------|--------------------|
| Al Jazeera | Qatar | center-left | 0.55 | No |
| Al Arabiya | Saudi Arabia | right | 0.40 | Yes |
| Sky News Arabia | UAE | center-right | 0.48 | Yes |
| Middle East Eye | UK (ME Focus) | left | 0.48 | No |

---

## Quick Start

### 1. Prerequisites

- Python 3.11+
- pip
- Linux/macOS (home server)

### 2. Install

```bash
# Clone the repository
cd /path/to/project

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Copy and configure environment
cp .env.example .env
# Edit .env with your settings (defaults work for local development)
```

### 3. Launch Everything

```bash
# Start API server + all 4 spiders
python launch.py

# Or start components separately:
python launch.py --api-only       # API server only
python launch.py --spiders-only   # All spiders only
python launch.py --spiders aljazeera middleeasteye  # Specific spiders
```

### 4. Access the API

- **API Docs:** http://localhost:8000/docs (Swagger UI)
- **Health Check:** http://localhost:8000/v1/health
- **Latest Articles:** http://localhost:8000/v1/articles/latest
- **Sources:** http://localhost:8000/v1/sources

### 5. Use the Python Client

```python
from client import ArgosClient

# Connects to argos.news.ydns.eu automatically
client = ArgosClient()

# For local development:
# client = ArgosClient(base_url="http://localhost:8000")

# Get latest articles
latest = client.get_latest(count=10)
for article in latest:
    print(f"[{article['source_name']}] {article['title']}")

# Search with filters
results = client.get_articles(
    source_name="Al Jazeera",
    category="war",
    date_from="2024-01-01",
    page_size=50,
)
print(f"Found {results['pagination']['total_results']} articles")

# Get all sources
sources = client.get_sources()
for s in sources:
    print(f"{s['source_name']} ({s['country']}) — trust: {s['trust_score']}")
```

---

## Running Spiders Individually

```bash
# Activate venv
source venv/bin/activate

# Run a specific spider
scrapy crawl aljazeera
scrapy crawl alarabiya
scrapy crawl skynewsarabia
scrapy crawl middleeasteye

# Limit items for testing
scrapy crawl aljazeera -s CLOSESPIDER_ITEMCOUNT=10

# Resume a previously interrupted crawl (L-02)
scrapy crawl aljazeera -s JOBDIR=./data/crawl_jobs/aljazeera
```

## API Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/v1/health` | System status |
| GET | `/v1/sources` | List all source networks |
| GET | `/v1/articles` | Search with filters + pagination |
| GET | `/v1/articles/latest` | N most recent articles |
| GET | `/v1/articles/{id}` | Single article by ID |

### Filter Parameters (GET /v1/articles)

| Parameter | Type | Description |
|-----------|------|-------------|
| `source_name` | string | Filter by news network |
| `country` | string | Filter by source country |
| `category` | string | Topic: politics, economy, tech, war, health, science, culture, sports, environment, other |
| `topic` | string | Alias for category |
| `orientation` | string | Editorial leaning |
| `is_israeli_influenced` | bool | Israeli influence flag |
| `date_from` | string | Min date (ISO 8601) |
| `date_to` | string | Max date (ISO 8601) |
| `title` | string | Title search (partial match) |
| `tags` | string | Comma-separated tags (AND logic) |
| `page` | int | Page number (default 1) |
| `page_size` | int | Items per page (default 20, max 100) |
| `sort_by` | string | Sort column (default: published_at) |
| `order` | string | asc or desc (default: desc) |

---

## Scheduling (Continuous Scraping)

ARGOS is designed to run continuously via cron (FR-07):

```bash
# Edit crontab
crontab -e

# Run all spiders every hour
0 * * * * cd /path/to/project && /path/to/venv/bin/python launch.py --spiders-only

# Keep API running via systemd or screen
# Option 1: screen
screen -dmS argos-api python launch.py --api-only

# Option 2: systemd service (create /etc/systemd/system/argos-api.service)
```

## Environment Variables

See [`.env.example`](.env.example) for all variables with documentation.

| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_PATH` | `./data/argos.db` | SQLite database path |
| `API_HOST` | `0.0.0.0` | API bind address |
| `API_PORT` | `8000` | API port |
| `API_BASE_URL` | `http://argos.news.ydns.eu:8000` | Public API URL |
| `PROXY_URL` | *(empty)* | Optional proxy for geo-blocking |
| `LOG_LEVEL` | `INFO` | Logging verbosity |
| `JOBDIR` | `./data/crawl_jobs` | Scrapy job persistence |

---

## Project Structure

```
argos/
├── launch.py                    ← 🚀 Launch everything
├── requirements.txt             ← Dependencies
├── .env.example                 ← Environment config template
├── scrapy.cfg                   ← Scrapy project config
│
├── scraper/argos/               ← Scrapy scraping engine
│   ├── items.py                 ← ArticleItem (13 fields)
│   ├── registry.py              ← SOURCE_REGISTRY + apply_registry()
│   ├── settings.py              ← Fully documented settings
│   ├── middlewares.py           ← UA rotation + proxy
│   ├── pipelines/
│   │   ├── validation.py        ← Field validation (priority 100)
│   │   ├── dedup.py             ← URL dedup (priority 200)
│   │   └── storage.py           ← SQLite write (priority 800)
│   └── spiders/
│       ├── aljazeera.py         ← Al Jazeera spider
│       ├── alarabiya.py         ← Al Arabiya spider
│       ├── skynewsarabia.py     ← Sky News Arabia spider
│       └── middleeasteye.py     ← Middle East Eye spider
│
├── api/                         ← FastAPI REST API
│   ├── main.py                  ← App entry point
│   ├── config.py                ← Environment settings
│   ├── database.py              ← SQLite query layer
│   ├── models.py                ← Pydantic response models
│   ├── openapi.yaml             ← OpenAPI 3.1 spec
│   └── routes/
│       ├── articles.py          ← /v1/articles endpoints
│       ├── sources.py           ← /v1/sources endpoint
│       └── health.py            ← /v1/health endpoint
│
└── client/                      ← Python client library
    ├── argos_client.py          ← ArgosClient class
    └── exceptions.py            ← ArgosAPIError
```

---

## Data Model

Every article has exactly **13 fields** — no more, no less:

| Field | Type | Source |
|-------|------|--------|
| `title` | string | Article HTML |
| `content` | string | Article HTML |
| `summary` | string | Meta tags / first paragraph |
| `category` | enum | Spider classification |
| `tags` | list[str] | Article metadata |
| `source_name` | string | Registry |
| `source_url` | string | Spider URL |
| `published_at` | ISO 8601 | Article HTML |
| `scraped_at` | ISO 8601 | Auto-generated |
| `country` | string | Registry |
| `orientation` | enum | Registry |
| `trust_score` | float 0-1 | Registry |
| `is_israeli_influenced` | bool | Registry |

---

## Developer

**Zayani Mouhamed Mouheb / EGen Labs**

ARGOS v0.1.0 — *Argus Panoptes sees everything.*
