# Phase 2: Exness daily tick cache seeder service with registry integration - Context

**Gathered:** 2026-03-23
**Status:** Ready for planning
**Mode:** Auto-generated (--auto, infrastructure phase with user-specified requirements)

<domain>
## Phase Boundary

Build an Exness forex tick cache seeder that mirrors the Binance seeder (`scripts/tick_cache_seeder.py`) architecture. Downloads daily ZIPs from `ticks.ex2archive.com` and writes daily Parquet files to `~/.cache/opendeviationbar/ticks/`. Wires Exness forex symbols into the unified `symbols.toml` registry as SSoT. Deploys as a systemd oneshot + 1h timer with Telegram status reporting and monit monitoring.

</domain>

<decisions>
## Implementation Decisions

### Registry Integration (SSoT)

- All 10 Exness forex pairs MUST be added to `python/opendeviationbar/data/symbols.toml` as the single source of truth
- Registry entries follow the exact same schema as Binance symbols: `enabled`, `asset_class = "forex"`, `exchange = "exness"`, `market = "raw_spread"`, `tier`, `listing_date`, `effective_start`, `thresholds`, `min_threshold`
- The seeder script reads symbols from `get_registered_symbols(enabled_only=True)` filtered by `exchange = "exness"` — no hardcoded symbol lists
- Instruments: EURUSD, GBPUSD, USDJPY, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, XAUUSD
- Thresholds per CLAUDE.md: EURUSD at [50, 100, 200], XAUUSD TBD (researcher should determine)

### Seeder Script Design

- New script: `scripts/exness_tick_cache_seeder.py` — mirrors `tick_cache_seeder.py` structure exactly
- Uses `ExnessFetcher.fetch_day(year, month, day)` from the Rust crate (already converted to daily in this session)
- Writes daily Parquet files: `~/.cache/opendeviationbar/ticks/EXNESS_RAW_SPREAD_{SYMBOL}/YYYY-MM-DD.parquet`
- Parquet schema: `timestamp` (ms), `bid` (f64), `ask` (f64) — no `agg_trade_id`, no `quantity`, no `is_buyer_maker` (Exness is quotes, not trades)
- Idempotent: skips days already cached (daily file existence check via `Path.exists()`)
- Newest-first ordering: matches kintsugi shard priority pattern
- Historical start date configurable via `OPENDEVIATIONBAR_EXNESS_SEEDER_HISTORICAL_START` env var (default: `2022-01-01`)
- Weekend/holiday 404s expected and silently skipped (forex markets closed)
- State file: `/var/tmp/opendeviationbar-exness-seeder-state.txt` — written on successful completion for monit freshness check

### Systemd Service + Timer

- `scripts/systemd/opendeviationbar-exness-seeder.service` — oneshot, mirrors Binance seeder service structure
- `scripts/systemd/opendeviationbar-exness-seeder.timer` — **1 hour interval** (not 5 min — no rush for forex data)
- `OnBootSec=300` (5 min after boot), `OnUnitActiveSec=3600` (every hour)
- `EnvironmentFile=` loads both `opendeviationbar.env` and `opendeviationbar.local.env` (same as all services)
- `TZ=UTC`, `LD_PRELOAD=libmimalloc.so.2`, `MIMALLOC_PURGE_DELAY=1000` (same memory pattern as Binance seeder)
- `MemoryHigh=2G`, `MemoryMax=4G` (lower than Binance seeder — forex data is ~10x smaller)
- `Conflicts=opendeviationbar-exness-seeder.service` (prevent overlapping runs)

### Telegram Reporting (@obdpy_bot)

- Verbose status reporting on each run via `opendeviationbar.notify.telegram`
- **Healthy run report** (ephemeral, auto-deleted after 15 min):
  - Symbols processed (count)
  - Days seeded this run (new files written)
  - Days skipped (already cached)
  - Total elapsed time
  - Example: `🟢 Exness seeder: 10 symbols, 3 new days seeded, 1247 skipped, 45s elapsed`
- **Error report** (persistent, LOUD notification):
  - Which symbol failed
  - HTTP error or ZIP extraction error
  - Number of consecutive failures
  - Example: `🔴 Exness seeder FAILED: XAUUSD download error (HTTP 503) — 3 consecutive failures`
- **Freshness alert** (via heartbeat, not seeder itself):
  - Heartbeat checks state file mtime. If >2h stale (2x the 1h interval), alert

### Monit Integration

- New monit monitor: `exness-seeder-state` — checks `/var/tmp/opendeviationbar-exness-seeder-state.txt` mtime freshness
- Pattern: identical to existing `seeder-state` monitor but with 2h threshold (2x the 1h timer interval)
- Added to `deploy/monit/opendeviationbar.conf` — the SSoT for all monit monitors
- Heartbeat script (`scripts/health_heartbeat.py`) should check `opendeviationbar-exness-seeder.timer` active status alongside existing timer checks

### CLAUDE.md Network (MANDATORY — every new directory gets a CLAUDE.md)

The hub-and-spoke CLAUDE.md network is how Claude Code navigates this repo. Every new directory MUST have its own CLAUDE.md file that:

1. States what the directory/module is for (1-2 sentences)
2. Lists key files with their purpose
3. Links back to parent CLAUDE.md
4. Documents any SSoT data (env vars, schemas, config keys)

**Existing CLAUDE.md files to UPDATE (add Exness sections):**

- `scripts/CLAUDE.md` — add "Exness Tick Cache Seeder" section (parallel to "Parquet Tick Cache Seeder (Issue #270)")
- `scripts/systemd/CLAUDE.md` — add 2 new units to the Services + Timers tables
- `deploy/CLAUDE.md` — add new monit monitor to the monitors table (10→11)
- `python/opendeviationbar/data/symbols.toml` — update header comment to mention Exness forex alongside Binance crypto
- Root `CLAUDE.md` — add Exness seeder to Quick Reference table and Architecture section

**New files that MUST have self-documenting headers:**

- `scripts/exness_tick_cache_seeder.py` — module docstring explaining: what it does, how it mirrors the Binance seeder, which registry it reads from, what it writes, how to run it
- Each new systemd unit file — comment block explaining purpose (same pattern as existing units)

**Scaffolding principle:** A future Claude Code session landing in any directory should be able to read the local CLAUDE.md and immediately understand: what's here, why it exists, what depends on it, and where to look next.

### Claude's Discretion

- Exact concurrent download count for Exness (suggest 4 — lower volume than Binance, no rate limiting observed)
- Whether to add a `source_url_pattern` field to symbols.toml registry entries (could be useful for future data sources)
- Parquet compression level (ZSTD-3 to match Binance seeder)
- Whether the Exness seeder shares `TickStorage` or uses a parallel implementation (recommend: share TickStorage with Exness-specific column schema)

</decisions>

<canonical_refs>

## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Existing patterns to mimic

- `scripts/tick_cache_seeder.py` — Binance seeder (the pattern to follow)
- `scripts/systemd/opendeviationbar-seeder.service` — systemd service template
- `scripts/systemd/opendeviationbar-seeder.timer` — timer template
- `python/opendeviationbar/data/symbols.toml` — registry SSoT (add Exness entries here)
- `python/opendeviationbar/storage/parquet.py` — TickStorage (daily Parquet write/read)

### Architecture docs

- `scripts/CLAUDE.md` — Seeder section, bigblack capacity envelope
- `scripts/systemd/CLAUDE.md` — Service table, timer patterns, OOM protection
- `deploy/CLAUDE.md` — Monit monitors, env files
- `deploy/monit/opendeviationbar.conf` — Monit config SSoT

### Exness data source

- `crates/opendeviationbar-providers/src/exness/client.rs` — `fetch_day()` Rust client (daily ZIPs)
- `crates/opendeviationbar-providers/src/exness/mod.rs` — Exness architecture overview
- `scripts/download_exness_forex.py` — Existing download script (reference for URL pattern)

</canonical_refs>

<specifics>
## Specific Ideas

- URL pattern confirmed: `https://ticks.ex2archive.com/ticks/{SYMBOL}_Raw_Spread/{YYYY}/{MM}/{DD}/Exness_{SYMBOL}_Raw_Spread_{YYYY}_{MM}_{DD}.zip` (verified HTTP 200, ~18MB/day for XAUUSD)
- Exness has zero rate limiting (100% reliability observed) — simpler than Binance seeder (no rate limiter needed)
- Forex markets closed on weekends — seeder must gracefully handle 404s for Saturday/Sunday without logging errors
- CSV format: 5 columns WITH header row: `"Exness","Symbol","Timestamp","Bid","Ask"` — different from Binance Vision (no header, 8 columns)
- Timestamp format: ISO 8601 UTC with millisecond precision (e.g., `2024-01-15 00:00:00.032Z`)
- The seeder writes raw quotes (bid/ask), NOT synthetic mid-price trades — bar construction is a separate concern

</specifics>

<deferred>
## Deferred Ideas

- Exness bar construction pipeline (processing quotes into open deviation bars via mid-price) — separate phase
- Exness kintsugi integration (gap detection + repair for forex bars) — depends on bar construction
- Exness ClickHouse table/schema for forex bars — depends on bar construction
- Exness streaming (if real-time quotes become available) — future capability

</deferred>

---

_Phase: 02-exness-daily-tick-cache-seeder-service-with-registry-integration_
_Context gathered: 2026-03-23 via --auto mode_
