---
phase: 02-exness-daily-tick-cache-seeder-service-with-registry-integration
plan: 02
type: execute
wave: 2
depends_on: ["02-01"]
files_modified:
  - scripts/exness_tick_cache_seeder.py
autonomous: true
requirements:
  - EXNS-DL
  - EXNS-PARSE
  - EXNS-WRITE
  - EXNS-IDEM
  - EXNS-TEL
must_haves:
  truths:
    - "Seeder downloads Exness daily ZIPs and writes 3-column Parquet files"
    - "Weekend 404s are silently skipped without error logging"
    - "Already-cached days are skipped (idempotent)"
    - "State file written on completion for monit freshness check"
    - "Telegram reports sent: silent on success, persistent on failure"
    - "Symbols read from registry filtered by exchange=exness, not hardcoded"
  artifacts:
    - path: "scripts/exness_tick_cache_seeder.py"
      provides: "Exness forex tick cache seeder script"
      min_lines: 200
  key_links:
    - from: "scripts/exness_tick_cache_seeder.py"
      to: "python/opendeviationbar/symbol_registry.py"
      via: "get_symbol_entries() filtered by exchange='exness'"
      pattern: "get_symbol_entries|exchange.*exness"
    - from: "scripts/exness_tick_cache_seeder.py"
      to: "python/opendeviationbar/config/exness_seeder.py"
      via: "ExnessSeederConfig() instantiation"
      pattern: "ExnessSeederConfig"
    - from: "scripts/exness_tick_cache_seeder.py"
      to: "python/opendeviationbar/storage/parquet_io.py"
      via: "_atomic_write_parquet() for daily Parquet write"
      pattern: "_atomic_write_parquet"
    - from: "scripts/exness_tick_cache_seeder.py"
      to: "ticks.ex2archive.com"
      via: "requests.get() HTTP download"
      pattern: "ticks.ex2archive.com"
---

<objective>
Create the Exness forex tick cache seeder script that downloads daily ZIPs, parses CSV, writes Parquet, and reports via Telegram.

Purpose: This is the core data ingestion script. It mirrors `scripts/tick_cache_seeder.py` (Binance seeder) architecture but for Exness forex quotes.

Output: `scripts/exness_tick_cache_seeder.py` — a complete, runnable seeder script.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-CONTEXT.md
@.planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-RESEARCH.md
@.planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-01-SUMMARY.md

<interfaces>
<!-- Contracts from Plan 01 and existing codebase -->

From python/opendeviationbar/config/exness_seeder.py (created in Plan 01):

```python
class ExnessSeederConfig(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="OPENDEVIATIONBAR_EXNESS_SEEDER_", case_sensitive=False)
    historical_start: str = "2022-01-01"
    max_concurrent_downloads: int = 4
    download_timeout_s: int = 60
    state_file: str = "/var/tmp/opendeviationbar-exness-seeder-state.txt"
```

From python/opendeviationbar/symbol_registry.py:

```python
def get_symbol_entries() -> MappingProxyType[str, SymbolEntry]: ...
# SymbolEntry fields: symbol, enabled, asset_class, exchange, market, tier,
#                     listing_date, effective_start, thresholds, min_threshold, keywords
```

From python/opendeviationbar/storage/parquet_io.py:

```python
def _atomic_write_parquet(df: pl.DataFrame, target_path: Path, *, compression="zstd", compression_level=3) -> None: ...
def get_cache_dir() -> Path: ...  # Returns ~/.cache/opendeviationbar
```

From python/opendeviationbar/storage/parquet_query.py:

```python
def _get_parquet_path(ticks_dir: Path, symbol: str, day_key: str) -> Path: ...
def _get_symbol_dir(ticks_dir: Path, symbol: str) -> Path: ...
```

From python/opendeviationbar/notify/telegram.py:

```python
def send_telegram(message: str, *, parse_mode="HTML", disable_notification=False, timeout=None) -> int | bool: ...
```

Exness URL pattern:

```
https://ticks.ex2archive.com/ticks/{SYMBOL}_Raw_Spread/{YYYY}/{MM}/{DD}/Exness_{SYMBOL}_Raw_Spread_{YYYY}_{MM}_{DD}.zip
```

Exness CSV format: 5 columns WITH header row: "Exness","Symbol","Timestamp","Bid","Ask"
Timestamp format: ISO 8601 UTC with millisecond precision (e.g., "2024-01-15 00:00:00.032Z")
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create Exness seeder script — download, parse, write pipeline</name>
  <files>scripts/exness_tick_cache_seeder.py</files>
  <read_first>
    - scripts/tick_cache_seeder.py (Binance seeder — the pattern to mirror, especially main(), _seed_symbol(), _download_vision_day())
    - python/opendeviationbar/storage/parquet_io.py (get_cache_dir, _atomic_write_parquet signatures)
    - python/opendeviationbar/storage/parquet_query.py (_get_parquet_path, _get_symbol_dir)
    - python/opendeviationbar/symbol_registry.py (get_symbol_entries, SymbolEntry, get_effective_start_date)
    - python/opendeviationbar/notify/telegram.py (send_telegram signature)
    - python/opendeviationbar/config/exness_seeder.py (ExnessSeederConfig — created in Plan 01)
    - .planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-CONTEXT.md (locked decisions)
    - .planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-RESEARCH.md (patterns, pitfalls)
  </read_first>
  <action>
    Create `scripts/exness_tick_cache_seeder.py` — a complete seeder script mirroring `tick_cache_seeder.py` structure. The script must contain:

    **IMPORTANT — Data fetch override:** CONTEXT.md says "Uses ExnessFetcher.fetch_day() from the Rust crate" but ExnessFetcher is NOT exposed via PyO3 bindings. Use Python `requests.get()` instead, matching the Binance seeder pattern. Add a comment at the top of the download function:
    ```python
    # NOTE: ExnessFetcher (Rust crate) is not exposed via PyO3.
    # Using Python requests mirrors the Binance seeder pattern.
    # Rust fetcher reserved for future bar construction phase.
    ```

    **IMPORTANT — Telegram API:** Do NOT use `ephemeral_ttl_minutes` parameter — it does not exist on `send_telegram()`. Use `disable_notification=True` for silent healthy-run messages. Ephemeral auto-delete behavior described in CONTEXT.md is aspirational and not yet implemented.

    **Module docstring** explaining: what it does, how it mirrors the Binance seeder, which registry it reads from, what it writes, how to run it (one-off + systemd timer).

    **Imports:**
    ```python
    from __future__ import annotations
    import concurrent.futures
    import io
    import logging
    import os
    import sys
    import time
    import zipfile
    from datetime import UTC, date, datetime, timedelta
    from pathlib import Path
    from typing import TYPE_CHECKING
    import polars as pl
    import requests
    if TYPE_CHECKING:
        from opendeviationbar.config.exness_seeder import ExnessSeederConfig
    ```

    **Constants:**
    - `_EXNESS_BASE_URL = "https://ticks.ex2archive.com/ticks"`
    - `_CACHE_KEY_PREFIX = "EXNESS_RAW_SPREAD_"` (per CONTEXT.md locked decision)

    **Functions to implement:**

    1. `_exness_url(symbol: str, d: date) -> str`
       - Constructs URL: `{_EXNESS_BASE_URL}/{symbol}_Raw_Spread/{YYYY}/{MM}/{DD}/Exness_{symbol}_Raw_Spread_{YYYY}_{MM}_{DD}.zip`
       - Zero-padded month and day

    2. `_download_exness_day(symbol: str, d: date, timeout_s: int) -> pl.DataFrame | None`
       - Downloads ZIP via `requests.get(url, timeout=timeout_s)`
       - Returns `None` on 404 (weekend/holiday) — log at DEBUG only, NOT error
       - Raises on other HTTP errors via `resp.raise_for_status()`
       - Extracts first file from ZIP using `zipfile.ZipFile(io.BytesIO(resp.content))`
       - Parses CSV with `pl.read_csv(csv_f, has_header=True)` — CRITICAL: `has_header=True` (Exness has headers, unlike Binance)
       - Handles `Z` suffix in timestamps: strip `Z` before parsing if present, use `str.replace("Z", "")` on the Timestamp column
       - Parses timestamps: `pl.col("Timestamp").str.to_datetime("%Y-%m-%d %H:%M:%S%.3f").dt.epoch(time_unit="ms")`
       - Selects and renames to 3-column schema: `timestamp` (i64 ms), `bid` (f64), `ask` (f64)
       - Sorts by `timestamp`

    3. `_seed_symbol(symbol: str, start_date: date, ticks_dir: Path, max_workers: int, timeout_s: int) -> int`
       - Signature has 5 params only — NO `state_path` or `run_start_iso` (state file writing belongs in `main()` after the symbol loop, mirroring Binance seeder)
       - Generates date range from `start_date` to yesterday (T-1 UTC), newest-first
       - Cache key: `f"{_CACHE_KEY_PREFIX}{symbol}"`
       - For each day, check if Parquet already exists via `_get_parquet_path(ticks_dir, cache_key, day_str).exists()`
       - Skip existing days (idempotent)
       - Use `concurrent.futures.ThreadPoolExecutor(max_workers=max_workers)` for parallel downloads
       - For each successful download, write via `_atomic_write_parquet(df, parquet_path)` after ensuring parent dir exists
       - Count and return total days seeded
       - Log progress every 50 days

    4. `main() -> int`
       - Configure logging (INFO level, format with timestamp)
       - Load `ExnessSeederConfig()`
       - Load `TickStorage` path via `get_cache_dir() / "ticks"`
       - Get Exness symbols: `{name: entry for name, entry in get_symbol_entries().items() if entry.exchange == "exness" and entry.enabled}`
       - If no symbols, log warning and exit 0
       - Write initial state file: `running\ntimestamp={iso}\nsymbols={count}\n`
       - Loop over sorted symbols, call `_seed_symbol()` for each
       - Use per-symbol `effective_start` when later than `cfg.historical_start`
       - Track: total_seeded, symbols_processed, symbols_failed
       - Write final state file: `ok\ntimestamp={iso}\nsymbols={processed}\nseeded={total}\nelapsed_s={secs}\nfailed={count}\n`
       - Send Telegram: on success use `disable_notification=True` (silent); on failure use `disable_notification=False` (loud)
       - Telegram success format: `"<b>Exness seeder:</b> {n} symbols, {seeded} new days, {elapsed:.0f}s elapsed"`
       - Telegram failure format: `"<b>Exness seeder FAILED:</b> {failed_list} — {n} symbol(s) failed"`
       - Return 0 on success, 1 if any symbol failed

    5. `if __name__ == "__main__": sys.exit(main())`

    **Key design differences from Binance seeder:**
    - No batch rotation (10 symbols is small enough to process all each run)
    - No REST fallback (Exness has no equivalent REST API)
    - No HEAD check for data availability (just try downloading)
    - 4 concurrent downloads (not 12 — smaller volume, no rate limiting concern)
    - `has_header=True` for CSV parsing (Exness CSVs have headers)
    - 3-column schema (timestamp, bid, ask) instead of 7-column aggTrades
    - Bypass `write_ticks()` entirely — use `_atomic_write_parquet()` directly (no agg_trade_id validation)
    - Uses Python `requests.get()` for downloads (ExnessFetcher not exposed via PyO3)

    **Error handling:**
    - Wrap each symbol in try/except, log exception, add to symbols_failed list
    - Within `_download_exness_day`: 404 returns None, other errors propagate
    - Within `_seed_symbol`: individual day download failures logged and counted but don't abort the symbol

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python3 -c "
import ast, sys
with open('scripts/exness_tick_cache_seeder.py') as f:
    tree = ast.parse(f.read())
funcs = [n.name for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)]
required = ['_exness_url', '_download_exness_day', '_seed_symbol', 'main']
for r in required:
    assert r in funcs, f'Missing function: {r}'
# Check key patterns in source
src = open('scripts/exness_tick_cache_seeder.py').read()
assert 'ticks.ex2archive.com' in src, 'Missing Exness URL'
assert 'EXNESS_RAW_SPREAD_' in src, 'Missing cache key prefix'
assert 'has_header=True' in src or 'has_header = True' in src, 'Missing has_header=True'
assert 'get_symbol_entries' in src, 'Not reading from registry'
assert '_atomic_write_parquet' in src, 'Not using atomic write'
assert 'send_telegram' in src, 'Missing Telegram integration'
assert 'ExnessSeederConfig' in src, 'Missing config class usage'
assert 'requests.get' in src, 'Missing requests.get (Python HTTP download)'
assert 'ephemeral_ttl' not in src, 'MUST NOT use ephemeral_ttl_minutes — does not exist on send_telegram()'
# Verify _seed_symbol signature does not have state_path or run_start_iso
import inspect, importlib.util
# Check via AST that _seed_symbol has exactly 5 params (symbol, start_date, ticks_dir, max_workers, timeout_s)
for node in ast.walk(tree):
    if isinstance(node, ast.FunctionDef) and node.name == '_seed_symbol':
        args = node.args
        param_names = [a.arg for a in args.args]
        assert 'state_path' not in param_names, '_seed_symbol must NOT have state_path param'
        assert 'run_start_iso' not in param_names, '_seed_symbol must NOT have run_start_iso param'
        break
print('Exness seeder script structure validated OK')
"</automated>
  </verify>
  <acceptance_criteria>
    - File `scripts/exness_tick_cache_seeder.py` exists with 200+ lines
    - Contains functions: _exness_url, _download_exness_day, _seed_symbol, main
    - URL pattern uses `ticks.ex2archive.com/ticks/{SYMBOL}_Raw_Spread/...`
    - Cache key prefix is `EXNESS_RAW_SPREAD_`
    - CSV parsing uses `has_header=True`
    - Symbols loaded via `get_symbol_entries()` filtered by `exchange == "exness"`
    - Parquet write via `_atomic_write_parquet()` (NOT `write_ticks()`)
    - Downloads via `requests.get()` (NOT ExnessFetcher — not exposed via PyO3)
    - 404 responses return None (not logged as error)
    - State file written to `/var/tmp/opendeviationbar-exness-seeder-state.txt`
    - Telegram: `send_telegram()` called with `disable_notification=True` for success, `False` for failure — NO `ephemeral_ttl_minutes`
    - `_seed_symbol()` signature has NO `state_path` or `run_start_iso` params
    - Script has `if __name__ == "__main__": sys.exit(main())`
    - Module docstring present explaining purpose and usage
    - Comment in `_download_exness_day` noting ExnessFetcher override rationale
  </acceptance_criteria>
  <done>Complete Exness seeder script that downloads ZIPs, parses CSV to 3-column Parquet, reports via Telegram, with idempotent operation and newest-first ordering</done>
</task>

<task type="auto">
  <name>Task 2: Add unit tests for Exness seeder core functions</name>
  <files>tests/test_exness_seeder.py</files>
  <read_first>
    - scripts/exness_tick_cache_seeder.py (the script just created — to understand function signatures)
    - .planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-RESEARCH.md (test map EXNS-01 through EXNS-07)
  </read_first>
  <action>
    Create `tests/test_exness_seeder.py` with pytest tests covering the research-defined test map:

    ```python
    """Tests for Exness tick cache seeder (Phase 02).

    Covers EXNS-01 through EXNS-07 from 02-RESEARCH.md test map.
    """
    from __future__ import annotations

    import io
    import zipfile
    from datetime import date
    from pathlib import Path
    from unittest.mock import MagicMock, patch

    import polars as pl
    import pytest
    ```

    Tests to implement:

    1. `test_exness_url_construction()` — verify `_exness_url("EURUSD", date(2024, 1, 15))` returns correct URL with zero-padded components

    2. `test_download_parse_csv()` (EXNS-01) — mock `requests.get` to return a ZIP containing a CSV with Exness header row (`"Exness","Symbol","Timestamp","Bid","Ask"`), verify returned DataFrame has 3 columns (timestamp, bid, ask), timestamp is in ms epoch, bid/ask are f64

    3. `test_weekend_404_returns_none()` (EXNS-02) — mock `requests.get` returning status_code=404, verify `_download_exness_day()` returns None (not raises)

    4. `test_registry_filter_exness_only()` (EXNS-03) — call `get_symbol_entries()`, filter by `exchange == "exness"`, verify only Exness symbols returned (10 expected), no Binance symbols leak through

    5. `test_parquet_write_3_columns()` (EXNS-04) — create a small DataFrame with (timestamp, bid, ask), write via `_atomic_write_parquet`, read back, verify schema and values match

    6. `test_idempotent_skip()` (EXNS-05) — create a mock Parquet file at the expected path, verify `_seed_symbol` does not re-download that day (mock requests.get should not be called for that date)

    7. `test_state_file_written()` (EXNS-06) — run `main()` with mocked downloads (all return None = weekend), verify state file contains "ok\n" and "timestamp=" and "failed=0"

    8. `test_registry_entries_count()` (EXNS-07) — verify exactly 10 Exness entries in symbols.toml with correct fields (exchange="exness", asset_class="forex", market="raw_spread", enabled=true)

    Use `tmp_path` fixture for file system tests. Mock `requests.get` and `send_telegram` to avoid network calls.

    For the CSV mock data in test_download_parse_csv, use:
    ```
    "Exness","EURUSD","2024-01-15 00:00:00.032Z","1.08765","1.08775"
    "Exness","EURUSD","2024-01-15 00:00:00.155Z","1.08766","1.08776"
    "Exness","EURUSD","2024-01-15 00:00:00.288Z","1.08764","1.08774"
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_exness_seeder.py -x -q --timeout=30 2>&1 | tail -5</automated>
  </verify>
  <acceptance_criteria>
    - File `tests/test_exness_seeder.py` exists
    - Contains tests: test_exness_url_construction, test_download_parse_csv, test_weekend_404_returns_none, test_registry_filter_exness_only, test_parquet_write_3_columns, test_idempotent_skip, test_state_file_written, test_registry_entries_count
    - All tests pass with `pytest tests/test_exness_seeder.py -x`
    - No network calls made during tests (all HTTP mocked)
  </acceptance_criteria>
  <done>8 unit tests covering download, parse, write, idempotency, state file, registry — all passing</done>
</task>

</tasks>

<verification>
```bash
# Full test run
cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_exness_seeder.py -v --timeout=30

# Script syntax check

python3 -c "import ast; ast.parse(open('scripts/exness_tick_cache_seeder.py').read()); print('Syntax OK')"

# Verify script imports work

python3 -c "
import sys; sys.path.insert(0, 'scripts')

# Just check that imports resolve (don't run main)

exec(open('scripts/exness_tick_cache_seeder.py').read().split['if **name**'](0))
print('Imports OK')
"

```
</verification>

<success_criteria>
- Seeder script downloads Exness ZIPs, parses to 3-column Parquet, writes atomically
- Weekend 404s silently skipped
- Idempotent: existing Parquet files not re-downloaded
- State file written for monit
- Telegram reporting on success (silent via disable_notification=True) and failure (persistent)
- All 8 unit tests passing
</success_criteria>

<output>
After completion, create `.planning/phases/02-exness-daily-tick-cache-seeder-service-with-registry-integration/02-02-SUMMARY.md`
</output>
```
