# Storage Layer (Tier 1: Parquet Tick Cache)

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Tier-1 local Parquet cache for raw aggTrades. Stores downloaded ticks on disk to avoid re-downloading from Binance Vision/REST on subsequent runs.

---

## Files

| File               | Purpose                                                     |
| ------------------ | ----------------------------------------------------------- |
| `parquet.py`       | `TickStorage` — primary read/write API, daily partitioning  |
| `parquet_io.py`    | Low-level Parquet I/O helpers                               |
| `parquet_query.py` | Query-time utilities (filter, schema, timestamp conversion) |

---

## TickStorage (parquet.py)

**Path**: `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet`

Daily partitioned files per symbol (Issue #277). Each file is an atomic unit — writes are to `.parquet.tmp` then renamed (atomic on same filesystem). Key win: `has_ticks()` for a single day is O(1) via `Path.exists()` instead of reading Parquet metadata.

### Key Methods

| Method                                                | Description                                                                      |
| ----------------------------------------------------- | -------------------------------------------------------------------------------- |
| `write_ticks(symbol, ticks)`                          | Atomic write with dedup on `agg_trade_id`. LAZY-01 column-projected read (v7.2). |
| `read_ticks(symbol, start_ts_ms, end_ts_ms)`          | Lazy Polars scan with timestamp filter. Returns materialized DataFrame.          |
| `read_ticks_streaming(symbol, start, end, size)`      | Chunked iterator — bounded memory for large date ranges.                         |
| `has_ticks(symbol, start_ts_ms, end_ts_ms, cov=0.95)` | **O(1) file existence check** per day — no data loaded.                          |
| `list_days(symbol)`                                   | Returns list of `YYYY-MM-DD` strings with cached data.                           |
| `list_months(symbol)`                                 | Returns list of `YYYY-MM` strings (derived from daily files).                    |

### `has_ticks()` Is O(1) File Existence Check (Issue #277)

```python
# Kintsugi uses this in _shard_is_parquet_cached() — just Path.exists() per day
if storage.has_ticks("BTCUSDT", shard_start_ms, shard_end_ms, min_coverage=0.95):
    # fast-path: uses MAX_CONCURRENT_PARQUET=64
```

The `min_coverage=0.95` default tolerates up to 5% missing days in the range. With daily partitioning, this is a simple count of existing files vs total days in range.

### LAZY-01: Column-Projected Dedup (v7.2)

`write_ticks()` dedup reads only the `agg_trade_id` column from the existing Parquet file (`pl.read_parquet(columns=["agg_trade_id"])`) for the anti-join comparison. The full file (all 7 columns) is loaded only when new rows actually need to be merged, deferring the expensive read to the merge-needed branch.

### Compression

| Parameter            | Value  | Notes                                      |
| -------------------- | ------ | ------------------------------------------ |
| Codec                | ZSTD-3 | Level 3 balances speed and ratio           |
| Observed ratio       | 5.37×  | 6.50 MB for 761K trades (BTCUSDT daily)    |
| `_COMPRESSION_RATIO` | 4.0    | Conservative empirical in-memory expansion |

`_COMPRESSION_RATIO` is used by `estimate_tick_memory()` (MEM-004) to predict how much RAM a dataset will consume before materializing it.

---

## Memory Guards (MEM-001, MEM-004)

| Guard   | Mechanism                                                 | Location     |
| ------- | --------------------------------------------------------- | ------------ |
| MEM-001 | Native Polars `dt` operations instead of `map_elements()` | `parquet.py` |
| MEM-004 | Pre-flight memory estimate via `_COMPRESSION_RATIO`       | `parquet.py` |

MEM-001: Uses native Polars `dt` operations instead of `map_elements()` for timestamp manipulation — no Python map over rows. With daily partitioning (Issue #277), per-file data volume is small, but the principle remains: never use `map_elements()` for timestamp operations.

MEM-004: `storage.estimate_memory_mb(symbol, start_ts, end_ts)` returns an estimate before loading. If estimate exceeds budget, raises `MemoryError` with suggestion to use streaming path.

---

## Storage Layout

```
~/.cache/opendeviationbar/
├── ticks/
│   ├── BINANCE_SPOT_BTCUSDT/
│   │   ├── 2018-01-17.parquet    # daily files (Issue #277)
│   │   ├── 2018-01-18.parquet
│   │   └── ...
│   └── BINANCE_SPOT_ETHUSDT/
│       └── ...
├── checkpoints/               # populate_cache_resumable() state
└── heartbeat/                 # pending_messages.json, stathera_counts.json
```

Cache key format: `{SOURCE}_{MARKET}_{SYMBOL}` (uppercased). Examples: `BINANCE_SPOT_BTCUSDT`, `BINANCE_UM_BTCUSDT`.

---

## When the Parquet Cache Is Used

| Consumer                              | When                                 | Notes                                 |
| ------------------------------------- | ------------------------------------ | ------------------------------------- |
| `get_open_deviation_bars()`           | `has_ticks()` returns True           | Skips network download                |
| `populate_cache_resumable()`          | Before each day's Vision download    | Skips Vision if day already cached    |
| `kintsugi._shard_is_parquet_cached()` | Before each repair pass              | Switches to MAX_CONCURRENT_PARQUET=64 |
| `fetch_ticks_with_backoff()`          | All count-bounded backoff iterations | Reads cached months, fetches new ones |

**Multi-threshold efficiency**: Once BTCUSDT@250 populates the cache for a date range, BTCUSDT@500/750/1000 on the same dates reads from local Parquet — zero additional network downloads. Same raw aggTrades, different processor threshold.

---

## Key Properties

- **Daily files** (`YYYY-MM-DD.parquet`), one per UTC day
- **`has_ticks()` is O(1)** — `Path.exists()` per day, no Parquet metadata scan
- **Threshold-agnostic**: All thresholds (250, 500, 750, 1000) use the same raw aggTrades files. BTCUSDT@250 populates the cache; @500/750/1000 reuse it with zero re-downloads
- **`write_ticks()`** normalizes microsecond→millisecond timestamps and sorts by `agg_trade_id`
- **Symlinks**: `BINANCE_SPOT_BTCUSDT/ → BTCUSDT/` bridges the naming convention between seeder/checkpoint (`BINANCE_SPOT_` prefix) and canonical dirs (bare symbol)

### Anti-Pattern

**Do not check ClickHouse bar existence to determine Parquet tick availability.** Parquet ticks and ClickHouse bars are independent stores. A day may have ticks cached (raw aggTrades) but no bars (not yet processed), or bars but no ticks (processed from REST, ticks not persisted). Always use `storage.has_ticks()` for tick presence.

---

## Dead-Letter Store (Adjacent)

`/tmp/opendeviationbar-dead-letter/*.parquet` — bars that failed ClickHouse writes. NOT managed by `TickStorage`. Managed by Charon (periodic replay) and Niffler (immediate replay). Each file is named by `{symbol}-{threshold}-{timestamp}.parquet`.

---

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) — parent: caching architecture, MEM guards
- [/python/opendeviationbar/orchestration/CLAUDE.md](/python/opendeviationbar/orchestration/CLAUDE.md) — orchestration layer that reads/writes ticks
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) — pueue Parquet seeder (seeder service), kintsugi parallelism
- [/CLAUDE.md](/CLAUDE.md) — project hub
