---
name: duckdb
description: Run SQL directly on pandas DataFrames, Parquet, and CSV files with DuckDB — window functions, joins, aggregations, and larger-than-memory scans without a separate database server. Use when a data task is more naturally expressed in SQL than pandas (rolling/cross-sectional ranks, multi-table joins of price×fundamentals×constituents, grouped aggregations) or when the data is too big to fit in memory.
---

# DuckDB skill

DuckDB (1.5.3) is an in-process analytical SQL engine. It queries pandas DataFrames, Parquet, and CSV **in place** — no server, no import step — and can scan datasets larger than RAM by streaming from Parquet. Ideal for the HS300 factor lab where daily/minute panels often exceed memory and many operations (rolling IC, cross-sectional rank, panel joins) read cleaner in SQL.

## Core idea: query DataFrames by variable name

```python
import duckdb, pandas as pd
prices = pd.read_parquet("prices.parquet")
# the DataFrame `prices` is queryable by its python variable name — zero copy
df = duckdb.sql("SELECT date, code, close FROM prices WHERE close > 10").df()
```
`duckdb.sql(...)` returns a relation; finish with `.df()` (→ pandas), `.pl()` (→ Polars),
`.fetchall()` (→ list of tuples), or `.fetchone()`.

## Query Parquet/CSV without loading first (larger-than-memory)

```python
# scans the file lazily — never materializes the whole thing in RAM
duckdb.sql("SELECT code, AVG(ret) FROM 'data/daily/*.parquet' GROUP BY code").df()
duckdb.sql("SELECT * FROM read_csv_auto('big.csv') LIMIT 5").df()
```
Glob patterns work (`'data/**/*.parquet'`). Push filters/aggregations into SQL so only
the needed columns/rows are read.

## Quant patterns (why this beats pandas here)

**Cross-sectional rank per day (factor → percentile):**
```sql
SELECT date, code, factor,
       (PERCENT_RANK() OVER (PARTITION BY date ORDER BY factor)) AS pctile
FROM panel
```

**Rolling N-day mean / z-score per code:**
```sql
SELECT date, code, ret,
       AVG(ret)  OVER w AS roll_mean,
       STDDEV(ret) OVER w AS roll_std
FROM panel
WINDOW w AS (PARTITION BY code ORDER BY date
             ROWS BETWEEN 19 PRECEDING AND CURRENT ROW)
```

**Join price × fundamentals × constituents:**
```sql
SELECT p.date, p.code, p.close, f.pb, f.roe
FROM prices p
JOIN fundamentals f USING (code, date)
JOIN hs300_constituents c ON c.code = p.code AND p.date BETWEEN c.in_date AND c.out_date
```

**Quantile-group forward return (decile backtest):**
```sql
SELECT date, NTILE(10) OVER (PARTITION BY date ORDER BY factor) AS decile, fwd_ret
FROM panel
```

## Helper script

- **`scripts/duck_query.py`** — Run an ad-hoc SQL query against CSV/Parquet files (or a directory glob) from the command line and print/save the result. Register files as named tables.

```bash
python scripts/duck_query.py --table prices=data/prices.parquet \
    --sql "SELECT code, AVG(ret) r FROM prices GROUP BY code ORDER BY r DESC LIMIT 10"
python scripts/duck_query.py --table p='data/daily/*.parquet' \
    --sql "SELECT count(*) FROM p" --out result.csv
```

## Persistent database (optional)

```python
con = duckdb.connect("factor_lab.duckdb")     # file-backed, survives restarts
con.execute("CREATE TABLE ic AS SELECT * FROM read_parquet('ic.parquet')")
con.sql("SELECT * FROM ic").df()
con.close()
```

## Gotchas

- A DataFrame is only visible to `duckdb.sql` if it's an **in-scope variable** at call time; inside a function pass it explicitly or use `duckdb.sql(q).df()` in the same scope. For robustness use `con.register("name", df)` then query `name`.
- DuckDB is **case-insensitive** for identifiers and uses standard SQL `NULL` semantics — watch `NULL` vs pandas `NaN` (they map to each other but `NaN = NaN` is false in SQL).
- For reproducible results add explicit `ORDER BY`; DuckDB does not guarantee row order otherwise.
- `STDDEV` is sample stddev; use `STDDEV_POP` for population.
- Writing back: `duckdb.sql("...").df().to_parquet(...)` or `COPY (SELECT ...) TO 'out.parquet' (FORMAT parquet)`.
