Metadata-Version: 2.4
Name: quantmine
Version: 0.2.0
Summary: Equity factor research library with honest statistics: Newey-West IC tests, multiple-testing control, point-in-time universe, turnover-based costs, Carhart attribution
License: MIT
License-File: LICENSE
Requires-Python: >=3.13
Requires-Dist: numpy>=2.4
Requires-Dist: pandas>=3.0
Requires-Dist: pyarrow>=24.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: scipy>=1.17
Requires-Dist: statsmodels>=0.14
Provides-Extra: data
Requires-Dist: beautifulsoup4>=4.14; extra == 'data'
Requires-Dist: curl-cffi>=0.15; extra == 'data'
Requires-Dist: pandas-datareader>=0.11; extra == 'data'
Requires-Dist: yfinance>=1.4; extra == 'data'
Provides-Extra: viz
Requires-Dist: plotly>=6.8; extra == 'viz'
Description-Content-Type: text/markdown

# quantmine

An equity factor research library built around one principle: **statistical
honesty**. Every step that commonly inflates backtest results — survivorship
bias, overlapping-return autocorrelation, multiple testing, look-ahead in
orthogonalization, unrealistic transaction costs — is explicitly addressed,
and conclusions are reported with their uncertainty, not just their point
estimates.

The repo doubles as a full S&P 500 research case study: the library
(`quantmine/`), the daily Airflow pipeline (`pipelines/`), and the findings
below were produced by the same code you can `pip install`.

## Install

```bash
pip install quantmine          # library only
pip install "quantmine[data]"  # + yfinance download stack
```

For development (repo checkout, Python ≥ 3.13, [uv](https://docs.astral.sh/uv/)):

```bash
uv sync
python -m pytest test
```

## Quick start

Bring your own price data — any wide DataFrame (index = trading days,
columns = tickers) works; nothing is hard-wired to yfinance or the S&P 500:

```python
import pandas as pd
import quantmine as qf

# 1) Wrap your data (or use qf.ParquetSource / qf.YFinanceSource)
data = qf.MarketData(close=close_df, volume=volume_df)

# 2) Register a custom factor — built-in factors register automatically
@qf.factor_register("my_reversal")
def my_reversal(close: pd.DataFrame, tickers: list) -> pd.DataFrame:
    return -close[tickers].pct_change(5)

# 3) Compute all registered factors (dependencies resolve automatically)
pool = qf.build_param_pool(data, day=5, halflife=10, period=20)
failed, factors = qf.calculate_all_factors(pool)

# 4) Cross-sectional IC with Newey-West t-stats and multiple-testing control
fwd = qf.forward_return(data.close, periods=[1, 5, 20])
cs_ic = qf.CS_Information_Correlation(factors, fwd, output_path="cs_ic.parquet")
report = qf.multiple_testing(qf.newey_west_summary(cs_ic))

# 5) Quantile backtest on a point-in-time universe, turnover-based costs
universe = qf.MembershipTableSource(membership_df)   # or qf.StaticUniverse([...])
results, history = qf.quantile_backtest(universe, factors, ["my_reversal"], fwd)
daily = qf.expand_all_to_daily_returns(history, data.close)

# 6) Carhart four-factor attribution of the long-short returns (daily, HAC)
french = qf.load_french_factors("ff3_daily.csv", "momentum_daily.csv")
model = qf.carhart_attribution(daily[("my_reversal", 20)]["long_short"], french)
```

### Extension points

| Protocol / hook | Purpose | Ships with |
|---|---|---|
| `DataSource.load(...) -> MarketData` | plug in any price/volume source | `ParquetSource`, `CSVSource`, `ExcelSource`, `YFinanceSource` |
| `ConstituentsSource.get_constituents(date) -> set` | point-in-time universe from any provider | `MembershipTableSource` (interval table), `StaticUniverse` |
| `@factor_register(name)` | add factors; params injected by name, factor-on-factor dependencies resolved | 8 built-in factors |
| `config.example.yaml` | every pipeline parameter as validated dataclasses via `qf.load_configs` | defaults documented in-file |

## Headline result (S&P 500 case study)

The 20-day average volume factor (`TwentyDayAvgVol`) is the only candidate
that survives the full testing gauntlet:

| Stage | Result |
|---|---|
| Train IC (2015–2023, Newey-West) | t = 3.78 (20d holding), passes Bonferroni & Benjamini-Hochberg across all 18 factor × horizon tests |
| Out-of-sample quintiles (2024–2026) | Monotonic (Spearman 0.9); long-short gross ~14 %/yr (Sharpe ~1.5), ~1.4 Sharpe net of turnover-based costs |
| Carhart 4-factor attribution (daily, HAC, net of costs) | Market beta 0.24 (significant), large-cap tilt (SMB −0.13); momentum & value loadings insignificant |
| Net alpha | ~10 %/yr net of costs (t = 1.78, p ≈ 0.07, n = 603 daily obs) — economically meaningful, **marginally short of the 5 % significance bar** on 2.4 years of out-of-sample data |

The honest conclusion: the factor's IC is robustly significant in-sample under
conservative testing; its out-of-sample net alpha is economically meaningful
but does not clear conventional significance. Live verification over a longer
window is required — and that is exactly what a research report should say.
(A low-power period-level regression, n ≈ 30, says nothing either way —
p ≈ 0.38 with a confidence interval wide enough to hold any conclusion. Test
power and test bookkeeping move the verdict as much as the signal does.)

The remaining seven candidate factors (momentum, short-term reversal,
volatility, downside volatility, volume-price correlation, …) fail the
corrected significance tests. Documenting *why* they fail is part of the
point.

## Methodology highlights

- **Survivorship-bias correction** — the universe is rebuilt from historical
  S&P 500 membership (764 tickers over 2015–2026); 569 were recoverable via
  yfinance, and the residual gap is disclosed rather than hidden.
- **Point-in-time universe** — each backtest cross-section only contains
  stocks that were actually index members on that date.
- **Newey-West IC tests** — daily ICs on overlapping k-day forward returns are
  autocorrelated; plain `t = IR·√n` overstates significance several-fold. NW
  (Bartlett kernel, lag = 2(k−1)) uses all daily observations while correcting
  the standard error. A down-sampled IID test is kept as a robustness control.
- **Multiple-testing control** — Bonferroni and Benjamini-Hochberg across all
  factor × holding-period combinations.
- **Train/test split with embargo** — factors are selected and the
  orthogonalization is fit on 2015–2023 only; a gap of one month before the
  test window prevents overlapping forward returns from leaking across the
  split.
- **Expanding-window orthogonalization** — correlated factors are residualized
  with betas estimated on data available up to each date (no full-sample
  look-ahead).
- **Turnover-based transaction costs** — costs are charged on actual
  membership turnover per rebalance, not on a flat 100 %-turnover assumption.
- **Sanity checks** — factor displacement and cross-sectional shuffling tests
  confirm the backtest machinery itself is not the source of the returns.
- **Tested** — the research chain is covered by a unit + golden-value test
  suite (`test/`), including hand-computed backtest fixtures, determinism
  checks, and point-in-time universe edge cases.

## Pipeline

```
yfinance (batch download, retry, blacklist, checkpoints)
        │  quantmine/data_acquisition.py · pipelines/task_1.py
        ▼
cleaning & merge (ffill, dedup)          Airflow DAG: pipelines/DAG_pipeline.py
        │  pipelines/task_2.py
        ▼
factor computation (registry-driven, vectorized pandas)
        │  quantmine/factor_mining.py · pipelines/task_3.py
        ▼
IC testing: cross-sectional & time-series IC, NW t, BH/Bonferroni,
train/test split, orthogonalization        quantmine/ic_calculator.py
        ▼
quintile backtest: PIT universe, monotonicity, turnover costs,
displacement/shuffle sanity tests          quantmine/back_testing.py
        ▼
Carhart 4-factor attribution (daily, HAC)  quantmine/factor_attribution.py
```

Repository layout:

```
quantmine/          research library (importable package)
pipelines/            Airflow DAG + daily CLI tasks
test/                 pytest suite (unit + golden-value)
config.example.yaml   all pipeline parameters, documented defaults
```

## Reproducing the case study

**Market data is not included** (Yahoo Finance terms of service do not permit
redistribution). To reproduce:

1. Historical S&P 500 membership: provide a CSV with `ticker`, `start_date`,
   `end_date` columns and point `SP500_MEMBERSHIP_CSV` at it.
2. Prices/volumes: `python pipelines/task_1.py --date <ds> --batch manual`
   downloads in batches with checkpointing, then `pipelines/task_2.py` cleans
   and merges.
3. Fama-French factors: download the daily FF3 and momentum CSVs from the
   [Ken French Data Library](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html)
   into `tmp/ff3/`.
4. Run the research chain: `python pipelines/task_3.py ...` for factors, then
   the IC → backtest → attribution steps as in the quick start (or
   `python -m quantmine.ic_calculator` for the packaged train/test workflow).

For the Airflow DAG, set `QUANT_PROJECT_ROOT` and `QUANT_PYTHON_BIN` (see
`DAG_pipeline.py` docstring) and copy `airflow.cfg.example` keys into your own
config — never commit a real `airflow.cfg`.

## Known limitations

- ~195 of 764 historical members could not be recovered from yfinance (mostly
  true delistings/acquisitions), so a residual survivorship bias remains and
  likely flatters the results slightly.
- The out-of-sample window (2024–2026) covers a single market regime.
- The long-short portfolio carries a significant 0.24 market beta; a
  beta-hedged variant is on the roadmap.
- Transaction cost model is a flat per-turnover rate; no market-impact or
  borrow-cost modeling.

## Roadmap

- [ ] Migrate storage from parquet files to PostgreSQL/DuckDB
- [ ] REST API + MCP server exposing the research chain as agent-callable tools
- [ ] RAG-based automated research reports (ChromaDB + LLM)
- [ ] Extend the Airflow DAG to cover IC testing → backtest → reporting
- [ ] Rebuild the analytics dashboard (frontend rewrite in progress)
- [ ] Scale the data layer (full US market) with PySpark
- [ ] Beta-hedged long-short variant; GARCH volatility targeting

## License

MIT
