Metadata-Version: 2.4
Name: alphadataforge
Version: 0.1.0
Summary: A modular data fetching and feature engineering pipeline for quantitative trading.
Project-URL: Homepage, https://github.com/RETCx/alphadataforge
Project-URL: Issues, https://github.com/RETCx/alphadataforge/issues
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: pandas
Requires-Dist: numpy
Requires-Dist: yfinance
Requires-Dist: tiingo
Requires-Dist: google-cloud-bigquery
Requires-Dist: requests
Requires-Dist: requests-cache
Requires-Dist: tenacity
Requires-Dist: scikit-learn
Requires-Dist: python-dotenv
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-vcr; extra == "dev"
Provides-Extra: publish
Requires-Dist: build; extra == "publish"
Requires-Dist: twine; extra == "publish"

# AlphaDataForge

AlphaDataForge is a modular data fetching pipeline for quantitative trading and financial analysis. It provides a unified interface to fetch **Price** and **Fundamental** data across multiple providers with automatic caching, rate limit protection, and standardized output formats.

---

## Quick Start

After the first PyPI release is published, install with:

```bash
python -m pip install alphadataforge
```

For development or before the first release:

```bash
git clone https://github.com/RETCx/alphadataforge.git
cd alphadataforge
pip install -e .
```

Maintainers: see [PUBLISHING.md](PUBLISHING.md) for release instructions.

Copy the environment template and add your API keys:
```bash
cp .env.example .env
```

```
# .env
TIINGO_API_KEY="your_tiingo_key"
ALPHAVANTAGE_API_KEY="your_alphavantage_key"
FMP_API_KEY="your_fmp_key"
STOCKFIT_API_KEY="your_stockfit_key"
```

*YFinance does not require an API key.*

---

## Architecture

```
alphadataforge/
├── data/
│   ├── price.py          ← Unified Price Facade (Price.get(), Price.compare(), Price.consensus())
│   └── fundamental.py    ← Unified Fundamental Facade (Fundamentals.get_info(), etc.)
│
├── providers/
│   ├── yfinance_fetcher.py
│   ├── alphavantage_fetcher.py
│   ├── tiingo_fetcher.py
│   ├── fmp_fetcher.py
│   ├── stockfit_fetcher.py ← StockFit API (20yr SEC EDGAR XBRL lineage, health scores)
│   └── hybrid_fetcher.py  ← AlphaVantage (raw) + Tiingo (div/splits) + local math engine
│
├── scripts/
│   └── snapshot_fundamentals.py ← Offline snapshot utility (Parquet & SQLite)
│
├── utils/
│   └── finance_math.py   ← Local back-adjustment engine (calculate_adjusted_prices)
│
└── core/
    ├── base_fetcher.py   ← BaseDataFetcher (caching, retry, normalization)
    └── exceptions.py     ← AlphaDataForgeError, RateLimitExceededError, etc.
```

**Built-in features (all providers)**:
- **SQLite Caching** (24h TTL) — auto-stored in OS temp folder, no setup needed.
- **Auto-Retry** with exponential backoff for network errors (via `tenacity`).
- **Normalized Output** — all prices return `[Open, High, Low, Close, Volume, Adj Close]` with a `DatetimeIndex`. All financials return standardized columns (`Net_Income`, `Total_Revenue`, etc.).

---

## Module 1: Price

**Entry point:** `from alphadataforge.data.price import Price`

```python
from alphadataforge.data.price import Price

# Single symbol
df = Price.get("AAPL", start_date="2023-01-01", end_date="2023-12-31")

# Multiple symbols (returns dict)
data = Price.get(["AAPL", "GOOGL", "MSFT"], provider="yfinance")

# Specific provider
df = Price.get("TSLA", provider="tiingo", start_date="2022-01-01")
```

### `Price.get()` — All Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `symbols` | `str` or `List[str]` | *required* | Ticker(s) e.g. `"AAPL"` or `["AAPL", "MSFT"]` |
| `provider` | `str` | `"yfinance"` | `"yfinance"`, `"alphavantage"`, `"tiingo"`, `"fmp"`, `"hybrid_av_tiingo"` |
| `start_date` | `str` | `None` | `"YYYY-MM-DD"` |
| `end_date` | `str` | `None` | `"YYYY-MM-DD"` |
| `provider_params` | `dict` | `{}` | Extra params passed to the underlying provider (see per-provider sections below) |

**Returns:** `pd.DataFrame` (single symbol) or `Dict[str, pd.DataFrame]` (multiple symbols)

**Output Columns:**

| Column | Type | Description |
|---|---|---|
| `Open` | float | Opening price |
| `High` | float | Daily high |
| `Low` | float | Daily low |
| `Close` | float | Closing price |
| `Volume` | float | Trading volume |
| `Adj Open` | float | Adjusted opening price (if `adjusted=True`) |
| `Adj High` | float | Adjusted daily high (if `adjusted=True`) |
| `Adj Low` | float | Adjusted daily low (if `adjusted=True`) |
| `Adj Close` | float | Dividend & split adjusted close (if available or `adjusted=True`) |
| `Adj Volume` | float | Split-adjusted volume (if `adjusted=True`) |

> **Note on Adjusted Prices:** If `adjusted=True` is passed (or if the provider returns it natively), AlphaDataForge guarantees the presence of full Adjusted OHLC columns (`Adj Open`, `Adj High`, `Adj Low`, `Adj Close`). For providers like YFinance that only natively return `Adj Close`, our pipeline automatically backfills the remaining adjusted columns using the ratio of `Adj Close` / `Close`.

### Hybrid Provider (`hybrid_av_tiingo`)
AlphaDataForge offers a special `hybrid_av_tiingo` provider for optimal data quality and API usage:
- **Raw Prices:** Fetched from Alpha Vantage (deepest historical data, 20+ years).
- **Dividends & Splits:** Fetched from Tiingo (highly accurate, saves Alpha Vantage premium limits).
- **Processing:** Fully adjusted prices are calculated locally via our `finance_math` engine.

### Cross-Provider Adjusted Price Comparison
To ensure data consistency across all platforms, we tested the fully adjusted prices for `AAPL` on `2026-05-05` across all available providers and the hybrid engine. The results perfectly align:

**Adjusted Close:**
- `yfinance`: 283.9184
- `tiingo`: 283.9181
- `fmp`: 283.9200
- `alphavantage`: 283.9184
- `hybrid_av_tiingo`: 283.9184

**Adjusted Open:**
- `yfinance`: 276.6751
- `tiingo`: 276.6698
- `fmp`: 276.6800
- `alphavantage`: 276.6701
- `hybrid_av_tiingo`: 276.6701

### `Price.compare()` — Side-by-Side Multi-Provider

Fetches from multiple providers **concurrently** (ThreadPoolExecutor) and returns a single **MultiIndex DataFrame** with providers as level-0 columns.

```python
# Compare all 5 providers with adjusted prices
multi_df = Price.compare("AAPL", start_date="2023-01-01", provider_params={"adjusted": True})

# Access a specific provider's data
yfinance_df = multi_df["yfinance"]

# Extract one column across all providers
adj_close_all = multi_df.xs("Adj Close", axis=1, level=1)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `symbol` | `str` | *required* | Single ticker |
| `start_date` | `str` | `None` | `"YYYY-MM-DD"` |
| `end_date` | `str` | `None` | `"YYYY-MM-DD"` |
| `providers` | `List[str]` | all 5 providers | Limit which providers to query |
| `provider_params` | `dict` | `{}` | Passed to all providers (e.g. `{"adjusted": True}`) |
| `custom_data` | `Dict[str, DataFrame]` | `None` | Inject your own DataFrames to compare alongside API data |

**Returns:** `pd.DataFrame` with MultiIndex columns `(provider, column_name)`

---

### `Price.consensus()` — Aggregated Median/Mean Price

Builds on `compare()` and aggregates across all providers to return a **single reliable DataFrame**, neutralizing outliers from any one source.

```python
# Median across all providers (recommended — more robust to outliers)
consensus_df = Price.consensus("AAPL", start_date="2023-01-01", method="median")

# Limit to specific providers
consensus_df = Price.consensus(
    "AAPL",
    providers=["yfinance", "tiingo", "alphavantage"],
    provider_params={"adjusted": True},
    method="median"
)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `symbol` | `str` | *required* | Single ticker |
| `start_date` | `str` | `None` | `"YYYY-MM-DD"` |
| `end_date` | `str` | `None` | `"YYYY-MM-DD"` |
| `providers` | `List[str]` | all 5 providers | Limit which providers to query |
| `provider_params` | `dict` | `{}` | Passed to all providers |
| `custom_data` | `Dict[str, DataFrame]` | `None` | Inject your own DataFrames into the aggregation |
| `method` | `str` | `"median"` | `"median"` (robust to outliers) or `"mean"` |

**Returns:** `pd.DataFrame` with standard OHLCV columns sorted in standard order

#### Injecting Custom External Data (`custom_data`)

Both `compare()` and `consensus()` accept a `custom_data` parameter — a dictionary of your own DataFrames to include in the comparison or aggregation. Your DataFrame must have a `DatetimeIndex` and column names that match the AlphaDataForge standard (`Open`, `High`, `Low`, `Close`, `Volume`, `Adj Close`, etc.).

```python
import pandas as pd
from alphadataforge.data.price import Price
from alphadataforge.providers.tiingo_fetcher import TiingoFetcher
from alphadataforge.utils.finance_math import calculate_adjusted_prices

# Step 1: Load & Normalize column names to AlphaDataForge standard
df_custom = pd.read_csv('my_data.csv')
df_custom = df_custom.rename(columns={'Price': 'Close', 'Vol.': 'Volume'})
df_custom['Date'] = pd.to_datetime(df_custom['Date'])
df_custom.set_index('Date', inplace=True)

# Step 2 (Optional): Back-adjust with Tiingo dividends/splits if your data is raw
tf = TiingoFetcher()
df_tiingo = tf.fetch_single('AAPL', start_date='2023-01-01', end_date='2023-12-31')
dividends_df = df_tiingo[['divCash']].rename(columns={'divCash': 'Dividend'})
dividends_df = dividends_df[dividends_df['Dividend'] > 0]
splits_df = df_tiingo[['splitFactor']].rename(columns={'splitFactor': 'SplitFactor'})
splits_df = splits_df[splits_df['SplitFactor'] != 1.0]
df_custom_adjusted = calculate_adjusted_prices(df_custom, dividends_df, splits_df)

# Step 3: Feed into consensus engine
consensus_df = Price.consensus(
    "AAPL",
    start_date="2023-01-01",
    providers=["yfinance", "alphavantage"],
    provider_params={"adjusted": True},
    custom_data={"my_local_db": df_custom_adjusted},
    method="median"
)
```

---

## Module 2: Fundamentals

**Entry point:** `from alphadataforge.data.fundamental import Fundamentals`

### `Fundamentals.get_info()` — Company Profile

```python
info = Fundamentals.get_info("AAPL", provider="yfinance")
print(info["sector"], info["marketCap"])
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `symbol` | `str` | *required* | Ticker e.g. `"AAPL"` |
| `provider` | `str` | `"yfinance"` | `"yfinance"`, `"stockfit"`, `"alphavantage"`, `"fmp"` |

**Returns:** `dict`

---

### `Fundamentals.get_financials()` — Financial Statements

```python
df = Fundamentals.get_financials("AAPL", statement="income", period="annual", provider="yfinance")
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `symbol` | `str` | *required* | Ticker e.g. `"AAPL"` |
| `statement` | `str` | `"income"` | `"income"`, `"balance"`, `"cashflow"`, `"growth"`, `"scores"`, `"shares_outstanding"` |
| `period` | `str` | `"annual"` | `"annual"` or `"quarterly"` |
| `provider` | `str` | `"yfinance"` | `"yfinance"`, `"stockfit"`, `"alphavantage"`, `"fmp"` |

**Returns:** `pd.DataFrame` with `DatetimeIndex` and normalized columns

**Normalized Columns (Standardized Across All Providers):**

| Column | Description |
|---|---|
| `Net_Income` | Net profit/loss after tax |
| `Total_Revenue` | Total revenue |
| `Operating_Income` | EBIT / Operating profit |
| `Free_Cash_Flow` | Free cash flow |
| `Total_Assets` | Total assets |
| `Total_Liabilities` | Total liabilities |
| `Total_Equity` | Shareholders' equity |
| `Shares_Outstanding` | Basic shares outstanding |

---


### `Fundamentals.get_scores()` — Financial Health Scores (StockFit)

```python
scores = Fundamentals.get_scores("AAPL", provider="stockfit")
print("Piotroski F-Score:", scores["piotroskiFScore"])
# Returns Piotroski details: positiveNetIncome, positiveOperatingCashFlow, risingRoa, etc.
```

### SEC XBRL 20-Year Statement History (StockFit)

```python
# 20 years of annual statements with SEC filing audit lineage
df_income = Fundamentals.get_financials("AAPL", statement="income", period="annual", provider="stockfit", limit=100)

# 70+ quarters of balance sheet history
df_balance = Fundamentals.get_financials("AAPL", statement="balance", period="quarterly", provider="stockfit", limit=100)
```

### Snapshot & Offline Storage (`scripts/snapshot_fundamentals.py`)

Capture 20-year financial statement history for target symbols into local **Parquet** and **SQLite** storage:

```bash
# Snapshot Magnificent 7 (Annual + Quarterly, Profiles, Scores)
python scripts/snapshot_fundamentals.py --preset mag7

# Snapshot top 25 US blue-chips
python scripts/snapshot_fundamentals.py --preset top25

# Snapshot specific tickers
python scripts/snapshot_fundamentals.py --symbols AAPL,MSFT,NVDA,GOOGL
```
Data is saved to `data_store/stockfit_snapshots/parquet/` and `data_store/stockfit_snapshots/fundamentals.db`.

### `Fundamentals.get_earnings_calendar()` — Earnings Calendar

```python
df = Fundamentals.get_earnings_calendar(horizon="3month", symbol="AAPL", provider="alphavantage")
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `horizon` | `str` | `"3month"` | `"3month"`, `"6month"`, or `"12month"` |
| `symbol` | `str` | `None` | Filter for specific ticker (optional) |
| `provider` | `str` | `"alphavantage"` | Only `"alphavantage"` supported |

**Returns:** `pd.DataFrame` with columns: `symbol`, `name`, `reportDate`, `fiscalDateEnding`, `estimate`, `currency`

---

## Provider Reference

---

### Provider 1: YFinance (`provider="yfinance"`)

| | |
|---|---|
| **API Key** | No Not required |
| **Rate Limit** | Practically unlimited |
| **Best For** | Development, batch fetching, comprehensive fundamentals |

#### Price

```python
from alphadataforge.data.price import Price

# Basic fetch
df = Price.get("AAPL", start_date="2023-01-01")

# With intraday interval (available last 60 days only)
df = Price.get("AAPL", provider_params={"interval": "1h"})

# Crypto & Forex (use yfinance ticker format)
df = Price.get("BTC-USD")           # Bitcoin
df = Price.get("ETH-USD")           # Ethereum
df = Price.get("EURUSD=X")          # EUR/USD Forex
df = Price.get("^VIX")              # VIX Index
```

**Extra `provider_params` for YFinance Price:**

| Param | Type | Default | Description |
|---|---|---|---|
| `interval` | `str` | `"1d"` | `"1m"`, `"5m"`, `"15m"`, `"30m"`, `"1h"`, `"1d"`, `"1wk"`, `"1mo"`, `"3mo"`. Intraday (< 1d) only for last 60 days. |
| `auto_adjust` | `bool` | `False` | Pass `True` if you want yfinance to overwrite the Open/High/Low columns with adjusted prices. |

**Under the hood (YFinance):**
- **API Calls:** 1 call per fetch.
- **Adjusted Prices:** Natively returns `Adj Close`. If you want `Adj Open/High/Low`, you must pass `provider_params={"auto_adjust": True}`.

#### Fundamentals (YFinance)

| Method | `statement=` value | Notes |
|---|---|---|
| `get_info()` | — | Returns 80+ fields: `sector`, `industry`, `marketCap`, `trailingPE`, `forwardPE`, `dividendYield`, `beta`, `52WeekHigh`, `52WeekLow`, etc. |
| `get_financials()` | `"income"` | Up to 4 years annual / 8 quarters quarterly |
| `get_financials()` | `"balance"` | Up to 4 years annual / 8 quarters quarterly |
| `get_financials()` | `"cashflow"` | Up to 4 years annual / 8 quarters quarterly |

```python
from alphadataforge.data.fundamental import Fundamentals

# Company overview
info = Fundamentals.get_info("AAPL", provider="yfinance")
print(info["sector"])        # "Technology"
print(info["marketCap"])     # 3_000_000_000_000
print(info["trailingPE"])    # 33.2

# Financial Statements
df_income   = Fundamentals.get_financials("AAPL", statement="income",   period="annual",    provider="yfinance")
df_balance  = Fundamentals.get_financials("AAPL", statement="balance",  period="quarterly", provider="yfinance")
df_cashflow = Fundamentals.get_financials("AAPL", statement="cashflow", period="annual",    provider="yfinance")

# Merge all three by date index
df_all = df_income.join(df_balance, how="outer", rsuffix="_bal").join(df_cashflow, how="outer", rsuffix="_cf")
```

---

### Provider 2: Alpha Vantage (`provider="alphavantage"`)

| | |
|---|---|
| **API Key** | Yes Required (`ALPHAVANTAGE_API_KEY`) |
| **Rate Limit (Free)** | 25 requests / day, 1 request / second |
| **Auto-Sleep** | Yes Built-in 1.2s delay between each request (no manual sleep needed) |
| **Data Depth** | Up to 20 years of price history, full SEC-normalized financials |
| **Best For** | Deep historical financial statements, Earnings Calendar |

> Warning **25 requests/day** is the hard ceiling on the free plan. Each call to `get_financials()` or `get_info()` costs 1 request. Plan accordingly.

#### Price

```python
from alphadataforge.data.price import Price

# Basic EOD price (compact = last 100 days)
df = Price.get("AAPL", provider="alphavantage")

# Full history (~20 years)
df = Price.get("AAPL", provider="alphavantage", provider_params={"outputsize": "full"})

# Adjusted prices (costs 3 API calls: Price + Dividends + Splits)
df = Price.get("AAPL", provider="alphavantage", provider_params={"adjusted": True, "outputsize": "full"})
```

**Extra `provider_params` for AlphaVantage Price:**

| Param | Type | Default | Description |
|---|---|---|---|
| `outputsize` | `str` | `"compact"` | `"compact"` (last 100 days) or `"full"` (full ~20yr history). |
| `adjusted` | `bool` | `False` | Fetches dividends/splits and back-adjusts all OHLCV. |

**Under the hood (Alpha Vantage):**
- **API Calls (`adjusted=False`):** 1 call (`TIME_SERIES_DAILY`).
- **API Calls (`adjusted=True`):** **3 calls** (`TIME_SERIES_DAILY`, `DIVIDENDS`, `SPLITS`).
- **Adjusted Prices:** Since Alpha Vantage made their adjusted endpoint premium, we built a custom math engine. When `adjusted=True`, we fetch raw prices, dividends, and splits on the free endpoints and calculate `Adj Open`, `Adj High`, `Adj Low`, `Adj Close`, `Adj Volume` entirely locally. You get premium-grade adjusted data for free!

#### Fundamentals (AlphaVantage)

| Method | `statement=` value | Notes |
|---|---|---|
| `get_info()` | — | Maps to `OVERVIEW` endpoint. Returns company ratios, P/E, EPS, dividends, 52-week data etc. |
| `get_financials()` | `"income"` | SEC-normalized annual or quarterly income statement |
| `get_financials()` | `"balance"` | SEC-normalized annual or quarterly balance sheet |
| `get_financials()` | `"cashflow"` | SEC-normalized annual or quarterly cash flow |
| `get_financials()` | `"shares_outstanding"` | Historical quarterly shares (basic + diluted). Note: `period` param ignored — always quarterly. |
| `get_financials()` | `"earnings"` | Annual/quarterly EPS history with analyst estimates and surprise % |
| `get_earnings_calendar()` | — | Upcoming earnings dates for next 3/6/12 months |

```python
from alphadataforge.data.fundamental import Fundamentals

# Company overview (maps to OVERVIEW endpoint)
info = Fundamentals.get_info("IBM", provider="alphavantage")

# Income Statement (annual, goes back 5+ years)
df_income = Fundamentals.get_financials("IBM", statement="income", period="annual", provider="alphavantage")

# Balance Sheet (quarterly)
df_balance = Fundamentals.get_financials("IBM", statement="balance", period="quarterly", provider="alphavantage")

# Cash Flow
df_cf = Fundamentals.get_financials("IBM", statement="cashflow", period="annual", provider="alphavantage")

# Historical Shares Outstanding (quarterly, to track buybacks/dilution)
df_shares = Fundamentals.get_financials("IBM", statement="shares_outstanding", provider="alphavantage")
# Returns: Shares_Outstanding (basic), shares_outstanding_diluted

# Earnings History with analyst surprise
df_earnings = Fundamentals.get_financials("IBM", statement="earnings", period="quarterly", provider="alphavantage")
# Returns: reportedEPS, estimatedEPS, surprise, surprisePercentage, reportTime

# Earnings Calendar (upcoming earnings dates)
df_cal = Fundamentals.get_earnings_calendar(horizon="3month", provider="alphavantage")             # all upcoming
df_cal = Fundamentals.get_earnings_calendar(horizon="12month", symbol="AAPL", provider="alphavantage")  # specific ticker
```

---

### Provider 3: Tiingo (`provider="tiingo"`)

| | |
|---|---|
| **API Key** | Yes Required (`TIINGO_API_KEY`) |
| **Rate Limit (Free)** | 500 requests / hour (very generous) |
| **Best For** | Clean institutional-grade EOD prices, News API with tag filtering |

#### Price

```python
from alphadataforge.data.price import Price

# Basic EOD price
df = Price.get("AAPL", provider="tiingo", start_date="2020-01-01")

# With weekly frequency
df = Price.get("AAPL", provider="tiingo", provider_params={"frequency": "weekly"})

# Adjusted prices (natively returned — no extra API calls)
df = Price.get("AAPL", provider="tiingo", provider_params={"adjusted": True})
```

**Extra `provider_params` for Tiingo Price:**

| Param | Type | Default | Description |
|---|---|---|---|
| `frequency` | `str` | `"daily"` | `"daily"`, `"weekly"`, `"monthly"`, `"annually"`. Intraday requires paid plan. |
| `adjusted` | `bool` | `False` | Returns `Adj Open`, `Adj High`, `Adj Low`, `Adj Close`, `Adj Volume` (natively — no extra API calls). |

**Under the hood (Tiingo):**
- **API Calls:** 1 call per fetch.
- **Adjusted Prices:** The Tiingo API natively sends a full suite of adjusted prices. The resulting DataFrame will always contain `Adj Open`, `Adj High`, `Adj Low`, `Adj Close`, and `Adj Volume`. No manual calculation needed.

#### Crypto (Tiingo)

```python
# Tiingo crypto uses different ticker format (no dash): BTCUSD, ETHUSD
from alphadataforge.providers.tiingo_fetcher import TiingoFetcher

tiingo = TiingoFetcher()
crypto_data = tiingo.fetch_crypto(
    tickers=["BTCUSD", "ETHUSD"],
    start_date="2023-01-01",
    resample_freq="1Day"   # "1Day", "1Hour", "30Min"
)
```

#### News (Tiingo)

Tiingo has the most powerful news filtering of all providers.

```python
from alphadataforge.providers.tiingo_fetcher import TiingoFetcher

tiingo = TiingoFetcher()
news_df = tiingo.fetch_news(
    tickers=["AAPL", "MSFT"],         # Filter by ticker (optional)
    tags=["Earnings"],                 # Filter by tag (optional)
    sources=["reuters.com"],           # Filter by source domain (optional)
    start_date="2024-01-01",
    end_date="2024-06-01",
    limit=50
)
# Returns columns: publishedDate, title, description, url, tickers, tags
```

#### Fundamentals (Tiingo)

| Method | Support | Notes |
|---|---|---|
| `get_info()` | Yes Supported | Returns basic ticker metadata (name, description, exchange, startDate) |
| `get_financials()` | No Not Supported | Returns empty DataFrame. Full financial statements require Tiingo's paid `fundamentals` plan. |

---

### Provider 4: FMP — Financial Modeling Prep (`provider="fmp"`)

| | |
|---|---|
| **API Key** | Yes Required (`FMP_API_KEY`) |
| **Rate Limit (Free)** | 250 requests / day |
| **Best For** | EOD Price data (free), Deep fundamental history (paid) |

> Warning **Important:** Financial statements (`get_financials`) are **locked behind FMP's paid plan** and will return a `403 Forbidden` error with a free API key. `get_info()` and `Price.get()` work on the free tier.

#### Price

```python
from alphadataforge.data.price import Price

# Basic EOD price
df = Price.get("AAPL", provider="fmp", start_date="2023-01-01")

# Dividend & split adjusted
df = Price.get("AAPL", provider="fmp", provider_params={"adjusted": True})
```

**Extra `provider_params` for FMP Price:**

| Param | Type | Default | Description |
|---|---|---|---|
| `adjusted` | `bool` | `False` | `False` = raw non-split-adjusted prices. `True` = fully dividend & split adjusted. |

**Under the hood (FMP):**
- **API Calls:** 1 call per fetch (regardless of `adjusted` flag).
- **Adjusted Prices:** If `adjusted=True`, we hit FMP's `dividend-adjusted` endpoint, which natively returns adjusted values for Open, High, Low, and Close. The resulting DataFrame will contain `Adj Open`, `Adj High`, `Adj Low`, and `Adj Close`.

#### Fundamentals (FMP)

| Method | `statement=` value | Free Tier | Notes |
|---|---|---|---|
| `get_info()` | — | ✅ Free | Returns company profile: `sector`, `industry`, `mktCap`, `beta`, `description`, `ceo`, etc. |
| `get_financials()` | `"income"` | ✅ Free | 5 periods returned on free tier |
| `get_financials()` | `"balance"` | ✅ Free | 5 periods returned on free tier |
| `get_financials()` | `"cashflow"` | ✅ Free | 5 periods returned on free tier |
| `get_financials()` | `"shares_float"` | ✅ Free | Returns `freeFloat`, `floatShares`, and `outstandingShares` snapshot |

```python
from alphadataforge.data.fundamental import Fundamentals

# Company profile (works on free tier)
info = Fundamentals.get_info("AAPL", provider="fmp")
print(info["sector"])       # "Technology"
print(info["ceo"])          # "Timothy D. Cook"

# Financial statements (PAID ONLY — will raise error on free key)
# df = Fundamentals.get_financials("AAPL", statement="income", provider="fmp")
```

---

---

### Provider 5: StockFit (`provider="stockfit"`)

| | |
|---|---|
| **API Key** | Yes Required (`STOCKFIT_API_KEY`) |
| **Rate Limit** | 500 requests / minute, 500,000 / day (Pro Plan) |
| **Data Source** | Direct SEC EDGAR XBRL filings with point-in-time audit lineage |
| **Best For** | Deep historical US financial statements (20 years), Piotroski Health Scores, Offline Snapshots |

#### Financial Statements (20-Year SEC XBRL)

StockFit parses official 10-K and 10-Q filings directly, providing up to 20 years of annual statements and 70+ quarters:

```python
from alphadataforge.data.fundamental import Fundamentals

# 1. Income Statement (20 years)
df_income = Fundamentals.get_financials("AAPL", statement="income", period="annual", provider="stockfit", limit=100)

# 2. Balance Sheet (70+ quarters)
df_balance = Fundamentals.get_financials("AAPL", statement="balance", period="quarterly", provider="stockfit", limit=100)

# 3. Cash Flow
df_cf = Fundamentals.get_financials("AAPL", statement="cashflow", period="annual", provider="stockfit")

# 4. Financial Health Scores (Piotroski F-Score)
scores = Fundamentals.get_scores("AAPL", provider="stockfit")
print("Piotroski F-Score:", scores["piotroskiFScore"])
```

#### Cross-Provider Accuracy Benchmark (AAPL)

We tested and compared StockFit's normalized financials against YFinance and AlphaVantage:

| Metric (2025) | StockFit | YFinance | AlphaVantage | Alignment |
|---|---|---|---|---|
| `Total_Revenue` | $416,161,000,000 | $416,161,000,000 | $416,161,000,000 | **100% Identical** |
| `Net_Income` | $112,010,000,000 | $112,010,000,000 | $112,010,000,000 | **100% Identical** |
| `Operating_Income` | $133,050,000,000 | $133,050,000,000 | $133,050,000,000 | **100% Identical** |
| `Total_Assets` | $359,241,000,000 | $359,241,000,000 | $359,241,000,000 | **100% Identical** |
| `Free_Cash_Flow` | $98,767,000,000 | $98,767,000,000 | N/A | **100% Identical** |
| **History Depth** | **19–20 Years** | 4 Years | ~5 Years | **StockFit 5x Deeper** |

## Provider Comparison Table

### Price

| Feature | YFinance | AlphaVantage | Tiingo | FMP | Hybrid (AV+Tiingo) |
|---|---|---|---|---|---|
| API Key Required | No | Yes | Yes | Yes | Yes (both AV + Tiingo) |
| Rate Limit | Unlimited | 25/day | 500/hr | 250/day | 3 AV calls + 2 Tiingo calls |
| Adjusted Prices | Yes | Yes (3 calls) | Yes (native) | Yes (native) | Yes (local math engine) |
| Intraday | Yes (last 60d) | No | Paid only | No | No |
| Crypto | Yes (`BTC-USD`) | No | Yes (`BTCUSD`) | No | No |
| Forex | Yes (`EURUSD=X`) | No | No | No | No |
| Historical Depth | ~20yr | ~20yr | ~30yr | ~30yr | ~20yr |
| Batch Fetching | Yes (1 call) | ⚠️ 1 call/symbol | Yes (1 call) | ⚠️ 1 call/symbol | ⚠️ 1 call/symbol |

### Fundamentals

| Feature | YFinance | StockFit (SEC XBRL) | AlphaVantage | Tiingo | FMP |
|---|---|---|---|---|---|
| Historical Depth | 4yr (8 quarters) | **15–20yr (70+ quarters)** | ~5yr | None | Locked |
| Rate Limit | Unlimited | 500k/day (Pro) | 25/day | 500/hr | 250/day |
| `get_info()` | Yes Rich (80+ fields) | Yes (SEC details) | Yes Good | Yes Basic | Yes Good |
| `statement="income"` | Yes (4yr) | **Yes (20yr XBRL)** | Yes (5yr+) | No | No Paid |
| `statement="balance"` | Yes (4yr) | **Yes (20yr XBRL)** | Yes (5yr+) | No | No Paid |
| `statement="cashflow"` | Yes (4yr) | **Yes (20yr XBRL)** | Yes (5yr+) | No | No Paid |
| Health Scores | No | **Yes (Piotroski F-Score)** | No | No | No |
| SEC Traceability | No | **Yes (10-K/Q Accession #)** | No | No | No |
| `statement="shares_outstanding"` | Warning (in balance sheet) | Yes Historical | Yes Historical | No | No Paid |
| `statement="earnings"` | Warning (basic) | Yes | Yes + Analyst Est. | No | No Paid |
| `get_earnings_calendar()` | No | Yes | Yes | No | No |
| News | Yes | Yes | No | Yes (Best) | No |

---

## Roadmap

- [x] SQLite Caching (24h TTL, OS temp folder)
- [x] Concurrent batch fetching (ThreadPoolExecutor)
- [x] Auto Rate Limit protection (AlphaVantage)
- [x] Custom Exception hierarchy
- [x] Fundamentals Facade (Income, Balance, Cashflow, Earnings, Shares Outstanding)
- [x] Universal Adjusted Prices — all providers return full `Adj Open/High/Low/Close/Volume`
- [x] Local Back-Adjustment Math Engine (`finance_math.calculate_adjusted_prices`)
- [x] Hybrid Provider (`hybrid_av_tiingo`) — deepest history + accurate dividends/splits
- [x] `Price.compare()` — concurrent multi-provider comparison (MultiIndex DataFrame)
- [x] `Price.consensus()` — mean/median aggregation across providers
- [x] Custom external DataFrame injection (`custom_data` parameter)
- [x] StockFit Integration (20-year SEC XBRL Statements, Piotroski Scores, Snapshot Utility)
- [ ] Data Quality Checks (anomaly detection, missing data handling)
- [ ] Technical Indicators Module (RSI, MACD, Bollinger Bands)
- [ ] Streaming / real-time price feed

---

## Testing

This project utilizes `pytest-vcr` and `unittest.mock` to ensure robust and comprehensive test coverage without consuming actual API rate limits. 

### VCR Cassettes (`pytest-vcr`)
VCR records real HTTP interactions and saves them as `.yaml` files (cassettes) within the `tests/cassettes/` directory.

- **Rate Limit Protection**: By replaying recorded responses, our test suite can execute reliably in CI/CD environments (e.g., GitHub Actions) without exhausting API provider quotas.
- **Security**: Sensitive query parameters, such as API keys, are automatically sanitized and replaced with placeholder strings (`DUMMY`) before being persisted to the cassettes.
- **Version Control**: The `.yaml` cassette files **must be committed** to version control. They act as the definitive mock data for automated CI pipelines.

### Generating New Cassettes
When adding new tests or modifying existing API requests, you will need to generate new cassettes:
1. Ensure your valid API keys are configured in the `.env` file.
2. Execute `pytest` locally. If a cassette for a specific test does not exist, VCR will automatically perform a real HTTP request and record the sanitized response.
3. Commit the newly generated `.yaml` files to Git.

### Running Tests & Checking Coverage
To run the test suite and see a detailed coverage report for all fetchers:
```bash
pytest --cov=src/alphadataforge --cov-report=term-missing
```
