Metadata-Version: 2.4
Name: veloxyz-scraper
Version: 0.2.0
Summary: velo.xyz market-data scraper (price, funding, open interest, premium, spot CVD)
Project-URL: Homepage, https://github.com/0xAtasoy/veloxyz-scraper
Project-URL: Repository, https://github.com/0xAtasoy/veloxyz-scraper
Project-URL: Issues, https://github.com/0xAtasoy/veloxyz-scraper/issues
Project-URL: Changelog, https://github.com/0xAtasoy/veloxyz-scraper/blob/main/CHANGELOG.md
Author: 0xAtasoy
License-Expression: MIT
License-File: LICENSE
Keywords: crypto,funding-rate,market-data,open-interest,scraper,velo
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: pandas>=2.2
Requires-Dist: requests>=2.32
Requires-Dist: tenacity>=9.0.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: typer>=0.15.0
Description-Content-Type: text/markdown

# veloxyz-scraper

[![PyPI](https://img.shields.io/pypi/v/veloxyz-scraper)](https://pypi.org/project/veloxyz-scraper/)
[![Python](https://img.shields.io/pypi/pyversions/veloxyz-scraper)](https://pypi.org/project/veloxyz-scraper/)
[![CI](https://github.com/0xAtasoy/veloxyz-scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/0xAtasoy/veloxyz-scraper/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/LICENSE)

A **cross-exchange aggregate time-series** scraper for [velo.xyz](https://velo.xyz) market data
— aggregated funding rate, premium, open interest, volume delta, liquidations and more, returned
as a `pandas.DataFrame`. It splits long ranges into chunks, retries transient errors, and resumes
interrupted fetches where they stopped.

> **Unofficial.** Not affiliated with Velo. It reads velo.xyz's free, undocumented chart
> endpoints (the ones the [charts](https://velo.xyz/chart) use) — these can change or
> rate-limit without notice. Please respect Velo's terms of service. Velo's official,
> supported, authenticated API is documented at [docs.velo.xyz](https://docs.velo.xyz/) and
> wrapped by the `velodata` package — a **different** API from the one used here.

## Quick start

```bash
pip install veloxyz-scraper
```

```python
from veloxyz_scraper import VeloScraper

scraper = VeloScraper()

# The last 1000 hourly, OI-weighted, cross-exchange funding-rate candles:
df = scraper.get_metric("BTCUSDT", "aggregated_funding_rate", "1h", last=1000)

print(df.tail(3))
#                 datetime   timestamp  aggregated_funding_rate
# 997  2024-02-11 09:00:00  1707642000                 0.000078
# 998  2024-02-11 10:00:00  1707645600                 0.000075
# 999  2024-02-11 11:00:00  1707649200                 0.000071
```

No API key, no account. For a guided tour, clone the repo and run the interactive example — it
asks for a coin, metric, resolution and range, then prints the equivalent Python call:

```bash
uv run python examples/01_quickstart.py
```

## Highlights

- **Cross-exchange aggregates** — funding rate, premium, open interest, spot/futures volume
  delta, liquidations, plus raw price candles and a computed Coinbase premium.
- **Resumable** — every chunk hits disk as it arrives; an interrupted fetch continues where it
  stopped instead of restarting.
- **Robust** — retry with exponential backoff on 429 / 5xx / timeouts, `Retry-After` aware, and
  automatic chunking around the endpoint's 450-candle cap.
- **Conventions you already know** — ccxt-style resolutions (`"1h"`, `"1d"`), closed
  `[begin, end]` windows, `pandas.DataFrame` out.
- **Typed and thread-safe** — ships `py.typed`; one client is safe to share across threads.
- **Library and CLI** — `import veloxyz_scraper`, or `veloscraper fetch`.

## Contents

- [Installation](#installation)
- [Library usage](#library-usage) — [metrics & resolutions](#metrics-and-resolutions),
  [last N candles](#last-n-candles), [time windows](#time-windows),
  [aggregation exchanges](#aggregation-exchanges), [coins vs dollars](#coins-vs-dollars),
  [coinbase premium](#coinbase-premium), [discovery](#discovery),
  [checkpoint & resume](#checkpoint--resume), [client options](#client-options),
  [custom metrics](#custom-metrics)
- [Supported metrics](#supported-metrics)
- [Per-exchange data availability](#per-exchange-data-availability)
- [CLI](#cli)
- [Examples](#examples)
- [Development](#development)
- [License](#license)

## Installation

Requires **Python 3.11+**. The wheel is pure Python — no compiled extensions.

```bash
pip install veloxyz-scraper        # from PyPI
uv add veloxyz-scraper             # or, in a uv project
```

From a clone, for development:

```bash
git clone https://github.com/0xAtasoy/veloxyz-scraper.git
cd veloxyz-scraper
uv sync                            # installs dev dependencies too
```

## Library usage

### Metrics and resolutions

`get_metric(ticker, metric, resolution, begin, end, *, last, exchanges, exchange, unit,
checkpoint, progress)` is the single entry point. Column shapes (scalar / OHLC / OHLCV) are
detected automatically.

```python
from veloxyz_scraper import VeloScraper

scraper = VeloScraper()

df = scraper.get_metric(
    "BTCUSDT",
    "aggregated_funding_rate",
    resolution="1h",
    begin="2024-01-01",
    end="2024-02-01",
)

# Open interest as OHLC candles:
oi = scraper.get_metric("ETHUSDT", "aggregated_open_interest", "4h",
                        begin="2024-06-01", end="2024-06-10")

# Raw price candles come from a single exchange, chosen by `exchange`:
px = scraper.get_metric("BTCUSDT", "price", "1d", begin="2024-01-01", end="2024-06-01",
                        exchange="binance-futures")
```

`resolution` is a **ccxt-style lowercase token** (case-sensitive): `"1m"`, `"5m"`, `"15m"`,
`"30m"`, `"1h"`, `"2h"`, `"4h"`, `"12h"`, `"1d"`, `"1w"`. Both metrics and resolutions are also
available as enums, which give you editor autocomplete:

```python
from veloxyz_scraper import Metric, Resolution

scraper.get_metric("BTCUSDT", Metric.aggregated_open_interest, Resolution.h4, last=100)
```

`begin`/`end` accept a date `str` (`"2024-01-01"`), an epoch `int` (seconds or ms), or a
`datetime`. If omitted, `end=now` and `begin=end-30 days`. **Naive dates/datetimes are
interpreted as UTC**, so the same input gives the same result on any machine.

velo data starts at **2021-01-01 00:00:00 UTC**; an earlier `begin` is clamped there
automatically (exported as `DATA_START_MS`).

### Last N candles

```python
# The last 1000 hourly candles:
df = scraper.get_metric("BTCUSDT", "aggregated_funding_rate", "1h", last=1000)

# Scales with resolution: the last 500 four-hour candles
oi = scraper.get_metric("BTCUSDT", "aggregated_open_interest", "4h", last=500)
```

`last` returns **exactly N candles**, ending at the bar that contains `end` — i.e. the current,
in-progress bar when `end` defaults to now. It cannot be combined with `begin`.

> In the first moments of a new period velo may not have published the current bar yet, so a
> `last=N` call made right at a rollover can return `N-1` candles until it appears.

### Time windows

One rule covers every case:

> **`get_metric` returns every candle whose interval intersects `[begin, end]`.**

Both `begin` and `end` are snapped down to the resolution grid, so each becomes the open of the
bar that contains it. For grid-aligned inputs — dates, aligned timestamps, i.e. almost always —
this is the familiar closed range used by pandas (`df.loc[a:b]`, `pd.date_range`) and by the
Binance/Coinbase kline endpoints: both endpoints are included.

```python
# Daily, both dates included -> 32 candles (2024-05-01 .. 2024-06-01)
scraper.get_metric("BTCUSDT", "aggregated_open_interest", "1d",
                   begin="2024-05-01", end="2024-06-01")

# Hourly, a mid-bar end: the 10:00 bar contains 10:37, so it is the last candle
scraper.get_metric("BTCUSDT", "price", "1h",
                   begin="2024-01-01 00:00", end="2024-01-01 10:37")
```

A partially-overlapping bar is kept rather than dropped: losing it would be a silent gap in data
you asked for, while an extra edge candle is visible and easy to trim.

**Chaining windows.** Because both ends are inclusive, consecutive windows share their boundary
candle. When paging through history, either step `end` back by one bar or dedupe:

```python
frames = [scraper.get_metric("BTCUSDT", "price", "1d", begin=a, end=b) for a, b in months]
df = pd.concat(frames).drop_duplicates(subset="timestamp").reset_index(drop=True)
```

**Note on velo's raw endpoint.** velo's `/api/m/range` is half-open — it returns bars with
`begin <= open < end`, so a grid-aligned `end` drops its own bar there. This library translates:
the fetch layer compensates internally, and `resolution` / `last` / `[begin, end]` follow the
ccxt and pandas conventions instead. If you compare a DataFrame against a raw velo network
request you will see one extra candle at the end — that is expected.

### Aggregation exchanges

Aggregate metrics combine a default set of venues. Pass `exchanges=` to restrict or change it —
useful when you need a **constant-composition** series (see
[per-exchange data availability](#per-exchange-data-availability)).

```python
from veloxyz_scraper import DEFAULT_EXCHANGES, SPOT_EXCHANGES

df = scraper.get_metric(
    "BTCUSDT", "aggregated_open_interest", "1d",
    exchanges=["binance-futures", "bybit", "okex-swap"],   # instead of DEFAULT_EXCHANGES
    last=365,
)
```

`exchanges` applies to aggregate metrics. The separate `exchange` argument is **only** for the
raw `price` metric, which comes from one venue; aggregates ignore it.

### Coins vs dollars

`aggregated_open_interest`, `aggregated_spot_volume_delta`, `aggregated_volume_delta` and
`aggregated_liquidations` accept `unit="coins"` (default) or `unit="dollars"`:

```python
oi_usd = scraper.get_metric("BTCUSDT", "aggregated_open_interest", "1d", unit="dollars")
liq    = scraper.get_metric("BTCUSDT", "aggregated_liquidations", "1h", last=168)
```

The unit is not encoded in the column names or the file name, so track which one you fetched.
Rates (`funding_rate` / `premium`) and `price` have no coins/dollars unit and reject `unit`.

### Coinbase premium

A **computed** metric: the spot-price difference between Coinbase (`BTC-USD`) and Binance
(`BTCUSDT`). Two price series are fetched and aligned on timestamp.

```python
cp = scraper.get_coinbase_premium("BTCUSDT", "1h", begin="2024-01-01", end="2024-02-01")
# datetime, timestamp, binance_close, coinbase_close, premium, premium_pct
```

`get_metric("BTCUSDT", "coinbase_premium", ...)` routes to the same result. Because it is not a
single API metric, it rejects `exchanges`, `exchange` and `unit`.

### Discovery

Look up what velo supports instead of hardcoding symbols:

```python
"BTC" in scraper.list_coins()          # coins available for aggregation (list[str])

products = scraper.list_products()     # futures products per exchange (DataFrame)
products[products["coin"] == "BTC"]    # columns: exchange, product, coin, min
```

### Checkpoint & resume

Pass `checkpoint=` and every chunk is written to CSV as it arrives, so a long fetch is
crash-safe and a repeat run continues where it stopped.

```python
df = scraper.get_metric(
    "BTCUSDT", "aggregated_funding_rate", "1h",
    begin="2021-01-01", end="2026-01-01",
    checkpoint="btc_funding.csv",
)
```

Four things are worth knowing:

- **The file is an accumulating cache — a superset.** `get_metric` always returns exactly the
  requested `[begin, end]` window, while the CSV keeps the union of everything ever fetched into
  it. `len(pd.read_csv("btc_funding.csv"))` can therefore exceed `len(df)`. That is expected.
- **Resume is bidirectional.** A later run backfills a missing head range as well as extending
  the tail, and it **re-fetches the last cached bar** (which may have been partial) and
  overwrites it. Only the current, still-open bar is ever provisional. The final write is atomic
  (temp file + `os.replace`), so an interrupted write cannot corrupt the checkpoint.
- **A sidecar guards identity.** `btc_funding.csv.meta.json` records the symbol, resolution and
  exchange the file was written for. Reusing it for a different request raises

  ```text
  ValueError: Checkpoint btc_funding.csv was written for a different request
  ({'resolution': {'file': '60', 'requested': '240'}}). Use a different checkpoint path
  or delete the existing file.
  ```

  Include the resolution in the filename to avoid this (as `examples/all_metrics.py` does).
- **`coinbase_premium` uses sidecar checkpoints.** Each price leg caches independently to
  `btc_cbp.binance.csv` and `btc_cbp.coinbase.csv` (each with its own `.meta.json`), so both
  resume on their own; the premium is then recomputed into the main checkpoint.

### Client options

All constructor arguments are keyword-only:

```python
scraper = VeloScraper(
    timeout=(10.0, 30.0),                # (connect, read) seconds
    max_attempts=5,                      # retries on 429 / 5xx / timeout / connection error
    request_pause=0.15,                  # seconds between consecutive requests
    api_base="https://velo.xyz/api/m/",  # override for a proxy or a test server
)
```

Retries use exponential backoff and honour a `Retry-After` header on 429. Long ranges are split
into chunks of **450 candles** (the endpoint's hard per-request cap). A single `VeloScraper` is
safe to share across threads — each thread lazily gets its own `requests.Session`.

Pass `progress=False` to `get_metric` to silence the `tqdm` bar.

### Custom metrics

You don't need to fork the library to add a metric — pass a `MetricSpec` (the symbol tail,
copied verbatim from a velo chart request) straight to `get_metric`:

```python
from veloxyz_scraper import MetricSpec, VeloScraper

spec = MetricSpec("my_metric", "some#velo#tail")   # tail after {ticker}#{exchanges}#
df = VeloScraper().get_metric("BTCUSDT", spec, "1h", last=100)
```

The built-in presets are exported read-only as `METRICS`; `resolve_metric` and
`resolve_resolution` expose the same normalisation the client uses. To contribute a verified
preset upstream, add it to the `METRICS` dict in `src/veloxyz_scraper/metrics.py`.

## Supported metrics

| Metric | Output shape | Columns | Exchanges |
|---|---|---|---|
| `price` | OHLCV | `open`, `high`, `low`, `close`, `volume` | single (`exchange` arg) |
| `aggregated_funding_rate` | scalar | `aggregated_funding_rate` | futures |
| `aggregated_premium` | scalar | `aggregated_premium` (OI-weighted) | futures |
| `aggregated_open_interest` | OHLC | `open`, `high`, `low`, `close` | futures |
| `aggregated_spot_volume_delta` | scalar | `aggregated_spot_volume_delta` (spot buy - sell) | spot |
| `aggregated_volume_delta` | scalar | `aggregated_volume_delta` (futures buy - sell) | futures |
| `aggregated_liquidations` | 2 values | `short_liquidations`, `long_liquidations` | futures |
| `coinbase_premium` | computed | `binance_close`, `coinbase_close`, `premium`, `premium_pct` | binance + coinbase spot |

Every frame also carries `datetime` and `timestamp`. Metric names follow the velo.xyz interface
(`aggregated_*`). All are verified against the real API.

Per-metric behaviour worth knowing:

- **`aggregated_premium`**: of velo.xyz's three premiums, this is the *aggregated Premium*
  (OI-weighted, cross-exchange). The single-exchange premium is a separate metric (not added
  yet).
- **`aggregated_open_interest`**: since OI is a *level* metric, the velo API returns degenerate
  candles at fine resolutions (e.g. hourly): `open == close`, with `high`/`low` from the
  intra-bar range. To match velo's own chart, `open` is derived from the **previous close** on
  those rows. The fix applies only where `open == close` and only below `2h` (120 minutes; `2h`
  and coarser already return real OHLC). The first row keeps the API value, and
  `close`/`high`/`low` are never touched.
- **`aggregated_spot_volume_delta` / `aggregated_volume_delta`**: **per-bar** volume delta
  (buy - sell), not a running total — velo's "Delta" and "Cumulative Delta" return the same
  values; the cumulative line is a UI-only view. Take `.cumsum()` yourself for a running CVD.
- **`aggregated_liquidations`**: two values per row — `short_liquidations` (velo's green series)
  then `long_liquidations` (red). Use `unit="dollars"` for USD notional.

## Per-exchange data availability

An aggregate combines several exchanges, but **each exchange only contributes from the date its
own data begins** — so the composition of the aggregate changes over time. The dates below were
measured (BTCUSDT) by requesting each exchange on its own
(`symbol=BTCUSDT#<exchange>#<metric-tail>`) and binary-searching the first day that returns data.

**Futures exchanges** — same for `aggregated_funding_rate`, `aggregated_premium`,
`aggregated_open_interest`:

| Exchange | Included from |
|---|---|
| binance-futures | <= 2021-01-01 (data start) |
| bybit | <= 2021-01-01 |
| okex-swap | <= 2021-01-01 |
| deribit | <= 2021-01-01 |
| binance-coin-margin | <= 2021-01-01 |
| bybit-coin-margin | <= 2021-01-01 |
| okex-coin-margin | <= 2021-01-01 |
| **hyperliquid** | **2024-06-11** |

**Spot exchanges** — `aggregated_spot_volume_delta`:

| Exchange | Included from |
|---|---|
| binance | <= 2021-01-01 |
| coinbase | <= 2021-01-01 |
| **bybit-spot** | **2024-09-25** |
| **okex** | **2024-09-26** |

> **Heads-up for modeling:** because a new venue joins mid-series, the aggregate can show a
> **step change** when it is added (e.g. aggregated open interest jumps up around 2024-06-11 when
> hyperliquid enters; spot CVD around 2024-09-25 when bybit-spot/okex enter). This is not a data
> error — it is a composition change. If a constant-composition series matters for your model,
> restrict `exchanges` to venues present over your whole window.

## CLI

```bash
veloscraper fetch --metric <METRIC> --out <FILE.csv> [options]
```

`--metric` and `--out` are required. `<METRIC>` is any name from
[Supported metrics](#supported-metrics).

| Flag | Default | Notes |
|---|---|---|
| `--ticker` | `BTCUSDT` | Symbol, e.g. `ETHUSDT`, `BTC-USD` |
| `--resolution` | `1h` | `1m 5m 15m 30m 1h 2h 4h 12h 1d 1w` |
| `--begin` / `--end` | `end=now`, `begin=end-30d` | Date or epoch |
| `--last` | — | Candle count; cannot be combined with `--begin` |
| `--exchanges` | metric's default set | Comma-separated; aggregates only |
| `--exchange` | `binance-futures` | `price` only; ignored for aggregates |
| `--unit` | `coins` | `dollars` for OI / `*_volume_delta` / liquidations |
| `--resume` / `--no-resume` | `--resume` | `--out` doubles as a checkpoint |
| `--timeout` | `30.0` | Per-request read timeout, seconds |
| `--max-attempts` | `5` | Retries on transient errors |
| `--quiet` / `-q` | — | Silence progress and logs |

```bash
# Hourly funding rate, last 30 days (default range)
veloscraper fetch --metric aggregated_funding_rate --out btc.csv

# The last 90 daily open-interest candles, in USD
veloscraper fetch --metric aggregated_open_interest --resolution 1d --last 90 \
    --unit dollars --out btc_oi_usd.csv

# Price candles from a specific exchange
veloscraper fetch --metric price --exchange bybit --last 168 --out bybit_price.csv

# Restrict the aggregation venues
veloscraper fetch --metric aggregated_funding_rate \
    --exchanges binance-futures,bybit,okex-swap --last 168 --out btc_3exch.csv

# A long historical fetch. If it is interrupted, the SAME command resumes it.
veloscraper fetch --metric aggregated_funding_rate --begin 2021-01-01 --out btc_full.csv

veloscraper --version
veloscraper fetch --help
```

Two mistakes the CLI catches for you:

```bash
veloscraper fetch --out btc.csv
#   -> Missing option '--metric'.

veloscraper fetch --metric price --begin 2024-01-01 --last 100 --out x.csv
#   -> Error: 'begin' and 'last' cannot be used together.
```

## Examples

Runnable scripts live in [`examples/`](https://github.com/0xAtasoy/veloxyz-scraper/tree/main/examples)
— see the [index](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/examples/README.md). The
numbered scripts are progressive: `01`–`07` cover the API, `08`–`10` show real analyses.

- [`01_quickstart.py`](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/examples/01_quickstart.py)
  — **interactive**: a guided first fetch that asks for a coin, metric, resolution and range,
  then prints the equivalent Python call.
- [`06_checkpoint_resume.py`](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/examples/06_checkpoint_resume.py)
  — crash-safe incremental fetch and resume.
- [`08_funding_rate_analysis.py`](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/examples/08_funding_rate_analysis.py)
  — funding regime and extremes.
- [`09_open_interest_vs_price.py`](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/examples/09_open_interest_vs_price.py)
  — merge OI with price.
- [`all_metrics.py`](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/examples/all_metrics.py)
  — fetch every metric to a separate CSV.

## Development

```bash
uv sync
uv run pytest                # tests; the network is mocked, no internet needed
uv run ruff check            # lint
uv run ruff format --check   # formatting
uvx basedpyright src/        # type check
```

Tests never touch the network: `requests-mock` supplies responses and `pytest-socket` blocks
real sockets. A new metric must be verified against a live velo chart request (DevTools →
Network → `/api/m/range?...`) before its `MetricSpec` is added.

## License

[MIT](https://github.com/0xAtasoy/veloxyz-scraper/blob/main/LICENSE).
