Metadata-Version: 2.4
Name: veloxyz-scraper
Version: 0.1.0
Summary: velo.xyz market-data scraper (price, funding, open interest, premium, spot CVD)
Project-URL: Homepage, https://github.com/0xAtasoy/veloxyz-scraper
Project-URL: Repository, https://github.com/0xAtasoy/veloxyz-scraper
Project-URL: Issues, https://github.com/0xAtasoy/veloxyz-scraper/issues
Project-URL: Changelog, https://github.com/0xAtasoy/veloxyz-scraper/blob/main/CHANGELOG.md
Author: 0xAtasoy
License-Expression: MIT
License-File: LICENSE
Keywords: crypto,funding-rate,market-data,open-interest,scraper,velo
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: pandas>=2.2
Requires-Dist: requests>=2.32
Requires-Dist: tenacity>=9.0.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: typer>=0.15.0
Description-Content-Type: text/markdown

# veloxyz-scraper

A **cross-exchange aggregate time-series** scraper for [velo.xyz](https://velo.xyz) market
data (aggregated funding rate, aggregated premium, aggregated open interest, and more). It
automatically splits long ranges into chunks, retries on transient errors, and resumes
interrupted fetches where they stopped.

> **Unofficial.** Not affiliated with Velo. It reads velo.xyz's free, undocumented chart
> endpoints (the ones the [charts](https://velo.xyz/chart) use) — these can change or
> rate-limit without notice. Please respect Velo's terms of service. Velo's official,
> supported, authenticated API is documented at [docs.velo.xyz](https://docs.velo.xyz/) and
> wrapped by the `velodata` package — a **different** API from the one used here.

## Features

- **Robust fetching:** shared `requests.Session`, timeout, and retry with exponential
  backoff (429 / 5xx / timeout / connection error). Long ranges are split into chunks of
  450 candles (the endpoint's hard per-request cap).
- **Discovery:** `list_coins()` and `list_products()` to see what velo supports.
- **Thread-safe:** one `VeloScraper` can be shared across threads (per-thread HTTP session).
- **Checkpoint + resume:** each chunk is written to disk; if a fetch is interrupted it
  continues from where it stopped.
- **Generic metric support:** `price`, `aggregated_funding_rate`, `aggregated_premium`,
  `aggregated_open_interest`, `aggregated_spot_volume_delta`, `aggregated_volume_delta`,
  `aggregated_liquidations`, and the computed `coinbase_premium` (scalar / OHLC / OHLCV
  columns detected automatically); extensible via a custom `MetricSpec`.
- **Coins or dollars:** `unit="dollars"` for `open_interest` / `*_volume_delta` /
  `liquidations`.
- **Flexible resolution:** minutes (`1, 5, 15, 60, 240 ...`) and daily/weekly (`1D`, `1W`).
- **CLI + library:** both a `veloscraper` command and `import veloxyz_scraper`.
- Returns a `pandas.DataFrame`, with an optional `tqdm` progress bar.

## Installation

With [uv](https://docs.astral.sh/uv/):

```bash
uv sync           # development environment (includes dev dependencies)
# or as a dependency in a project:
uv add veloxyz-scraper
```

With pip:

```bash
pip install .
```

## Library usage

```python
from veloxyz_scraper import VeloScraper

scraper = VeloScraper()

# OI-weighted, cross-exchange aggregate funding rate (hourly):
df = scraper.get_metric(
    "BTCUSDT",
    "aggregated_funding_rate",
    resolution=60,            # minutes; 60 = hourly
    begin="2024-01-01",
    end="2024-02-01",
)
print(df.head())
#              datetime   timestamp  aggregated_funding_rate
# 0 2024-01-01 00:00:00  1704067200                 0.000080

# Generic API — any metric:
oi = scraper.get_metric("ETHUSDT", "aggregated_open_interest", resolution=240,
                        begin="2024-06-01", end="2024-06-10")
# returns OHLC: datetime, timestamp, open, high, low, close

# Raw price candles (daily) — the exchange param picks which exchange's candles:
px = scraper.get_metric("BTCUSDT", "price", resolution="1D",
                        begin="2024-01-01", end="2024-06-01", exchange="binance-futures")
# returns price OHLCV: datetime, timestamp, open, high, low, close, volume
```

`begin`/`end` accept a date `str` (`"2024-01-01"`), an epoch `int` (seconds or ms), or a
`datetime`. If omitted, `end=now` and `begin=end-30 days`. **Naive dates/datetimes are
interpreted as UTC** (so the same input gives the same result on any machine).

**Last N candles** — use `last` to compute `begin` automatically
(`begin = end - last*resolution`):

```python
# The last 1000 hourly candles:
df = scraper.get_metric("BTCUSDT", "aggregated_funding_rate", resolution=60, last=1000)

# Scales with resolution: the last 500 four-hour candles
oi = scraper.get_metric("BTCUSDT", "aggregated_open_interest", resolution=240, last=500)
```

`last` is a **candle count** (int) and works together with `resolution`; it cannot be
combined with `begin`. velo data starts at **2021-01-01 00:00:00 UTC**; if `begin` (explicit,
derived from `last`, or the default) is earlier than that, it is clamped there automatically
(`DATA_START_MS`).

`resolution`: minutes as an `int` (`1, 5, 10, 15, 30, 60, 120, 240, 360, 720`) or `"1D"` /
`"1W"` for daily/weekly. `begin`/`end` are aligned to the resolution grid (begin floored,
end ceiled) so the bars containing them are included.

### Discovery

Look up what velo supports without hardcoding symbols:

```python
"BTC" in scraper.list_coins()          # coins available for aggregation (list[str])

products = scraper.list_products()     # futures products per exchange (DataFrame)
products[products["coin"] == "BTC"]    # every BTC perp: exchange, product, coin, min
```

### Checkpoint / resume

```python
df = scraper.get_metric(
    "BTCUSDT", "aggregated_funding_rate", 60, "2021-01-01", "2026-01-01",
    checkpoint="btc_funding.csv",   # each chunk is written here
)
```

If the fetch is interrupted, calling it again with the same `checkpoint` path resumes from
where it stopped. The last bar of the previous run is **re-fetched and overwritten**, so a
still-forming / partial bar gets finalized on the next run; only the current (still-open) bar
is ever provisional.

## CLI

Every `fetch` command requires `--out` and `--metric`.

```bash
veloscraper fetch --metric <METRIC> --out <FILE.csv>
```

`<METRIC>`: `price` · `aggregated_funding_rate` · `aggregated_premium` ·
`aggregated_open_interest` · `aggregated_spot_volume_delta` · `aggregated_volume_delta` ·
`aggregated_liquidations` · `coinbase_premium`

`--unit dollars` switches `open_interest` / `*_volume_delta` / `liquidations` to USD. See all
flags with `veloscraper fetch --help`.

### Basics

```bash
# BTCUSDT hourly funding rate, last 30 days (default range)
veloscraper fetch --metric aggregated_funding_rate --out btc.csv

# A different symbol
veloscraper fetch --ticker ETHUSDT --metric aggregated_funding_rate --out eth.csv
```

### Metrics

```bash
veloscraper fetch --metric price                    --out btc_price.csv   # OHLCV
veloscraper fetch --metric aggregated_open_interest --out btc_oi.csv      # OHLC
veloscraper fetch --metric aggregated_premium       --out btc_prem.csv
veloscraper fetch --metric aggregated_spot_volume_delta --out btc_spotdelta.csv
veloscraper fetch --metric aggregated_volume_delta      --out btc_voldelta.csv
veloscraper fetch --metric aggregated_liquidations      --out btc_liq.csv    # short + long
veloscraper fetch --metric coinbase_premium             --out btc_cbp.csv    # computed

# Dollar-denominated (open_interest / *_volume_delta / liquidations):
veloscraper fetch --metric aggregated_open_interest --unit dollars --out btc_oi_usd.csv
```

### Time range

```bash
# Explicit start/end
veloscraper fetch --metric aggregated_funding_rate --begin 2024-01-01 --end 2024-02-01 --out btc_jan.csv

# Only a start (end = now)
veloscraper fetch --metric aggregated_funding_rate --begin 2024-06-01 --out btc_since_jun.csv

# A begin that is too early is clamped to 2021-01-01 (where the data starts)
veloscraper fetch --metric aggregated_funding_rate --begin 2015-01-01 --out btc_all.csv
```

### Last N candles (`--last`, works with `--resolution`)

```bash
# The last 1000 hourly candles
veloscraper fetch --metric aggregated_funding_rate --resolution 60  --last 1000 --out btc_1000h.csv

# The last 500 four-hour candles
veloscraper fetch --metric price --resolution 240 --last 500 --out btc_500x4h.csv

# The last 90 daily candles
veloscraper fetch --metric aggregated_open_interest --resolution 1D --last 90 --out btc_90d.csv
```

### Resolution

```bash
# Minutes: 1, 5, 10, 15, 30, 60, 120, 240, 360, 720
veloscraper fetch --metric price --resolution 15 --last 200 --out btc_15m.csv

# Daily / weekly
veloscraper fetch --metric aggregated_funding_rate --resolution 1D --begin 2021-01-01 --out btc_daily.csv
veloscraper fetch --metric aggregated_funding_rate --resolution 1W --begin 2021-01-01 --out btc_weekly.csv
```

### `price` + exchange (`--exchange` is only for `price`)

```bash
veloscraper fetch --metric price --exchange bybit    --last 168 --out bybit_price.csv
veloscraper fetch --metric price --exchange coinbase --ticker BTC-USD --last 168 --out cb_price.csv
```

### Override the aggregation exchanges

```bash
veloscraper fetch --metric aggregated_funding_rate \
    --exchanges binance-futures,bybit,okex-swap --last 168 --out btc_3exch.csv
```

### Checkpoint / resume

`--out` acts as a checkpoint by default (disable with `--no-resume`).

```bash
# A long historical fetch — if it is interrupted, the SAME command resumes it
veloscraper fetch --metric aggregated_funding_rate --begin 2021-01-01 --resolution 60 --out btc_full.csv

# Disable resume (start over each time)
veloscraper fetch --metric price --last 500 --no-resume --out btc_fresh.csv
```

### Other

```bash
veloscraper fetch --metric price --last 100 --out btc.csv --quiet   # silence progress/logs
veloscraper --version
veloscraper fetch --help
```

### Common mistakes

```bash
# ✗ --metric is required
veloscraper fetch --out btc.csv
#   -> Missing option '--metric'.

# ✗ begin and last cannot be used together
veloscraper fetch --metric price --begin 2024-01-01 --last 100 --out x.csv
#   -> ValueError: 'begin' and 'last' cannot be used together.
```

## Examples

- [`examples/quickstart.py`](examples/quickstart.py) — fetch every metric and write a
  separate CSV for each.

## Development

```bash
uv run pytest       # tests (network is mocked, no internet needed)
uv run ruff check   # lint
```

## Supported metrics

| Metric                     | Output shape | Columns                                    | Exchanges |
|----------------------------|--------------|--------------------------------------------|-----------|
| `price`                    | OHLCV        | `open`, `high`, `low`, `close`, `volume`   | single (exchange param) |
| `aggregated_funding_rate`  | scalar       | `aggregated_funding_rate`                  | futures   |
| `aggregated_premium`       | scalar       | `aggregated_premium` (OI-weighted)         | futures   |
| `aggregated_open_interest` | OHLC         | `open`, `high`, `low`, `close`             | futures   |
| `aggregated_spot_volume_delta` | scalar   | `aggregated_spot_volume_delta` (spot buy−sell) | spot |
| `aggregated_volume_delta`      | scalar   | `aggregated_volume_delta` (futures buy−sell)   | futures |
| `aggregated_liquidations`  | 2 values     | `short_liquidations`, `long_liquidations`  | futures   |
| `coinbase_premium`         | computed     | `binance_close`, `coinbase_close`, `premium`, `premium_pct` | binance + coinbase spot |

Metric names follow the velo.xyz interface (`aggregated_*`). All are verified against the
real API.

**Coins vs dollars.** `open_interest`, `spot_volume_delta`, `volume_delta` and
`liquidations` accept `unit="coins"` (default) or `unit="dollars"`:

```python
oi_usd = scraper.get_metric("BTCUSDT", "aggregated_open_interest", "1D", unit="dollars")
liq    = scraper.get_metric("BTCUSDT", "aggregated_liquidations", 60, last=168)
#   liq columns: short_liquidations (green), long_liquidations (red)
```

The unit is not encoded in the column names or file name (like resolution), so track which
one you fetched. Rates (`funding_rate` / `premium`) and `price` have no coins/dollars unit.

- **`aggregated_premium`**: of velo.xyz's three premiums, this is the *aggregated Premium*
  (OI-weighted, cross-exchange). The single-exchange premium is a separate metric (not added
  yet).
- **`aggregated_open_interest`**: since OI is a *level* metric, the velo API returns
  degenerate candles at low resolutions (e.g. hourly): `open == close`, with `high`/`low`
  from the intra-hour range. To match velo's own chart, `open` is derived from the
  **previous close** on degenerate rows. This fix applies only to rows where `open == close`
  and only when `resolution < 120` (120+ is already real OHLC); the first row of the series
  keeps the API value. `close`/`high`/`low` are never touched.
- **`aggregated_spot_volume_delta` / `aggregated_volume_delta`**: **per-bar** volume delta
  (buy − sell), not a running total — velo's "Delta" and "Cumulative Delta" return the same
  values; the cumulative line is a UI-only view. Take `.cumsum()` yourself for a running CVD.
- **`aggregated_liquidations`**: two values per row — `short_liquidations` (velo's green
  series) then `long_liquidations` (red). Use `unit="dollars"` for USD notional.
- **`coinbase_premium`**: not a single API metric but a **computed** one — the spot price
  difference between coinbase (`BTC-USD`) and binance (`BTCUSDT`). Two requests are made and
  aligned on timestamp:

  ```python
  cp = scraper.get_coinbase_premium("BTCUSDT", resolution=60,
                                    begin="2024-01-01", end="2024-02-01")
  # datetime, timestamp, binance_close, coinbase_close, premium, premium_pct
  ```

  `get_metric("BTCUSDT", "coinbase_premium", ...)` routes to the same result.

### Custom metrics

You don't need to fork the library to add a metric — pass a `MetricSpec` (the symbol tail,
verified from a velo chart request) straight to `get_metric`:

```python
from veloxyz_scraper import MetricSpec, VeloScraper

spec = MetricSpec("my_metric", "some#velo#tail")   # tail after {ticker}#{exchanges}#
df = VeloScraper().get_metric("BTCUSDT", spec, resolution=60, last=100)
```

The built-in presets are exported read-only as `METRICS` (`from veloxyz_scraper import
METRICS`) for discovery. To contribute a verified preset upstream, add it to the `METRICS`
dict in `src/veloxyz_scraper/metrics.py`.

## Per-exchange data availability

An aggregate combines several exchanges, but **each exchange only contributes from the date
its own data begins** — so the composition of the aggregate changes over time. The dates
below were measured (BTCUSDT) by requesting each exchange on its own
(`symbol=BTCUSDT#<exchange>#<metric-tail>`) and binary-searching the first day that returns
data.

**Futures exchanges** — same for `aggregated_funding_rate`, `aggregated_premium`,
`aggregated_open_interest`:

| Exchange              | Included from |
|-----------------------|---------------|
| binance-futures       | ≤ 2021-01-01 (data start) |
| bybit                 | ≤ 2021-01-01 |
| okex-swap             | ≤ 2021-01-01 |
| deribit               | ≤ 2021-01-01 |
| binance-coin-margin   | ≤ 2021-01-01 |
| bybit-coin-margin     | ≤ 2021-01-01 |
| okex-coin-margin      | ≤ 2021-01-01 |
| **hyperliquid**       | **2024-06-11** |

**Spot exchanges** — `aggregated_spot_volume_delta`:

| Exchange    | Included from |
|-------------|---------------|
| binance     | ≤ 2021-01-01 |
| coinbase    | ≤ 2021-01-01 |
| **bybit-spot** | **2024-09-25** |
| **okex**    | **2024-09-26** |

> **Heads-up for modeling:** because a new venue joins mid-series, the aggregate can show a
> **step change** when it is added (e.g. aggregated open interest jumps up around 2024-06-11
> when hyperliquid enters; spot CVD around 2024-09-25 when bybit-spot/okex enter). This is
> not a data error — it is a composition change. If a constant-composition series matters for
> your model, restrict `--exchanges` to venues present over your whole window.
