Metadata-Version: 2.4
Name: hyperliquid-data
Version: 0.1.0
Summary: Download Hyperliquid historical data (REST + S3 archive) into Parquet, with an up-front egress cost gate.
Project-URL: Homepage, https://github.com/bond-labs-dev/hyperliquid-data
Project-URL: Issues, https://github.com/bond-labs-dev/hyperliquid-data/issues
Author-email: bondlabs <hello@bondlabs.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: funding,hyperliquid,market-data,orderbook,perpetuals
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Requires-Python: >=3.10
Requires-Dist: boto3>=1.28
Requires-Dist: lz4>=4.0
Requires-Dist: pyarrow>=14
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.16; extra == 'dev'
Description-Content-Type: text/markdown

# hyperliquid-data

[![ci](https://github.com/bond-labs-dev/hyperliquid-data/actions/workflows/ci.yml/badge.svg)](https://github.com/bond-labs-dev/hyperliquid-data/actions/workflows/ci.yml)

Pull Hyperliquid history into Parquet: candles, funding, L2 book, trades,
fills, liquidations. Price the S3 egress bill before you run the pull.

Both Hyperliquid S3 buckets are requester-pays. `fills` alone is 0.8–1.0 GiB
**per day** of all-coin data, so a casual one-year pull is a three-figure AWS
bill you find out about at the end of the month. `hl-data cost` lists the exact
prefixes a pull would touch and prints the number first.

```
$ hl-data cost --dataset fills --start 2026-06-01 --end 2026-06-30
dataset      fills
range        30 day(s), 3 sampled
per day      0.820 GiB  (24 objects)
TOTAL        24.61 GiB  (720 objects)
EGRESS COST  ~$2.21  (@ $0.09/GB, requester-pays)

samples:
  2026-06-01  0.960 GiB  24 obj
  2026-06-15  0.856 GiB  24 obj
  2026-06-30  0.644 GiB  24 obj

note: extrapolated from 3 sampled day(s) (min 0.644 / max 0.960 GiB). Actual volume varies with market activity.
```

The estimator and the pullers read the same prefix map
([`prefixes.py`](src/hyperliquid_data/prefixes.py)), so they cannot disagree
about which objects a date lives under. That is not a hypothetical worry: the
fill stream changed format in mid-2025, and a hardcoded prefix under-prices any
range that crosses the handoff.

## Why this exists

| | official [python-sdk](https://github.com/hyperliquid-dex/hyperliquid-python-sdk) | [hyperliquid-historical](https://github.com/c-i/hyperliquid-historical) | hyperliquid-data |
|---|---|---|---|
| REST candles / funding | ✅ raw calls | — | ✅ paginated to exhaustion |
| L2 book archive → tabular | — | ✅ CSV | ✅ Parquet |
| `node_fills_by_block` (trades) | — | — | ✅ |
| Fills with wallet + `closedPnl` | — | — | ✅ |
| Liquidations | — | — | ✅ |
| Egress cost gate | — | — | ✅ |
| Coverage manifest (gaps/dupes) | — | — | ✅ REST partitions |

## Install

```bash
pip install hyperliquid-data
# or, from a checkout:
pip install -e .
```

Python ≥3.10, and nothing outside the wheel. Archives are decompressed
in-process by the `lz4` package, streaming: no decompressed copy ever touches
disk, and there is no external binary to install.

## Quickstart

```bash
# free: public REST, no AWS account
hl-data candles all --coin BTC --interval 1h --start 2024-01-01 \
    --raw-dir raw --root data
hl-data funding pull --coins BTC ETH HYPE --root data

# requester-pays: size it first, then pull
hl-data cost   --dataset l2book --start 2026-06-01 --end 2026-06-07 --coins BTC
hl-data l2book pull --coins BTC --start 2026-06-01 --end 2026-06-07 --root data
```

## Datasets

| command | source | output | notes |
|---|---|---|---|
| `candles` | REST `/info` | `candles/hyperliquid/<coin>_perp/<tf>/candles.parquet` | 5000-bar cap per request, windowed |
| `funding` | REST `/info` | `funding/hyperliquid/<coin>_perp/funding.parquet` | `list` / `probe` / `pull` |
| `l2book` | `s3://hyperliquid-archive` | `l2book/hyperliquid/<coin>_perp/date=…/l2book.parquet` | full depth + resting-order count `n` |
| `trades` | `s3://hl-mainnet-node-data` | `trades/hyperliquid/<coin>_perp/date=…/trades.parquet` | taker row only, deduped by `tid` |
| `fills` | `s3://hl-mainnet-node-data` | `fills/hyperliquid/date=…/fills.parquet` | both sides + wallet + `closedPnl`, all coins |
| `liquidations` | `s3://hl-mainnet-node-data` | `liquidations/hyperliquid/<coin>_perp/liquidations.parquet` | 5-minute buckets, long/short notional |

`funding`, `l2book` and `liquidations` take a subcommand (`pull`, plus `probe`
or `list` where those make sense); `candles` takes `candles`, `funding`,
`ingest` or `all`. `trades` and `fills` take flags directly.
`hl-data <command> --help` spells out either one. Everywhere `--coins` appears
it accepts both spellings: `--coins BTC ETH` and `--coins BTC,ETH`.

Every REST partition (`candles`, `funding`) gets a sibling `manifest.json`: row
count, first and last timestamp, duplicates, out-of-order rows, and the gaps
against the expected grid. A backtest can assert its coverage instead of
trusting the file. The S3 datasets write Parquet without one.

Two things worth knowing before you pick a coin list:

- **Coin names are case-sensitive**, exactly as Hyperliquid spells them:
  `kPEPE`, `kBONK`, not `KPEPE`. This is checked, not assumed. Upper-casing them
  used to make `l2book` a silent no-op and `trades`/`fills` download a full day
  for nothing.
- **`fills` and `liquidations` are all-coin streams.** A coin filter narrows what
  is *written*, never what is *downloaded* — the bill is the same for one coin as
  for two hundred. `trades` fetches each day once and fans out to every requested
  coin in that pass, so adding coins there is free too.

## Credentials

**`candles` and `funding` need none.** They read the public REST endpoint. Only
the four S3 datasets need AWS credentials, and only ever for *reading*.

Nothing is passed to boto3 explicitly (there is no `aws_access_key_id=` anywhere
in this package), so a key never travels through a function argument, a log line
or a traceback. Resolution is boto3's own, in this order:

1. `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (+ `AWS_SESSION_TOKEN`)
2. `AWS_PROFILE` → `~/.aws/credentials`
3. `~/.aws/config`, including SSO and assume-role profiles
4. Container credentials / EC2 instance profile

### Where to keep them

**Best: nowhere this tool can see.** Use a named profile or SSO and let the AWS
CLI own the secret:

```bash
aws configure --profile hl          # or: aws configure sso --profile hl
AWS_PROFILE=hl hl-data fills --start 2026-06-01 --root data
```

**Or a dotenv**, if you want per-project isolation. `$HL_DATA_ENV` if set, else
`./.env` in the current working directory:

```bash
# .env  — never commit this file
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
```

```bash
HL_DATA_ENV=~/secrets/hl.env hl-data l2book pull --coins BTC --start 2026-06-01
```

The file is loaded with `setdefault`, so **a shell-exported value always beats
the file**: a stale `.env` cannot silently override the profile you meant to
use. A missing file is not an error. There is no baked-in path, and nothing is
written back.

**Add `.env` to your `.gitignore` before you write one.** A committed key is a
key you have to rotate, and this one can be charged against.

### Minimum IAM policy

Read-only on the two buckets is enough. Requester-pays needs no special
permission of its own. It needs *your* policy to allow the call, and the header
this package always sends (`RequestPayer: requester`) is what moves the bill to
you:

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::hyperliquid-archive",
      "arn:aws:s3:::hyperliquid-archive/*",
      "arn:aws:s3:::hl-mainnet-node-data",
      "arn:aws:s3:::hl-mainnet-node-data/*"
    ]
  }]
}
```

No write, no delete, no other bucket. If the key leaks the exposure is a bill,
not your data, which is a good reason not to reuse a general-purpose key here.

`hl-data l2book probe` and `hl-data liquidations probe` print the AWS account
the credentials resolve to, so you can check *which* account is about to be
charged before you pull.

## Typed API

The package ships `py.typed`, so the public surface type-checks in a consumer's
project. Every written schema has a `TypedDict` mirroring its Arrow columns:

```python
from hyperliquid_data import FillRow, estimate, write_funding

est = estimate("fills", "2026-06-01", "2026-06-30")
est.total_bytes / 1024**3     # -> float, Estimate is a dataclass

def pnl_by_wallet(rows: list[FillRow]) -> dict[str, float]:
    ...                        # row["side"] narrows to Literal["B", "A"]
```

Called as a library, `estimate()` uses whatever AWS configuration the process
already has. The dotenv above is loaded by the CLI, not on import; call
`hyperliquid_data.load_env()` yourself if you want it.

`BookLevel`, `CandleRow`, `FundingRow`, `TradeRow`, `FillRow`, `L2SnapshotRow`,
`LiquidationBucketRow`, plus the `Dataset` and `Side` literals, are all
re-exported from the package root. A test asserts the `TypedDict` keys stay in
lockstep with the Arrow schemas, so the types cannot silently drift from the
files.

### What counts as public

Whatever `hyperliquid_data.__all__` lists — the schemas above, `estimate`,
the parquet writers and partition paths, the date and naming helpers, and the
universe/funding reference calls (`hl_universe`, `funding_first_ts`,
`binance_onboard_map`, …). That set is pinned by a test, so it cannot be
dropped or renamed by accident.

Anything else — importing a submodule and taking a name off it, especially a
`_`-prefixed one — is internal and can move in a patch release. If you need
something that is not exported, open an issue rather than reaching in; that
coupling is exactly what this list exists to prevent.

## Gotchas this bakes in

Each of these cost a wrong pull to discover.

- **`node_trades` is dead.** The prefix exists and returns empty files. Trades
  live in `node_fills_by_block/hourly/<YYYYMMDD>/<H>.lz4`, one JSONL line per
  block envelope `{"events": [[address, fill], …]}`, all coins in one stream.
- **The fill stream changed format on 2025-07-27.** `node_fills/hourly` holds
  2025-05-25 up to that date and matches the public API shape;
  `node_fills_by_block/hourly` holds it onward and is the current one. Pullers
  and estimator both resolve the prefix per date. Assuming a single prefix
  under-reports a range that crosses the handoff by a third, and a range
  entirely before it does not resolve at all.
- **Every trade appears twice**, taker (`crossed=true`) and maker
  (`crossed=false`), sharing a `tid`. `trades` keeps the taker row, since it
  carries the aggressor side (`side="B"` lifts asks, `"A"` hits bids); `fills`
  keeps both, plus the wallet.
- `liquidations` writes **one aggregated file per coin, not one per date**, so a
  re-run has to derive its coverage from the timestamps already on disk.
  Otherwise extending a six-month dataset by a week re-buys the six months.
- **`fundingHistory` is not months-capped.** With `startTime=0` it paginates
  forward 500 rows at a time from the coin's true listing date (BTC →
  2023-05-12, HL mainnet launch). Funding needs no S3 backfill.
- Duplicate funding settlements from a resumed pull would be double-counted by
  any downstream `resample().sum()`, so the writer collapses them, last wins.
- **L2 snapshots are event-driven** (~550 ms cadence, ~1.8/s on an active alt),
  not a fixed grid. `time` is node wall-clock at ns precision; the nested
  `data.time` is exchange time in ms.
- Parquet timestamps are **`timestamp[us, UTC]` everywhere**, so mixing sources
  never silently compares ms against ns.

## Status

Alpha. The public API is pinned (see [CHANGELOG](CHANGELOG.md)), but breaking
changes are still possible before 1.0.

Not included: HIP-3 equity-perp carry pulls. They need a Yahoo hedge leg, which
is out of scope for a Hyperliquid data library.

## Development

```bash
pip install -e ".[dev]"
pytest
pytest --cov=hyperliquid_data --cov-report=term-missing  # 88%, CI floor 80%
ruff check src tests
mypy
```

Tests are offline. S3 and the REST endpoint are stubbed, so the suite never
spends egress. Coverage is enforced at 80% in CI, and the paths that decide
what a pull *costs* and what the data *says* sit well above that, which is the
point of the number.

## License

MIT.
