Metadata-Version: 2.4
Name: atabet-data
Version: 1.0.5
Summary: Atabet-Data: A data retrieval library for financial data from multiple sources
Author: ATABET LIMITED
Author-email: ATABET LIMITED <info@atabet.com>
Maintainer: ATABET LIMITED
Maintainer-email: ATABET LIMITED <info@atabet.com>
License: Proprietary
Project-URL: Homepage, https://www.atabet.com
Project-URL: Repository, https://github.com/atabet-com/atabet-data
Project-URL: Issues, https://github.com/atabet-com/atabet-data/issues
Project-URL: Bug Tracker, https://github.com/atabet-com/atabet-data/issues
Keywords: finance,data,bloomberg,tushare,xtdata,yahoo
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pandas
Requires-Dist: numpy
Requires-Dist: tqdm
Provides-Extra: auth
Requires-Dist: atabet-token; extra == "auth"
Provides-Extra: blpapi
Requires-Dist: blpapi; extra == "blpapi"
Requires-Dist: xbbg; extra == "blpapi"
Provides-Extra: bql
Requires-Dist: bql; extra == "bql"
Provides-Extra: tushare
Requires-Dist: tushare; extra == "tushare"
Provides-Extra: xtdata
Requires-Dist: xtquant; extra == "xtdata"
Provides-Extra: yahoo
Requires-Dist: yfinance; extra == "yahoo"
Provides-Extra: s3
Requires-Dist: boto3; extra == "s3"
Requires-Dist: pyarrow; extra == "s3"
Provides-Extra: all
Requires-Dist: atabet-token; extra == "all"
Requires-Dist: blpapi; extra == "all"
Requires-Dist: xbbg; extra == "all"
Requires-Dist: bql; extra == "all"
Requires-Dist: tushare; extra == "all"
Requires-Dist: xtquant; extra == "all"
Requires-Dist: yfinance; extra == "all"
Requires-Dist: boto3; extra == "all"
Requires-Dist: pyarrow; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: author
Dynamic: maintainer

# Atabet-Data

A data retrieval library for financial data from multiple sources with token-based authentication.

## Authentication

Atabet-Data uses token-based authentication. **Token authentication is mandatory** for all API calls.

Install authentication support with `pip install atabet-data[auth]` (requires **atabet-token >= 1.2.0** for licensing-server session support).

### Token types

| Token source | Format | Usage |
|--------------|--------|-------|
| Licensing server (`/admin/generate_license`) | JWT (three dot-separated segments) | Default — `AtabetData(token=jwt)` uses `validation_method='licensing'` |
| Dev HMAC (`generate_secure_token` with `DEV_MODE=1`) | Hex string | Pass `validation_method='original'` |

### Using the AtabetData Class

All API access is through the `AtabetData` class, which validates authentication tokens:

```python
import os
import socket
from atabet_data import AtabetData

# Licensing JWT (default validation)
client = AtabetData(
    token=os.environ["ATABET_DATA_TOKEN"],
    licensing_server_url=os.environ.get("LICENSING_SERVER_URL", "http://39.101.65.167:5001"),
    device_name=os.environ.get("ATABET_DEVICE_NAME", os.environ.get("ATABET_CLIENT_ID", socket.gethostname())),
)

# Dev HMAC token (offline / local testing)
client = AtabetData(token=dev_token, validation_method="original")
```

Set environment variables in the same shell that runs Python:

```bash
export ATABET_DATA_TOKEN='eyJhbGci...'
export LICENSING_SERVER_URL='http://39.101.65.167:5001'
export ATABET_DEVICE_NAME='my-macbook'
```

### Session limits

Licensing JWTs enforce `max_sessions` on the server. The client automatically persists the server-issued `session_token` to `~/.atabet/session.json` (override with `ATABET_SESSION_STORE`) and reuses it on subsequent launches to heartbeat the existing session instead of registering a new one.

If you see `Session limit (N) reached`, close other instances or call `client.close()` to free your slot via `POST /logout`. Licenses must be issued with `application_id: "com.atabet.data"`.

**Note:** 
- Token authentication is **mandatory** and validated when creating the `AtabetData` instance
- If an invalid token is provided, `InvalidTokenError` is raised during initialization
- Once initialized, all methods can be called without additional authentication
- Contact the administrator to obtain a valid authentication token

### Yahoo Finance (`source='yahoo'`)

Install the optional extra (`pip install atabet-data[yahoo]` or `pip install yfinance`), then pass `source='yahoo'`. Tickers must be Yahoo symbols (for example `AAPL`, `700.HK`, `^GSPC`), not Bloomberg-style identifiers. Supported fields are `PX_OPEN`, `PX_HIGH`, `PX_LOW`, `PX_LAST`, and `PX_VOLUME` for snapshot and daily history. `get_datasets`, `get_mem_weights`, and `run_query` are not implemented for this channel.

**1-minute intraday:** pass ``freq='1M'``. Yahoo limits 1m bars to ~7 calendar days per request and only within the last ~60 days; wider or older ranges raise ``ValueError`` before calling yfinance. For multi-exchange intraday requests (e.g. ``AAPL`` + ``700.HK``), ``atabet_meta['exchange_tz']`` is a per-ticker dict when zones differ. With ``output_tz='UTC'``, each ticker is converted using its own exchange zone before columns are merged.

### Channel capabilities (native operations)

Channels do **not** all implement the same method matrix. Discover what each
source supports at runtime:

```python
from atabet_data import AtabetData, list_channels, describe_channel

print(list_channels())                 # summary: sources + operation names
print(describe_channel("yahoo"))       # params, ticker convention, field map

client = AtabetData(token=token)
df = client.run_operation(
    "localcsv",
    "get_historical_data",
    {
        "tickers": ["FUT.CBOT.ZC"],
        "flds": ["PX_LAST"],
        "start": "2024-01-01",
        "end": "2024-01-31",
        "csv_path": "/path/to/K_1D",
    },
)
```

| Source | Native operations | Notes |
|--------|-------------------|-------|
| `yahoo` | `get_data`, `get_historical_data` | Yahoo symbols; daily + intraday |
| `localcsv` | `get_historical_data` | Requires `csv_path` |
| `bql` | `get_data`, `get_historical_data`, `get_datasets`, `get_mem_weights` | Daily; Bloomberg IDs |
| `blpapi` | above + `run_query` | Daily; yellow keys |
| `bqia` | `get_historical_data` | Intraday only |
| `xtdata` | `get_historical_data`, `get_datasets` | Daily |
| `tushare` | `get_historical_data` | Daily; needs Tushare token |
| `s3` | `get_datasets` | No OHLCV history |

Calling an unsupported convenience method (e.g. `get_historical_data` on `s3`)
raises `NotImplementedError` with the available operations and a pointer to
`run_operation`. The static catalog is also shipped as
`atabet_data/channel_catalog.json`.

Each channel includes **`docs_urls`** (official / primary references) and
**`search_hints`** (good `web_search` queries) so agents know where to look
before inventing vendor API usage.

### Maintaining the channel catalog

**Edit here (source of truth):** [`atabet_data/capabilities.py`](atabet_data/capabilities.py)
— `CHANNEL_CAPABILITIES` entries (operations, params, ticker conventions,
`docs_urls`, `search_hints`, notes).

**Do not hand-edit** [`atabet_data/channel_catalog.json`](atabet_data/channel_catalog.json);
it is generated.

| Change type | Where to edit | Then |
|-------------|---------------|------|
| Vendor doc URL moved / renamed | `docs_urls` / `search_hints` on that channel in `capabilities.py` | Sync + test (below) |
| New operation or param on a channel | `OperationSpec` / `ParamSpec` in `capabilities.py` **and** implement the function under `from_<channel>/` | Sync + test |
| New data channel | Add `ChannelCapability` + `from_<channel>/` module + wire in `api.py` `load_api` | Sync + test |
| QI agent behaviour only (wording) | atabet-quant-intelligence `src/lib/skills/default-skill.ts` | No catalog sync |

```bash
# from atabet-data repo root (expects sibling ../atabet-quant-intelligence by default)
python scripts/sync_channel_catalog.py
# optional: --qi-path /path/to/atabet-quant-intelligence   or   --skip-qi
pytest tests/test_capabilities.py
```

Commit **both** repos when the catalog changes:
- atabet-data: `capabilities.py` + `channel_catalog.json`
- atabet-quant-intelligence: `src/lib/data-channels/channel-catalog.json`

Do **not** hardcode vendor doc URLs only in the QI skill — the capability
catalog is the source of truth. See also `scripts/sync_channel_catalog.py`.

### Index and timezone contract

Historical outputs use a consistent multi-index layout across channels:

- **Daily** data: index levels ``(FLDS, DATE)`` — naive calendar dates at midnight (no timezone).
- **Intraday** data: index levels ``(FLDS, DATETIME)`` — exchange-local timestamps by default; pass ``output_tz='UTC'`` to convert (``exchange_tz`` may be required for naive sources such as ``localcsv``). Yahoo and ``localcsv`` multi-exchange intraday apply UTC conversion per ticker before columns are merged.
- **Parameters**: ``get_historical_data(..., freq='1M', output_tz='UTC')`` — ``freq`` selects bar size (``'1D'`` daily default; intraday e.g. ``'1M'``, ``'1H'`` for Yahoo); ``output_tz`` converts intraday timestamps when supported. For ``localcsv``, ``exchange_tz`` may be a single IANA string or a ``{ticker: zone}`` dict.
- **Metadata**: returned DataFrames include ``df.attrs['atabet_meta']`` with ``freq``, ``index_semantics`` (``calendar_date`` or ``timestamp``), ``output_tz``, ``exchange_tz`` (string or per-ticker dict for multi-exchange Yahoo/localcsv intraday), and ``frame_schema`` (e.g. ``members_weights`` for BQL index weights).

### Channel matrix (historical)

| Source | Daily `(FLDS, DATE)` | Intraday | ``freq`` / ``output_tz`` | ``atabet_meta`` |
|--------|----------------------|----------|--------------------------|-----------------|
| `bql` | yes | no | ignored (daily only) | yes |
| `blpapi` | yes | no | stripped by wrapper | yes |
| `xtdata` | yes | no | stripped by wrapper | yes |
| `yahoo` | yes | yes | supported | yes |
| `localcsv` | yes | yes | supported | yes |
| `bqia` | no | yes | supported | yes |
| `tushare` | yes | no | intraday raises | yes |

Passing ``freq`` or ``output_tz`` to daily-only channels (``blpapi``, ``xtdata``) is safe — the wrapper strips unsupported kwargs before calling the channel.

### Available Methods

The `AtabetData` class provides:

**Channel-native (preferred for full coverage)**
- `list_channels()` — summarize registered channels and operations
- `describe_channel(source)` — full params / ticker conventions / field map
- `run_operation(channel, operation, params)` — execute a native operation

**Convenience facades (best-effort OHLCV / datasets where supported)**
- `get_data()` — snapshot
- `get_historical_data()` — historical OHLCV
- `get_datasets()` — datasets
- `get_mem_weights()` — index member weights
- `run_query()` — raw vendor query (primarily blpapi)

Module-level helpers (no auth): `list_channels`, `describe_channel`, `catalog_as_dict`.

## Technical Analysis (`ta` accessor)

Atabet-Data ships with a built-in technical analysis module. Any OHLCV DataFrame
can compute indicators directly via the `.ta` accessor (no external dependency
required):

```python
from atabet_data import AtabetData

client = AtabetData(token=your_token)
df = client.get_historical_data(
    tickers=["AAPL US Equity"],
    flds=["PX_OPEN", "PX_HIGH", "PX_LOW", "PX_LAST", "PX_VOLUME"],
    start="2024-01-01", end="2024-12-31", source="yahoo",
)
ohlcv = df.ext.to_ohlcv("AAPL US Equity")

# Individual indicators
ohlcv.ta.sma(length=20)        # Simple Moving Average
ohlcv.ta.rsi(length=14)        # Relative Strength Index
ohlcv.ta.macd()                # MACD (returns DataFrame with line/signal/histogram)
ohlcv.ta.bbands(length=20)     # Bollinger Bands
ohlcv.ta.atr(length=14)        # Average True Range
ohlcv.ta.obv()                 # On-Balance Volume

# Append results to the DataFrame
ohlcv.ta.sma(length=50, append=True)
ohlcv.ta.rsi(length=14, append=True)

# Batch: run a preset strategy (Common or All)
ohlcv.ta.strategy("Common")

# Or define a custom strategy
from atabet_data.ta.strategy import Strategy
my_strat = Strategy(name="Quick", ta=[
    {"kind": "ema", "length": 9},
    {"kind": "rsi", "length": 7},
    {"kind": "bbands", "length": 20, "std": 2.5},
])
ohlcv.ta.strategy(my_strat)
```

### Indicator categories

| Category | Indicators |
|-------------|------------|
| **Overlap** | `sma`, `ema`, `wma`, `rma`, `dema`, `tema`, `hma`, `vwap`, `hlc3` |
| **Momentum** | `rsi`, `macd`, `stoch`, `roc`, `cci`, `willr`, `mfi` |
| **Trend** | `adx`, `aroon`, `psar` |
| **Volatility** | `true_range`, `atr`, `bbands`, `kc`, `donchian` |
| **Volume** | `obv`, `ad`, `adosc`, `cmf` |
| **Statistics** | `stdev`, `variance`, `mad`, `zscore` |
| **Performance** | `log_return`, `percent_return`, `drawdown` |

### Functional API

All indicators are also available as standalone functions:

```python
from atabet_data.ta import sma, rsi, macd
sma(ohlcv["close"], length=20)
rsi(ohlcv["close"], length=14)
```

### Custom indicators

Register your own indicators at runtime:

```python
from atabet_data.ta.custom import bind, create_dir, import_dir

# Scaffold a directory tree, write custom modules, then import them
create_dir("~/my_indicators")
import_dir("~/my_indicators")
```

See `samples/ta_example.py` for a complete walkthrough.

## Environment preparations


```bash
conda create -n py39_atabet_data python=3.9
conda activate py39_atabet_data
(py39_atabet_data_test) conda install conda-forge::cython
(py39_atabet_data_test) conda install anaconda::pandas
(py39_atabet_data_test) conda install anaconda::setuptools # Windows
```


### Packaging

```bash
# Basic usage - will cythonize all .py files including __init__.py
python atabet-data/run_packaging.py

# Remove source files from the wheel
python atabet-data/run_packaging.py --remove-source

# Specify a different module name, version, or distribution directory
python run_packaging.py --module="my-module" --version="2.0.0" --dist-dir="dist"

# Combine options
python run_packaging.py --remove-source --module="my-module" --version="2.0.0"
```

### Test

Unit tests run offline with mocked channels (no Bloomberg, Yahoo, or licensing server required):

```bash
conda activate atabet_data_env
pytest tests/ -v
```

Integration tests (real licensing server) are marked `@pytest.mark.integration` and excluded by default. Run them explicitly when the server is available:

```bash
pytest tests/ -m integration -v
```

GitHub Actions runs unit tests on every push/PR via [`.github/workflows/test.yml`](.github/workflows/test.yml). Wheel builds remain gated by `[build ci]` in [`.github/workflows/build-wheels.yml`](.github/workflows/build-wheels.yml).

Sample scripts live in [`samples/`](samples/) — see [`samples/README.md`](samples/README.md) for offline-friendly localcsv demos using bundled `sample_data/`.

## GitHub Actions: tests, wheels, and PyPI

**Unit tests:** [`.github/workflows/test.yml`](.github/workflows/test.yml) runs `pytest` on Ubuntu for every push and pull request (integration tests excluded by default).

**Wheel builds:** [`.github/workflows/build-wheels.yml`](.github/workflows/build-wheels.yml). Wheels are built with [cibuildwheel](https://cibuildwheel.readthedocs.io/); options live in [`pyproject.toml`](pyproject.toml) under `[tool.cibuildwheel]`.

**When jobs run (push):** the Linux / Windows / macOS matrix runs only if the **head commit message** includes one of:

| Tag | Effect |
|-----|--------|
| `[build ci]` (or `[Build CI]`, `[BUILD CI]`) | Build wheels and upload per-OS artifacts; no publish. |
| `[pub pypi]` (or `[Pub PyPI]`, `[PUB PYPI]`) | Build wheels, then publish merged wheels to [PyPI](https://pypi.org/) (production). |
| `[pub test.pypi]` (or `[Pub test.pypi]`, `[PUB TEST.PYPI]`) | Build wheels, then publish to [TestPyPI](https://test.pypi.org/). |

Plain pushes (no tag above) skip the matrix so CI does not run on every commit.

**Manual runs:** **Actions → Build wheels → Run workflow** runs the build matrix only; publishing is by **push** with `[pub pypi]` or `[pub test.pypi]` in the commit message.

**Authentication (publish):** the workflow uses **Trusted Publishing (OIDC)** by default — no API token in GitHub secrets. Register the GitHub repo and workflow file on each index (project **Manage → Publishing**). See the comment block at the top of `build-wheels.yml` for OIDC vs API-token options.

Private repositories are supported for OIDC the same as public ones.
