Metadata-Version: 2.4
Name: OrderPulse
Version: 0.2.31
Summary: High-performance exchange feed parser and orderflow analytics engine with Rust and Python bindings
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# fastreader / OrderPulse

A high-performance Rust + Python library for reading NSE binary market feed files, extracting order/trade messages, and building an in-memory order book snapshot from those messages.

This library is designed for quant developers, market-data engineers, and backtesting systems that need fast parsing of large binary feed files without loading unnecessary data into Python.

---

## What this library does

`fastreader` gives Python users three main tools:

| Class | Purpose | Best use case |
|---|---|---|
| `MessageCacheReader` | Loads all order/trade messages into memory first | Backtesting, repeated analysis, debugging, smaller/medium files |
| `StreamingBinaryLoader` | Reads one binary message at a time directly from disk | Very large files, low-memory processing, live-style replay |
| `OrderbookBuilder` | Applies order/trade messages and builds market depth | Creating snapshots, full depth, top-N levels, payoff/orderbook analytics |

Internally, the heavy work is done in Rust for speed, while Python receives simple classes, lists, dictionaries, and strings.

---

## Architecture explanation

```text
Binary Feed File
      |
      |  contains raw NSE order/trade packets
      v
+----------------------------+
| Binary Parser Layer        |
| - validates first header   |
| - skips padding spaces     |
| - reads message type       |
| - parses OrderPacket       |
| - parses TradePacket       |
+----------------------------+
      |
      v
+----------------------------+
| Reader Layer               |
|                            |
| MessageCacheReader         |
|   loads all messages       |
|   stores in RAM            |
|                            |
| StreamingBinaryLoader      |
|   keeps file handle open   |
|   reads next message only  |
+----------------------------+
      |
      v
+----------------------------+
| OrderbookBuilder           |
| - applies filters          |
| - processes N/M/X/T msgs   |
| - updates OrderBookManager |
+----------------------------+
      |
      v
+----------------------------+
| Python Output Layer        |
| - snapshot dict            |
| - full depth dict          |
| - CSV row string           |
| - formatted message string |
+----------------------------+
```

### Message flow

1. A binary file is opened.
2. The first valid message header is checked.
3. Each message is identified by `msg_type`.
4. Supported message types are parsed:
   - `N` = new order
   - `M` = modified order
   - `X` = cancelled/deleted order
   - `T` = trade
5. `OrderbookBuilder` sends the parsed message into `OrderBookManager`.
6. The user asks for snapshot, full depth, or CSV row output.

---

## Installation / build

This project is a Rust extension module exposed to Python using `pyo3`.

Typical local build:

```bash
maturin develop --release
```

Or build wheel:

```bash
maturin build --release
```

Then use in Python:

```python
from fastreader import MessageCacheReader, StreamingBinaryLoader, OrderbookBuilder
```

---

## Quick start: stream file and build order book

```python
from fastreader import StreamingBinaryLoader, OrderbookBuilder

file_path = "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"

reader = StreamingBinaryLoader()
reader.open_stream(file_path, count_messages=False)

builder = OrderbookBuilder()

while True:
    processed = builder.orderbook_add_msg(reader)
    if not processed:
        break

snapshot = builder.get_snapshot(token=1001, levels=5)
print(snapshot)
```

Example expected output:

```python
{
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150), (245650, 300), (245600, 75)],
    'asks': [(245800, 200), (245850, 100), (245900, 250)]
}
```

If no order book exists for that token:

```python
{
    'token': 1001,
    'found': False,
    'mid_price': 0,
    'best_bid': None,
    'best_ask': None,
    'spread': None,
    'bids': [],
    'asks': []
}
```

---

# Public Python API

## 1. `MessageCacheReader`

`MessageCacheReader` loads all order and trade messages into RAM. Use this when you want to read the file once and then analyze the messages many times.

### When to use

Use `MessageCacheReader` when:

- the file is small or medium size,
- you want repeated access to all messages,
- you want message summaries,
- you want to build order book multiple times with different filters.

Avoid it for extremely large binary files because it stores all parsed messages in memory.

---

## `MessageCacheReader()`

Creates an empty cache reader.

```python
from fastreader import MessageCacheReader

reader = MessageCacheReader()
```

Expected output:

```python
# No output. Empty reader object is created.
```

---

## `load_to_cache(file_path)`

Loads all supported order/trade messages from a binary file into memory.

### Signature

```python
reader.load_to_cache(file_path: str) -> int
```

### Parameters

| Parameter | Type | Meaning |
|---|---|---|
| `file_path` | `str` | Full path of the binary feed file |

### Returns

Number of messages loaded into cache.

### Example

```python
from fastreader import MessageCacheReader

reader = MessageCacheReader()
count = reader.load_to_cache("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin")
print(count)
```

Expected output:

```python
1254300
```

Meaning: `1,254,300` order/trade messages were loaded into RAM.

---

## `get_all_messages()`

Returns all cached messages as readable strings.

### Signature

```python
reader.get_all_messages() -> list[str]
```

### Example

```python
messages = reader.get_all_messages()
print(messages[:2])
```

Expected output:

```python
[
    "Order Message: SeqNo 1, MsgLen 38, MsgType 'N', ExchTs 1766998800000000000, LocalTs 1766998800000100000, OrderId 123456789, Token 1001, Side 'B', Price 245700, Quantity 150, Missed 0",
    "Trade Message: SeqNo 2, MsgLen 45, MsgType 'T', ExchTs 1766998801000000000, LocalTs 1766998801000100000, BuyOrderId 123456789, SellOrderId 987654321, Token 1001, Price 245750, Quantity 50, Missed 0"
]
```

---

## `get_order_message()`

Returns only order messages from the cache.

### Signature

```python
reader.get_order_message() -> list[str]
```

Order messages include:

- `N` = new order
- `M` = modify order
- `X` = cancel/delete order

### Example

```python
orders = reader.get_order_message()
print(len(orders))
print(orders[0])
```

Expected output:

```python
984500
Order Message: SeqNo 1, MsgLen 38, MsgType 'N', ExchTs 1766998800000000000, LocalTs 1766998800000100000, OrderId 123456789, Token 1001, Side 'B', Price 245700, Quantity 150, Missed 0
```

> Note: In the current `lib.rs`, the function name is `get_order_message()`, not `get_all_order_message()`.

---

## `get_trade_message()`

Returns only trade messages from the cache.

### Signature

```python
reader.get_trade_message() -> list[str]
```

### Example

```python
trades = reader.get_trade_message()
print(len(trades))
print(trades[0])
```

Expected output:

```python
269800
Trade Message: SeqNo 2, MsgLen 45, MsgType 'T', ExchTs 1766998801000000000, LocalTs 1766998801000100000, BuyOrderId 123456789, SellOrderId 987654321, Token 1001, Price 245750, Quantity 50, Missed 0
```

---

## `get_all_trade_message()`

Alias for `get_trade_message()`.

### Signature

```python
reader.get_all_trade_message() -> list[str]
```

### Example

```python
trades = reader.get_all_trade_message()
print(trades[:1])
```

Expected output:

```python
[
    "Trade Message: SeqNo 2, MsgLen 45, MsgType 'T', ExchTs 1766998801000000000, LocalTs 1766998801000100000, BuyOrderId 123456789, SellOrderId 987654321, Token 1001, Price 245750, Quantity 50, Missed 0"
]
```

---

## `get_cache_summary()`

Returns metadata about the cached file and message counts.

### Signature

```python
reader.get_cache_summary() -> dict
```

### Example

```python
summary = reader.get_cache_summary()
print(summary)
```

Expected output:

```python
{
    'file_source': '/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin',
    'total_messages': 1254300,
    'total_orders': 984500,
    'total_trades': 269800,
    'memory_usage_bytes': 80275200
}
```

---

# 2. `StreamingBinaryLoader`

`StreamingBinaryLoader` keeps a binary file open and reads one message at a time. This is the best choice for large files because it does not load all messages into RAM.

### When to use

Use `StreamingBinaryLoader` when:

- the binary file is very large,
- you want low-memory processing,
- you want to replay feed messages one by one,
- you want to build order book directly from disk.

---

## `StreamingBinaryLoader()`

Creates a new streaming reader.

```python
from fastreader import StreamingBinaryLoader

reader = StreamingBinaryLoader()
```

Expected output:

```python
# No output. Empty stream reader object is created.
```

---

## `open_stream(file_path, count_messages=True)`

Opens the binary feed file for streaming.

### Signature

```python
reader.open_stream(file_path: str, count_messages: bool = True) -> int
```

### Parameters

| Parameter | Type | Meaning |
|---|---|---|
| `file_path` | `str` | Full path of binary feed file |
| `count_messages` | `bool` | If `True`, scans the file once to count messages. If `False`, opens quickly and returns `0`. |

### Important meaning of `open_stream`

`open_stream()` does not load the full file into memory. It only:

1. opens the file handle,
2. validates that the file starts with a valid feed message,
3. resets the cursor to the start,
4. optionally counts total messages,
5. keeps the file ready for `get_next_message()` or `orderbook_add_msg()`.

### Example: fast open without counting

```python
reader = StreamingBinaryLoader()
count = reader.open_stream(
    "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin",
    count_messages=False,
)
print(count)
```

Expected output:

```python
0
```

Meaning: file is opened, but total message count was skipped for speed.

### Example: open and count messages

```python
reader = StreamingBinaryLoader()
count = reader.open_stream(
    "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin",
    count_messages=True,
)
print(count)
```

Expected output:

```python
1254300
```

Meaning: file was scanned once and `1,254,300` messages were found.

---

## `reset_cursor()`

Moves the file cursor back to the start.

### Signature

```python
reader.reset_cursor() -> None
```

### Example

```python
reader.get_next_message()
reader.get_next_message()

reader.reset_cursor()

first_message_again = reader.get_next_message()
print(first_message_again)
```

Expected output:

```python
Order Message: SeqNo 1, MsgLen 38, MsgType 'N', ExchTs 1766998800000000000, LocalTs 1766998800000100000, OrderId 123456789, Token 1001, Side 'B', Price 245700, Quantity 150, Missed 0
```

---

## `get_next_message()`

Reads the next message from the stream and returns it as a formatted string.

### Signature

```python
reader.get_next_message() -> str
```

### Example

```python
reader = StreamingBinaryLoader()
reader.open_stream("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin", count_messages=False)

print(reader.get_next_message())
print(reader.get_next_message())
```

Expected output:

```python
Order Message: SeqNo 1, MsgLen 38, MsgType 'N', ExchTs 1766998800000000000, LocalTs 1766998800000100000, OrderId 123456789, Token 1001, Side 'B', Price 245700, Quantity 150, Missed 0
Trade Message: SeqNo 2, MsgLen 45, MsgType 'T', ExchTs 1766998801000000000, LocalTs 1766998801000100000, BuyOrderId 123456789, SellOrderId 987654321, Token 1001, Price 245750, Quantity 50, Missed 0
```

When the file ends:

```python
END
```

---

# 3. `OrderbookBuilder`

`OrderbookBuilder` is the engine that builds and queries the order book.

It accepts messages from:

1. `MessageCacheReader`,
2. `StreamingBinaryLoader`,
3. Python `list[dict]` decoded messages.

---

## `OrderbookBuilder()`

Creates a new empty order book builder.

```python
from fastreader import OrderbookBuilder

builder = OrderbookBuilder()
```

Expected output:

```python
# No output. Empty order book is created.
```

---

## `apply_filter(logic_criteria=None)`

Filters which message types should be processed by the order book builder.

### Signature

```python
builder.apply_filter(logic_criteria: list[str] | None = None) -> None
```

### Supported message filters

| Filter | Meaning |
|---|---|
| `"N"` | Process new orders |
| `"M"` | Process modified orders |
| `"X"` | Process cancelled/deleted orders |
| `"T"` | Process trades |
| `None` | Process all supported messages |

### Example: process only order add/modify/delete

```python
builder.apply_filter(["N", "M", "X"])
```

Expected output:

```python
# No output. Future build/process calls will skip trades.
```

### Example: process only trades

```python
builder.apply_filter(["T"])
```

### Example: clear filter

```python
builder.apply_filter(None)
```

---

## `orderbook_add_msg(source)`

Reads exactly one message from a `StreamingBinaryLoader` and applies it to the order book.

### Signature

```python
builder.orderbook_add_msg(source: StreamingBinaryLoader) -> bool
```

### Returns

| Return | Meaning |
|---|---|
| `True` | One message was read and applied to the order book |
| `False` | Stream ended, or the message was skipped by filter |

### Example

```python
from fastreader import StreamingBinaryLoader, OrderbookBuilder

reader = StreamingBinaryLoader()
reader.open_stream("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin", count_messages=False)

builder = OrderbookBuilder()

processed = builder.orderbook_add_msg(reader)
print("Processed:", processed)
```

Expected output:

```python
Processed: True
```

### Practical loop

```python
while True:
    processed = builder.orderbook_add_msg(reader)
    if not processed:
        break
```

> Important: If you use a filter and the next message is skipped, this function returns `False`. In that case, a simple `break` loop may stop early. For full-file filtered processing, prefer `build_from_source(reader)` because it continues until the stream ends or the limit is reached.

---

## `build_from_list(source)`

Builds order book from either:

1. a `MessageCacheReader`, or
2. a Python `list[dict]` containing decoded messages.

### Signature

```python
builder.build_from_list(source: MessageCacheReader | list[dict]) -> int
```

### Returns

Number of messages actually processed.

---

## Example: build from `MessageCacheReader`

```python
from fastreader import MessageCacheReader, OrderbookBuilder

reader = MessageCacheReader()
reader.load_to_cache("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin")

builder = OrderbookBuilder()
count = builder.build_from_list(reader)

print("Processed:", count)
```

Expected output:

```python
Processed: 1254300
```

---

## Example: build from Python list of dictionaries

```python
from fastreader import OrderbookBuilder

messages = [
    {
        "msg_type": "N",
        "exch_ts": 1766998800000000000,
        "order_id": 101,
        "token": 1001,
        "order_type": "B",
        "price": 245700,
        "quantity": 150,
        "local_ts": 1766998800000100000,
        "flags": False,
    },
    {
        "msg_type": "N",
        "exch_ts": 1766998800001000000,
        "order_id": 102,
        "token": 1001,
        "order_type": "S",
        "price": 245800,
        "quantity": 200,
        "local_ts": 1766998800001100000,
        "flags": False,
    },
]

builder = OrderbookBuilder()
count = builder.build_from_list(messages)
print("Processed:", count)
print(builder.get_snapshot(1001, levels=5))
```

Expected output:

```python
Processed: 2
{
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150)],
    'asks': [(245800, 200)]
}
```

### Required dictionary keys for order messages

For `msg_type` = `N`, `M`, or `X`:

| Key | Required | Type | Meaning |
|---|---:|---|---|
| `msg_type` | yes | `str` or `int` | `N`, `M`, or `X` |
| `order_id` | yes | `int` | Exchange order id |
| `token` | yes | `int` | Instrument token |
| `order_type` | yes | `str` or `int` | `B` for buy, `S` for sell |
| `price` | yes | `int` | Price in feed tick format |
| `quantity` | yes | `int` | Quantity |
| `exch_ts` | no | `int` | Exchange timestamp; default `0` |
| `local_ts` | no | `int` | Local timestamp; default `0` |
| `flags` | no | `bool` | Missed flag; default `False` |

### Required dictionary keys for trade messages

For `msg_type` = `T`:

| Key | Required | Type | Meaning |
|---|---:|---|---|
| `msg_type` | yes | `str` or `int` | `T` |
| `buy_order_id` | yes | `int` | Buy order id |
| `sell_order_id` | yes | `int` | Sell order id |
| `token` | yes | `int` | Instrument token |
| `trade_quantity` | yes | `int` | Traded quantity |
| `trade_price` | no | `int` | Traded price; default `0` |
| `exch_ts` | no | `int` | Exchange timestamp; default `0` |
| `local_ts` | no | `int` | Local timestamp; default `0` |
| `flags` | no | `bool` | Missed flag; default `False` |

---

## `build_from_source(source, limit=None)`

Builds the order book from either a cache reader or a streaming reader.

### Signature

```python
builder.build_from_source(source: MessageCacheReader | StreamingBinaryLoader, limit: int | None = None) -> int
```

### Parameters

| Parameter | Type | Meaning |
|---|---|---|
| `source` | `MessageCacheReader` or `StreamingBinaryLoader` | Source of messages |
| `limit` | `int` or `None` | Maximum number of processed messages for streaming mode |

### Example: build from stream with no limit

```python
reader = StreamingBinaryLoader()
reader.open_stream("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin", count_messages=False)

builder = OrderbookBuilder()
count = builder.build_from_source(reader)

print("Processed:", count)
```

Expected output:

```python
Processed: 1254300
```

### Example: process only first 10,000 messages

```python
reader = StreamingBinaryLoader()
reader.open_stream("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin", count_messages=False)

builder = OrderbookBuilder()
count = builder.build_from_source(reader, limit=10_000)

print("Processed:", count)
```

Expected output:

```python
Processed: 10000
```

---

## `get_snapshot(token, levels=None)`

Returns top-N order book levels for one instrument token.

### Signature

```python
builder.get_snapshot(token: int, levels: int | None = None) -> dict
```

Default `levels` is `5`.

### Example

```python
snapshot = builder.get_snapshot(token=1001, levels=5)
print(snapshot)
```

Expected output:

```python
{
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150), (245650, 300), (245600, 75)],
    'asks': [(245800, 200), (245850, 100), (245900, 250)]
}
```

---

## `get_orderbook_snapshot(token, levels=None)`

Alias for `get_snapshot()`.

### Signature

```python
builder.get_orderbook_snapshot(token: int, levels: int | None = None) -> dict
```

### Example

```python
snapshot = builder.get_orderbook_snapshot(1001, levels=5)
print(snapshot)
```

Expected output:

```python
{
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150), (245650, 300), (245600, 75)],
    'asks': [(245800, 200), (245850, 100), (245900, 250)]
}
```

---

## `get_full_depth(token)`

Returns full available order book depth for one token.

### Signature

```python
builder.get_full_depth(token: int) -> dict
```

### Example

```python
full_depth = builder.get_full_depth(1001)
print(full_depth)
```

Expected output:

```python
{
    'token': 1001,
    'found': True,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [
        (245700, 150),
        (245650, 300),
        (245600, 75),
        (245550, 50)
    ],
    'asks': [
        (245800, 200),
        (245850, 100),
        (245900, 250),
        (245950, 125)
    ]
}
```

If token is not found:

```python
{
    'token': 999999,
    'found': False,
    'best_bid': None,
    'best_ask': None,
    'spread': None,
    'bids': [],
    'asks': []
}
```

---

## `snapshot_header()`

Returns the CSV header for `get_snapshot_row()`.

### Signature

```python
builder.snapshot_header() -> str
```

### Example

```python
print(builder.snapshot_header())
```

Expected output:

```csv
local_ts,exch_ts,mid_price,bid_price_0,bid_qty_0,ask_price_0,ask_qty_0,bid_price_1,bid_qty_1,ask_price_1,ask_qty_1,bid_price_2,bid_qty_2,ask_price_2,ask_qty_2,bid_price_3,bid_qty_3,ask_price_3,ask_qty_3,bid_price_4,bid_qty_4,ask_price_4,ask_qty_4
```

---

## `get_snapshot_row(token, levels=None)`

Returns one CSV row string for a token snapshot.

### Signature

```python
builder.get_snapshot_row(token: int, levels: int | None = None) -> str
```

Default `levels` is `5`.

### Example

```python
print(builder.snapshot_header())
print(builder.get_snapshot_row(1001, levels=5))
```

Expected output:

```csv
local_ts,exch_ts,mid_price,bid_price_0,bid_qty_0,ask_price_0,ask_qty_0,bid_price_1,bid_qty_1,ask_price_1,ask_qty_1,bid_price_2,bid_qty_2,ask_price_2,ask_qty_2,bid_price_3,bid_qty_3,ask_price_3,ask_qty_3,bid_price_4,bid_qty_4,ask_price_4,ask_qty_4
0,0,245750,245700,150,245800,200,245650,300,245850,100,245600,75,245900,250,245550,50,245950,125,0,0,0,0
```

> Current implementation sets `local_ts` and `exch_ts` to `0` in the CSV row. The order book levels and mid price come from the order book state.

---

# Complete examples

## Example 1: Read messages one by one

```python
from fastreader import StreamingBinaryLoader

reader = StreamingBinaryLoader()
reader.open_stream("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin", count_messages=False)

for _ in range(5):
    msg = reader.get_next_message()
    print(msg)
```

Expected output:

```text
Order Message: SeqNo 1, MsgLen 38, MsgType 'N', ExchTs 1766998800000000000, LocalTs 1766998800000100000, OrderId 123456789, Token 1001, Side 'B', Price 245700, Quantity 150, Missed 0
Trade Message: SeqNo 2, MsgLen 45, MsgType 'T', ExchTs 1766998801000000000, LocalTs 1766998801000100000, BuyOrderId 123456789, SellOrderId 987654321, Token 1001, Price 245750, Quantity 50, Missed 0
...
```

---

## Example 2: Build order book from cache

```python
from fastreader import MessageCacheReader, OrderbookBuilder

file_path = "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"

data = MessageCacheReader()
loaded = data.load_to_cache(file_path)
print("Loaded:", loaded)

builder = OrderbookBuilder()
processed = builder.build_from_list(data)
print("Processed:", processed)

print(builder.get_snapshot(1001, 5))
```

Expected output:

```python
Loaded: 1254300
Processed: 1254300
{
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150), (245650, 300), (245600, 75)],
    'asks': [(245800, 200), (245850, 100), (245900, 250)]
}
```

---

## Example 3: Build order book from stream

```python
from fastreader import StreamingBinaryLoader, OrderbookBuilder

file_path = "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"

reader = StreamingBinaryLoader()
reader.open_stream(file_path, count_messages=False)

builder = OrderbookBuilder()
processed = builder.build_from_source(reader)

print("Processed:", processed)
print(builder.get_snapshot(1001, 5))
```

Expected output:

```python
Processed: 1254300
{
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150), (245650, 300), (245600, 75)],
    'asks': [(245800, 200), (245850, 100), (245900, 250)]
}
```

---

## Example 4: Save snapshots to CSV

```python
from fastreader import StreamingBinaryLoader, OrderbookBuilder

file_path = "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"
token = 1001

reader = StreamingBinaryLoader()
reader.open_stream(file_path, count_messages=False)

builder = OrderbookBuilder()
builder.build_from_source(reader)

with open("snapshot.csv", "w") as f:
    f.write(builder.snapshot_header() + "\n")
    f.write(builder.get_snapshot_row(token, levels=5) + "\n")
```

Expected `snapshot.csv`:

```csv
local_ts,exch_ts,mid_price,bid_price_0,bid_qty_0,ask_price_0,ask_qty_0,bid_price_1,bid_qty_1,ask_price_1,ask_qty_1,bid_price_2,bid_qty_2,ask_price_2,ask_qty_2,bid_price_3,bid_qty_3,ask_price_3,ask_qty_3,bid_price_4,bid_qty_4,ask_price_4,ask_qty_4
0,0,245750,245700,150,245800,200,245650,300,245850,100,245600,75,245900,250,245550,50,245950,125,0,0,0,0
```

---

## Example 5: Process only order messages, skip trades

```python
from fastreader import StreamingBinaryLoader, OrderbookBuilder

reader = StreamingBinaryLoader()
reader.open_stream("/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin", count_messages=False)

builder = OrderbookBuilder()
builder.apply_filter(["N", "M", "X"])

processed = builder.build_from_source(reader)
print("Processed only orders:", processed)
```

Expected output:

```python
Processed only orders: 984500
```

---

# Error handling

The Rust layer converts common failures into Python exceptions.

## File not found

```python
reader = StreamingBinaryLoader()
reader.open_stream("missing_file.bin")
```

Expected error:

```text
RuntimeError: No such file or directory
```

## Invalid binary file

If the first valid message type is not one of `T`, `N`, `M`, or `X`, opening the file fails.

Expected error:

```text
RuntimeError: invalid first message type: <value>
```

## Wrong source type

```python
builder = OrderbookBuilder()
builder.build_from_source("not_a_reader")
```

Expected error:

```text
TypeError: build_from_source expects MessageCacheReader or StreamingBinaryLoader
```

## Missing dictionary key

```python
builder.build_from_list([
    {"msg_type": "N", "token": 1001}
])
```

Expected error:

```text
TypeError: missing key: order_id
```

---

# Performance guide

## Use `StreamingBinaryLoader` for large files

```python
reader.open_stream(file_path, count_messages=False)
builder.build_from_source(reader)
```

This avoids loading the full file into RAM.

## Use `MessageCacheReader` for repeated research

```python
reader.load_to_cache(file_path)
builder_1.build_from_list(reader)
builder_2.build_from_list(reader)
```

This is faster for repeated analysis but uses more memory.

## Avoid `count_messages=True` for production replay

`count_messages=True` scans the file once before real processing. For very large files, prefer:

```python
reader.open_stream(file_path, count_messages=False)
```

---

# Recommended user workflows

## For quick debugging

```python
reader = StreamingBinaryLoader()
reader.open_stream(file_path, count_messages=False)
print(reader.get_next_message())
```

## For backtesting

```python
cache = MessageCacheReader()
cache.load_to_cache(file_path)

builder = OrderbookBuilder()
builder.build_from_list(cache)
```

## For production-style low-memory processing

```python
stream = StreamingBinaryLoader()
stream.open_stream(file_path, count_messages=False)

builder = OrderbookBuilder()
builder.build_from_source(stream)
```

## For one-token snapshot

```python
snapshot = builder.get_snapshot(token=1001, levels=5)
```

## For full book depth

```python
depth = builder.get_full_depth(token=1001)
```

## For CSV export

```python
header = builder.snapshot_header()
row = builder.get_snapshot_row(token=1001, levels=5)
```

---

# Developer notes from current `lib.rs`

The current module exposes only these Python classes:

```python
MessageCacheReader
StreamingBinaryLoader
OrderbookBuilder
```

The current module exports these Python-visible methods:

```text
MessageCacheReader.new
MessageCacheReader.load_to_cache
MessageCacheReader.get_all_messages
MessageCacheReader.get_order_message
MessageCacheReader.get_trade_message
MessageCacheReader.get_all_trade_message
MessageCacheReader.get_cache_summary

StreamingBinaryLoader.new
StreamingBinaryLoader.open_stream
StreamingBinaryLoader.reset_cursor
StreamingBinaryLoader.get_next_message

OrderbookBuilder.new
OrderbookBuilder.apply_filter
OrderbookBuilder.orderbook_add_msg
OrderbookBuilder.build_from_list
OrderbookBuilder.build_from_source
OrderbookBuilder.get_full_depth
OrderbookBuilder.get_snapshot
OrderbookBuilder.get_orderbook_snapshot
OrderbookBuilder.snapshot_header
OrderbookBuilder.get_snapshot_row
```

These helper functions exist internally in Rust but are not directly exposed to Python:

```text
format_message
format_snapshot_row
parse_order_packet
parse_trade_packet
validate_binary_header
read_next_message_from_file
count_messages_in_file
```

---

# Small naming suggestion

Your current function name is:

```python
get_order_message()
```

You already added an alias for trades:

```python
get_all_trade_message()
```

For symmetry and easier user experience, you may also add this alias in `lib.rs`:

```rust
pub fn get_all_order_message(&self) -> Vec<String> {
    self.get_order_message()
}
```

Then users can write:

```python
orders = reader.get_all_order_message()
trades = reader.get_all_trade_message()
```

This is optional but cleaner for the public API.

---

# Best practice summary

| Need | Use |
|---|---|
| Read entire file once and inspect many times | `MessageCacheReader` |
| Process huge file with low memory | `StreamingBinaryLoader` |
| Build top-5 market depth | `OrderbookBuilder.get_snapshot()` |
| Build full order book depth | `OrderbookBuilder.get_full_depth()` |
| Export snapshot as CSV | `snapshot_header()` + `get_snapshot_row()` |
| Skip trades | `apply_filter(["N", "M", "X"])` |
| Process only first N streaming messages | `build_from_source(reader, limit=N)` |

---

# Minimal working script

```python
from fastreader import StreamingBinaryLoader, OrderbookBuilder

FILE_PATH = "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"
TOKEN = 1001

reader = StreamingBinaryLoader()
reader.open_stream(FILE_PATH, count_messages=False)

builder = OrderbookBuilder()
processed = builder.build_from_source(reader)

print("Processed messages:", processed)
print("Snapshot:", builder.get_snapshot(TOKEN, levels=5))
print("CSV header:", builder.snapshot_header())
print("CSV row:", builder.get_snapshot_row(TOKEN, levels=5))
```

Expected output:

```python
Processed messages: 1254300
Snapshot: {
    'token': 1001,
    'found': True,
    'mid_price': 245750,
    'best_bid': (245700, 150),
    'best_ask': (245800, 200),
    'spread': 100,
    'bids': [(245700, 150), (245650, 300), (245600, 75)],
    'asks': [(245800, 200), (245850, 100), (245900, 250)]
}
CSV header: local_ts,exch_ts,mid_price,bid_price_0,bid_qty_0,ask_price_0,ask_qty_0,...
CSV row: 0,0,245750,245700,150,245800,200,245650,300,245850,100,...
```

