Metadata-Version: 2.4
Name: OrderPulse
Version: 0.2.12
Summary: High-performance exchange feed parser and orderflow analytics engine with Rust and Python bindings
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# High-Performance Market Data Processing Engine

## Overview

This project is a **high-performance Rust-based market data processing engine** with Python bindings powered by **PyO3**. The system is designed to:

* Read binary market feed data
* Parse order and trade messages
* Filter messages by instrument token
* Stream messages sequentially
* Build and maintain an in-memory order book
* Expose Rust performance to Python users

The architecture follows a clean separation of concerns:

| Layer               | Responsibility                      |
| ------------------- | ----------------------------------- |
| Binary Reader       | Reads raw exchange feed data        |
| Message Parser      | Converts bytes into typed messages  |
| Message Abstraction | Unified enum-based message handling |
| Order Book Engine   | Maintains bid/ask state             |
| Python Interface    | Exposes Rust APIs to Python         |

---

# Architecture

## High-Level Architecture

```text
                 +----------------------+
                 |  Binary Feed File    |
                 +----------+-----------+
                            |
                            v
                 +----------------------+
                 | read_messages()      |
                 | Binary Parser        |
                 +----------+-----------+
                            |
                            v
                 +----------------------+
                 | Vec<Message>         |
                 | Message Enum Layer   |
                 +----------+-----------+
                            |
          +----------------+----------------+
          |                                 |
          v                                 v
+----------------------+      +---------------------------+
| ReadMsgFromBinary    |      | OrderBookBuilder          |
| Query/Filtering API  |      | Order Book Construction   |
+----------+-----------+      +-------------+-------------+
           |                                  |
           v                                  v
+----------------------+      +---------------------------+
| MessageBatch         |      | OrderBookManager          |
| Batch Selection API  |      | Matching Engine State     |
+----------------------+      +---------------------------+
```

---

# Core Modules

## 1. `orderbook`

Contains:

* `OrderBookManager`
* Bid/ask maintenance logic
* Top-of-book calculations
* Market depth generation

### Responsibilities

* Process incoming order messages
* Update order book state
* Maintain price levels
* Generate top bid/ask snapshots

---

## 2. `read_trd_ord_only`

Contains:

* `read_messages()`

### Responsibilities

* Read binary exchange feed files
* Decode binary structures
* Produce typed messages

---

## 3. `structure`

Contains:

* `Message` enum
* Packet structures
* Exchange packet layouts

### Responsibilities

Defines the low-level message schema.

Example:

```rust
pub enum Message {
    Order(OrderPacket),
    Trade(TradePacket),
}
```

This abstraction allows the engine to process heterogeneous messages uniformly.

---

## 4. `orderbook_processing`

Contains advanced order book processing utilities.

Likely responsibilities:

* Aggregation
* Snapshot generation
* Derived analytics
* Depth calculations

---

## 5. `tsc`

Most likely contains:

* Timestamp counter utilities
* Performance timers
* Latency measurement helpers

Useful for:

* HFT systems
* Benchmarking
* Exchange latency analysis

---

# Main Components

# 1. `format_message()`

```rust
fn format_message(msg: &Message) -> String
```

## Purpose

Converts internal message structures into human-readable strings.

This function acts as:

* A debugging utility
* A logging formatter
* A Python-readable serialization layer

---

## Flow

```text
Message Enum
     |
     +--> Order Message
     |
     +--> Trade Message
```

---

## Order Message Formatting

Extracts:

* Sequence number
* Message length
* Message type
* Exchange timestamp
* Local timestamp
* Order ID
* Token
* Price
* Quantity
* Missed packet flag

### Example

```rust
let output = format_message(&msg);
println!("{}", output);
```

### Sample Output

```text
Order Message: SeqNo12345, msg_len64, Msg_Type'B', Exch_ts123456789,
local_ts123456790, order_id99999, Token26000,
order_Type'B', Price24500, Quantity50, missed0
```

---

## Trade Message Formatting

Extracts:

* Buy order ID
* Sell order ID
* Trade price
* Trade quantity

### Example

```rust
let output = format_message(&trade_msg);
```

---

# 2. `MessageBatch`

```rust
#[pyclass]
pub struct MessageBatch {
    messages: Vec<Message>,
}
```

## Purpose

Acts as a lightweight batch container for filtered or selected messages.

This is a powerful abstraction because it:

* Avoids copying entire datasets repeatedly
* Enables chainable workflows
* Supports Python interoperability
* Allows selective order book construction

---

# Methods

## `len()`

```rust
fn len(&self) -> usize
```

Returns total messages in batch.

### Example

```python
batch.len()
```

### Output

```python
1200
```

---

## `is_empty()`

```rust
fn is_empty(&self) -> bool
```

Checks whether the batch contains messages.

### Example

```python
batch.is_empty()
```

### Output

```python
False
```

---

## `to_list(limit=None)`

```rust
fn to_list(&self, limit: Option<usize>) -> Vec<String>
```

Converts messages into formatted strings.

### Example

```python
batch.to_list(5)
```

### Output

```python
[
  "Order Message: ...",
  "Trade Message: ..."
]
```

---

# 3. `ReadMsgFromBinary`

```rust
#[pyclass]
pub struct ReadMsgFromBinary {
    messages: Vec<Message>,
    current_index: usize,
}
```

## Purpose

Primary interface for:

* Loading exchange binary feeds
* Filtering by token
* Iterating messages
* Performing analytics
* Feeding order book engine

This is effectively the:

> Core ingestion engine

---

# Constructor

## `new(path, token=None)`

```rust
fn new(path: String, token: Option<u32>) -> PyResult<Self>
```

## Responsibilities

1. Read binary file
2. Parse messages
3. Optionally filter by token
4. Store parsed messages

---

## Example

### Python Usage

```python
reader = ReadMsgFromBinary(
    path="market_feed.bin",
    token=26000
)
```

---

## Internal Flow

```text
Binary File
    |
    v
read_messages()
    |
    v
Vec<Message>
    |
    +--> Optional Token Filter
    |
    v
ReadMsgFromBinary
```

---

# Analytics Functions

## `total_messages()`

```rust
fn total_messages(&self) -> usize
```

Returns total parsed messages.

### Example

```python
reader.total_messages()
```

---

## `total_orders()`

```rust
fn total_orders(&self) -> usize
```

Counts only order messages.

### Example

```python
reader.total_orders()
```

---

## `total_trades()`

```rust
fn total_trades(&self) -> usize
```

Counts only trade messages.

### Example

```python
reader.total_trades()
```

---

## `summary()`

```rust
fn summary(&self)
```

Prints complete dataset statistics.

### Example

```python
reader.summary()
```

### Output

```text
Total Messages: 500000
Total Orders: 420000
Total Trades: 80000
```

---

# Cursor-Based Streaming

The engine supports sequential message streaming.

This design is critical for:

* Large datasets
* Replay systems
* Event-driven processing
* Memory-efficient workflows

---

## `reset_cursor()`

```rust
fn reset_cursor(&mut self)
```

Resets stream position.

### Example

```python
reader.reset_cursor()
```

---

## `get_next_msg()`

```rust
fn get_next_msg(&mut self) -> String
```

Returns next message in sequence.

### Example

```python
msg = reader.get_next_msg()
```

### Output

```python
"Order Message: ..."
```

---

# Message Selection APIs

## `get_all_messages(limit=None)`

Returns all messages.

### Example

```python
reader.get_all_messages(10)
```

---

## `get_order_messages(limit=None)`

Returns only order messages.

### Example

```python
reader.get_order_messages(20)
```

---

## `get_trade_messages(limit=None)`

Returns only trade messages.

### Example

```python
reader.get_trade_messages(20)
```

---

# Batch Selection APIs

These methods return `MessageBatch` objects.

---

## `select_all_messages()`

```rust
fn select_all_messages(&self) -> MessageBatch
```

### Example

```python
batch = reader.select_all_messages()
```

---

## `select_order_messages()`

Returns only order messages.

### Example

```python
orders = reader.select_order_messages()
```

---

## `select_trade_messages()`

Returns only trade messages.

### Example

```python
trades = reader.select_trade_messages()
```

---

## `select_next_messages(limit)`

```rust
fn select_next_messages(&mut self, limit: usize) -> MessageBatch
```

Returns the next N messages from stream.

### Example

```python
chunk = reader.select_next_messages(1000)
```

This is ideal for:

* Chunk processing
* Streaming pipelines
* Backtesting engines

---

# 4. `OrderBookBuilder`

```rust
#[pyclass]
pub struct OrderBookBuilder;
```

## Purpose

Constructs order books from:

* Entire datasets
* Selected batches
* Streaming chunks

This is the bridge between:

```text
Message Layer ---> Matching Engine
```

---

# Constructor

## `new()`

```rust
fn new() -> Self
```

### Example

```python
ob = OrderBookBuilder()
```

---

# Order Book Creation

## `create_orderbook_all_messages()`

```rust
fn create_orderbook_all_messages(&self, reader: PyRef<ReadMsgFromBinary>) -> usize
```

Processes all messages.

### Example

```python
builder.create_orderbook_all_messages(reader)
```

---

## `create_orderbook()`

```rust
fn create_orderbook(&self, batch: PyRef<MessageBatch>) -> usize
```

Processes only selected messages.

### Example

```python
batch = reader.select_order_messages()
builder.create_orderbook(batch)
```

---

# Internal Processing Pipeline

```text
Message
   |
   v
OrderBookManager::process_order_message()
   |
   v
Update Bid/Ask Levels
   |
   v
Generate Top Levels
   |
   v
Emit Snapshot
```

---

# `build_from_messages()`

```rust
fn build_from_messages(messages: &[Message]) -> usize
```

## Responsibilities

* Initialize order book manager
* Process every message
* Update order book state
* Generate top levels
* Track processed rows

---

# Performance Characteristics

## Why Rust?

This system is architected for:

* Low latency
* Deterministic memory management
* High throughput
* Zero-cost abstractions
* Safe concurrency

---

## Why PyO3?

PyO3 enables:

* Python interoperability
* Quant workflow integration
* Data science compatibility
* Jupyter support

This gives:

```text
Rust Speed + Python Flexibility
```

---

# Design Patterns Used

## 1. Enum-Based Message Dispatch

```rust
match msg {
    Message::Order(..) => {}
    Message::Trade(..) => {}
}
```

Benefits:

* Type safety
* Fast dispatch
* Cleaner branching

---

## 2. Batch Processing Pattern

`MessageBatch` avoids unnecessary re-parsing.

Benefits:

* Reduced memory allocations
* Faster analytics
* Reusable datasets

---

## 3. Streaming Iterator Pattern

`current_index` acts as an internal cursor.

Benefits:

* Efficient sequential access
* Large file support
* Replay compatibility

---

## 4. Separation of Concerns

Each module has a single responsibility.

Benefits:

* Easier testing
* Better maintainability
* Cleaner scaling

---

# Example End-to-End Workflow

## Python Example

```python
from your_module import ReadMsgFromBinary, OrderBookBuilder

# Load binary feed
reader = ReadMsgFromBinary(
    path="market_feed.bin",
    token=26000
)

# Dataset statistics
reader.summary()

# Select only orders
orders = reader.select_order_messages()

# Build order book
builder = OrderBookBuilder()

builder.create_orderbook(orders)
```


