Metadata-Version: 2.4
Name: zeropickle
Version: 0.1.0
Summary: Move numpy arrays and metadata between processes blazingly fast!
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: numpy>=2.4.2

# ZeroPickleQueue 🚀

**ZeroPickleQueue** is a high-performance, near drop-in replacement for Python's `multiprocessing.Queue`, optimized for **NumPy arrays**.

By leveraging Python's `shared_memory` module, it eliminates the massive overhead of pickling and unpickling large datasets between processes.

---

## 🏎️ Why ZeroPickleQueue?

In standard Python multiprocessing, sending a large NumPy array through a `Queue` involves:

1. **Serialization:** Pickling the array in the producer process.
2. **IPC Transfer:** Moving the serialized bytes through a pipe.
3. **Deserialization:** Unpickling the array in the consumer process.

For high-resolution video frames (e.g., 4K images), this process can take **50ms–200ms** per frame, creating a massive bottleneck. `ZeroPickleQueue` writes the data to a shared memory block and only passes compact metadata through the queue, reducing transfer latency substantially.

---

## ✨ Features

- **Near Drop-in Compatibility:** API follows `multiprocessing.Queue` (`put`, `get`, `empty`, `full`) with NumPy-first constraints.
- **Metadata Support:** Pass extra info (timestamps, labels) along with your arrays.
- **Automatic Resource Management:** Includes `atexit` handlers to clean up shared memory segments and prevent memory leaks.
- **Thread-Safe Slot Recycling:** Uses a background acknowledgment thread to manage memory slot reuse automatically.

---

## 📦 Installation

```bash
pip install zeropickle
```

---

## 🛠 Usage

### Basic Example

The interface is designed to be familiar to anyone who has used standard Python multiprocessing.

```python
import numpy as np
from multiprocessing import Process
from zeropickle import ZeroPickleQueue

def producer(queue):
    # Create a large dummy frame
    frame = np.random.randint(0, 255, (2160, 3840, 3), dtype=np.uint8)
    queue.put(frame)
    queue.put(None)  # Sentinel to signal completion

def consumer(queue):
    while True:
        item = queue.get()
        if item is None:
            break
        print(f"Received array with shape: {item.shape}")

if __name__ == "__main__":
    # Define the shape and dtype beforehand
    q = ZeroPickleQueue(
        data_shape=(2160, 3840, 3),
        data_dtype=np.uint8,
        max_size=10
    )

    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))

    p1.start()
    p2.start()
    p1.join()
    p2.join()

```

### Passing Metadata

You can pass a tuple where the **first element** is the NumPy array, and subsequent elements are your metadata.
`None` is also supported as a sentinel value for graceful consumer shutdown.
Other non-array payloads are not supported.

### Compatibility Notes

`ZeroPickleQueue` is a near drop-in replacement, not a full drop-in replacement.
It keeps the familiar queue methods, but requires fixed `data_shape`/`data_dtype` at initialization and only accepts NumPy-array payloads (plus `None` sentinel).

```python
# Producer
queue.put((image_array, {"frame_id": 42, "detected_objects": 3}))

# Consumer
image, meta = queue.get()
print(meta["frame_id"])

```

---

## ⚙️ How It Works

1. **Pre-allocation:** On initialization, a shared memory segment is created to hold `max_size` arrays of the specified shape.
2. **Slot Tracking:** An internal queue manages "slots" (indices) in that memory block.
3. **The Put Operation:** The producer claims a slot, copies the NumPy data into it using `np.copyto`, and sends the slot index via a standard queue.
4. **The Get Operation:** The consumer reads the slot index, accesses the shared memory, and immediately notifies an "acknowledgment queue."
5. **Recycling:** A background thread monitors acknowledgments and returns used slots back to the producer for reuse.

---

## ⚠️ Limitations & Tips

- **Fixed Dimensions:** Because memory is pre-allocated, every array must match the `data_shape` and `data_dtype` provided at initialization.
- **Copy on Get:** Currently, `.get()` returns a copy of the data. This is safer as it prevents the producer from overwriting memory that the consumer is still reading.
- **Cleanup:** Always call `.close()` or rely on the `atexit` handler to ensure `/dev/shm` (on Linux) doesn't fill up with orphaned memory segments.

---

## 🤝 Contributing

Contributions are welcome! If you have ideas for non-copying reads or support for variable-length data, feel free to open an issue or a PR.
