Metadata-Version: 2.4
Name: msgpack-chunk
Version: 0.3.0
Summary: MessagePack serializer for chunked data
Author-email: Valentin Valls <valentin.valls@esrf.fr>
License-Expression: Apache-2.0
Project-URL: Homepage, https://gitlab.com/vallsv/msgpack-chunk
Project-URL: Tracker, https://gitlab.com/vallsv/msgpack-chunk/issues
Keywords: msgpack,messagepack,serializer,serialization,binary
Classifier: Development Status :: 5 - Production/Stable
Classifier: Operating System :: OS Independent
Classifier: Topic :: File Formats
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: black==26.*; extra == "dev"
Requires-Dist: psutil; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: flake8-pyproject; extra == "dev"
Requires-Dist: msgpack; extra == "dev"
Requires-Dist: numpy; extra == "dev"
Requires-Dist: matplotlib; extra == "dev"

Chunked implementation of [MessagePack](https://msgpack.org/) for python.

# Description

The library is designed to efficiently pack and unpack large amounts of
binary data for data transfer.

This is achieved with an iterative API, which allow to reuse the
memory (reducing the memory footprint), and features zero-copy operations
(speeding up the transformation).

# Benchmark

The main use case is to transfer big binary data together with some
metadata. Here is benchmark with few different setups, including the
officiel `msgpack` library as reference.

Here is a log/log diagrams which show GB/s related to the amount of
data to process.

## Pack

![Byterate related to the size of the packed binary](./doc/bench-pack.png)

## Unpack

![Byterate related to the binary size to unpack](./doc/bench-unpack.png)

# Example

## Pack

```python
import msgpack_chunk
packer = msgpack_chunk.Packer(
    expose_object_buffer="full",
)

io = ...

packer.pack(python_object)
main_buffer = bytearray(64 * 1024)
while result_buffer := packer.read_into(main_buffer):
    io.write(result_buffer)
```

## Unpack

```python
import msgpack_chunk
unpacker = msgpack_chunk.Unpacker()

io = ...

main_buffer = bytearray(64 * 1024)
while True:
    nb = io.readinto(main_buffer)
    if nb == 0:
        break
    unpacker.feed(main_buffer[:nb])

python_objects = list(unpacker)
```

## Advanced unpack

Allow to write directly inside the final python object,
when dealing with big binary or ext type data.

```python
import msgpack_chunk
unpacker = msgpack_chunk.Unpacker(
    bin="bytearray",
    expose_direct_buffer=True,
)

io = ...

main_buffer = bytearray(64 * 1024)
while True:
    direct_buffer = unpacker.get_buffer()
    if direct_buffer is not None and len(direct_buffer) > 16 * 1024:
        nb = io.readinto(direct_buffer)
        if not data:
            break
        unpacker.buffer_updated(nb)
    else:
        nb = io.readinto(main_buffer)
        if nb == 0:
            break
        unpacker.feed(main_buffer[:nb])

python_objects = list(unpacker)
```

# Resources

- [msgpack](https://pypi.org/project/msgpack/), the official python library
  library provides an efficiant Cython implementation.
- The [msgpack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
