Metadata-Version: 2.1
Name: megstore
Version: 0.0.2
Summary: Common Formats. Uncommon Speed.
Author-Email: megvii-reng <megvii-reng@googlegroups.com>
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Project-URL: Homepage, https://github.com/megvii-research/megstore
Requires-Python: >=3.10
Requires-Dist: megfile
Requires-Dist: orjson
Provides-Extra: msgpack
Requires-Dist: msgpack>=0.6.0; extra == "msgpack"
Requires-Dist: msgpack-numpy>=0.4.6; extra == "msgpack"
Provides-Extra: doc
Requires-Dist: Sphinx>=7.3.7; python_version >= "3.10" and extra == "doc"
Requires-Dist: Sphinx; python_version < "3.10" and extra == "doc"
Requires-Dist: m2r2; extra == "doc"
Requires-Dist: sphinx-rtd-theme~=2.0; extra == "doc"
Requires-Dist: sphinxcontrib-jquery; extra == "doc"
Requires-Dist: sphinx-click; extra == "doc"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pytest-mock>3.6.0; extra == "test"
Requires-Dist: pytest-socket; extra == "test"
Requires-Dist: pytest-retry; extra == "test"
Requires-Dist: moto[s3]>=5.0.0; extra == "test"
Requires-Dist: cryptography>=3.3.2; extra == "test"
Requires-Dist: mutmut; extra == "test"
Requires-Dist: pyfakefs>=4.5; extra == "test"
Requires-Dist: mock; extra == "test"
Requires-Dist: requests-mock; extra == "test"
Provides-Extra: lint
Requires-Dist: ruff; extra == "lint"
Provides-Extra: typecheck
Requires-Dist: msgspec>=0.19.0; python_version >= "3.13" and extra == "typecheck"
Requires-Dist: pytype; extra == "typecheck"
Requires-Dist: pyre-check~=0.9.22; extra == "typecheck"
Requires-Dist: bandit; extra == "typecheck"
Requires-Dist: bandit-sarif-formatter; extra == "typecheck"
Provides-Extra: dev
Requires-Dist: megstore[doc,lint,msgpack,test,typecheck]; extra == "dev"
Description-Content-Type: text/markdown

megstore - Common Formats. Uncommon Speed.
---

[![Build](https://github.com/megvii-research/megstore/actions/workflows/run-tests.yml/badge.svg?branch=main)](https://github.com/megvii-research/megstore/actions/workflows/run-tests.yml)
[![Documents](https://github.com/megvii-research/megstore/actions/workflows/publish-docs.yml/badge.svg)](https://github.com/megvii-research/megstore/actions/workflows/publish-docs.yml)
[![Codecov](https://img.shields.io/codecov/c/gh/megvii-research/megstore)](https://app.codecov.io/gh/megvii-research/megstore/)
[![Latest version](https://img.shields.io/pypi/v/megstore.svg)](https://pypi.org/project/megstore/)
[![Support python versions](https://img.shields.io/pypi/pyversions/megstore.svg)](https://pypi.org/project/megstore/)
[![License](https://img.shields.io/pypi/l/megstore.svg)](https://github.com/megvii-research/megstore/blob/main/LICENSE)

* Docs: http://megvii-research.github.io/megstore

`megstore` is a Python library that provides a unified interface for file operations across various file formats. It aims to simplify file handling, while also enhancing performance and reliability.

## Why megstore

* Faster file read and write operations, the random read performance is also very fast with index support.
* Low memory usage with streaming read and write support.
* Supports popular file formats and easy to use.

## Quick Start

### Installation

```bash
pip3 install megstore

# for msgpack support
pip3 install 'megstore[msgpack]'
```

### Examples

- indexed jsonline

```python
from megstore import indexed_jsonline_open

with indexed_jsonline_open("data.jsonl", "w") as writer:
    writer.append({"key": "value"})
    writer.append({"number": 123})

# Buffer up to 1,000 records before writing a batch.
with indexed_jsonline_open("buffered.jsonl", "w", buffer_size=1000) as writer:
    writer.extend([{"key": "value"}, {"number": 123}])

with indexed_jsonline_open("data.jsonl", "r") as reader:
    second_item = reader[1]
    second_to_last_items = reader[1:]
    total_count = len(reader)
    for item in reader:
        print(item)
```

- indexed msgpack

```python
from megstore import indexed_msgpack_open

with indexed_msgpack_open("data.msg", "w") as writer:
    writer.append({"key": "value"})
    writer.append({"number": 123})

with indexed_msgpack_open("data.msg", "r") as reader:
    second_item = reader[1]
    second_to_last_items = reader[1:]
    total_count = len(reader)
    for item in reader:
        print(item)
```

- indexed txt

```python
from megstore import indexed_txt_open

with indexed_txt_open("data.txt", "w") as writer:
    writer.append("Hello, World!")
    writer.append("This is a test.")

with indexed_txt_open("data.txt", "r") as reader:
    second_item = reader[1]
    second_to_last_items = reader[1:]
    total_count = len(reader)
    for line in reader:
        print(line)
```

## How to Contribute
* We welcome everyone to contribute code to the `megstore` project, but the contributed code needs to meet the following conditions as much as possible:

    *You can submit code even if the code doesn't meet conditions. The project members will evaluate and assist you in making code changes*

    * **Code format**: Your code needs to pass **code format check**. `megstore` uses `ruff` as lint tool
    * **Static check**: Your code needs complete **type hint**. `megstore` uses `pytype` as static check tool. If `pytype` failed in static check, use `# pytype: disable=XXX` to disable the error and please tell us why you disable it.

    * **Test**: Your code needs complete **unit test** coverage. `megstore` uses `pyfakefs` and `moto` as local file system and s3 virtual environment in unit tests. The newly added code should have a complete unit test to ensure the correctness

* You can help to improve `megstore` in many ways:
    * Write code.
    * Improve [documentation](https://github.com/megvii-research/megstore/blob/main/docs).
    * Report or investigate [bugs and issues](https://github.com/megvii-research/megstore/issues).
    * If you find any problem or have any improving suggestion, [submit a new issuse](https://github.com/megvii-research/megstore/issues) as well. We will reply as soon as possible and evaluate whether to adopt.
    * Review [pull requests](https://github.com/megvii-research/megstore/pulls).
    * Star `megstore` repo.
    * Recommend `megstore` to your friends.
    * Any other form of contribution is welcomed.
