Metadata-Version: 2.4
Name: mq2db
Version: 1.0.0
Summary: Collect MQ messages and save them into database.
Author-email: Taishi Hashimoto <hashimoto.taishi@nipr.ac.jp>
License-Expression: MIT
Project-URL: Homepage, https://github.com/taishi-hashimoto/mq2db
Project-URL: Repository, https://github.com/taishi-hashimoto/mq2db
Project-URL: Issues, https://github.com/taishi-hashimoto/mq2db/issues
Keywords: database,message-queue,sqlalchemy,zeromq
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: System :: Logging
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sqlalchemy
Requires-Dist: pyzmq
Requires-Dist: PyYAML
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# mq2db

Dump data from a message queue into a database.

**Note:** mq2db currently uses ZeroMQ sockets as its message-queue backend.

mq2db connects to one or more configured MQ endpoints, parses each received
message with a loader, and writes the resulting rows into SQLAlchemy-supported
databases.

## Installation

After the package is published to PyPI:

```sh
python -m pip install mq2db
```

For development from this repository:

```sh
python -m pip install -e '.[test]'
```

## Usage

```sh
mq2db path/to/config.yaml
```

If the mq2db configuration is nested under a larger YAML file, pass the section:

```sh
mq2db path/to/config.yaml --section mq2db.specific.setting
```

## Configuration

Example configuration is here: [example.yaml](./example.yaml).
More examples can also be found in the [tests](./tests/) directory.

The top-level mq2db configuration contains a `targets` mapping. Each target
defines:

- ZeroMQ socket settings, such as `type`, `method`, `address`, `topic`, and
  receive method.
- A loader class, such as `mq2db.DictLoader`, `mq2db.JSONLoader`,
  `mq2db.TextLoader`, `mq2db.CSVLoader`, or a custom importable class.
- Database settings, including SQLAlchemy URL, table columns, keys, indices,
  automatic timestamp/raw columns, and write interval.

## Development

```sh
python -m pip install -e '.[dev]'
pytest
python -m build
twine check dist/*
```

## Release

Releases use semantic version tags prefixed with `v`, such as `v1.0.0`.
Pushing a matching tag publishes the package to PyPI through GitHub Actions:

```sh
git tag v1.0.0
git push origin v1.0.0
```

## License

mq2db is distributed under the MIT License. See [LICENSE](./LICENSE).
