Metadata-Version: 2.4
Name: arrakis-backend-kafka
Version: 0.14.0
Summary: Kafka backend for the Arrakis server
Project-URL: Homepage, https://git.ligo.org/ngdd/backends/arrakis-backend-kafka
Project-URL: Documentation, https://docs.ligo.org/ngdd/backends/arrakis-backend-kafka
Project-URL: Issue Tracker, https://git.ligo.org/ngdd/backends/arrakis-backend-kafka/issues
Project-URL: Source Code, https://git.ligo.org/ngdd/backends/arrakis-backend-kafka.git
Author-email: Olivia Godwin <olivia.godwin@ligo.org>
Maintainer-email: Olivia Godwin <olivia.godwin@ligo.org>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.11
Requires-Dist: arrakis-server>=0.16
Requires-Dist: arrakis>=0.20
Requires-Dist: confluent-kafka>=2.3
Requires-Dist: gpstime
Requires-Dist: numpy
Requires-Dist: platformdirs
Requires-Dist: pyarrow
Provides-Extra: dev
Requires-Dist: markdown-callouts>=0.2; extra == 'dev'
Requires-Dist: markdown-exec>=0.5; extra == 'dev'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'dev'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'dev'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'dev'
Requires-Dist: mkdocs-material-igwn; extra == 'dev'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'dev'
Requires-Dist: mkdocs>=1.3; extra == 'dev'
Requires-Dist: mkdocstrings[python]; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: mypy-extensions; extra == 'dev'
Requires-Dist: pip; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-arrakis[publish]>=0.6; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: toml>=0.10; extra == 'dev'
Provides-Extra: docs
Requires-Dist: markdown-callouts>=0.2; extra == 'docs'
Requires-Dist: markdown-exec>=0.5; extra == 'docs'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'docs'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'docs'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'docs'
Requires-Dist: mkdocs-material-igwn; extra == 'docs'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'docs'
Requires-Dist: mkdocs>=1.3; extra == 'docs'
Requires-Dist: mkdocstrings[python]; extra == 'docs'
Requires-Dist: toml>=0.10; extra == 'docs'
Provides-Extra: lint
Requires-Dist: mypy; extra == 'lint'
Requires-Dist: mypy-extensions; extra == 'lint'
Requires-Dist: pip; extra == 'lint'
Requires-Dist: ruff; extra == 'lint'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-arrakis[publish]>=0.6; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Description-Content-Type: text/markdown

<h1 align="center">arrakis-backend-kafka</h1>

<p align="center">Kafka backend for Arrakis server</p>

<p align="center">
  <a href="https://git.ligo.org/ngdd/backends/arrakis-backend-kafka/-/pipelines/latest">
    <img alt="ci" src="https://git.ligo.org/ngdd/backends/arrakis-backend-kafka/badges/main/pipeline.svg" />
  </a>
  <a href="https://git.ligo.org/ngdd/backends/arrakis-backend-kafka/-/pipelines/latest">
    <img alt="coverage" src="https://git.ligo.org/ngdd/backends/arrakis-backend-kafka/badges/main/coverage.svg" />
  </a>
  <a href="https://ngdd.docs.ligo.org/backends/arrakis-backend-kafka/">
    <img alt="documentation" src="https://img.shields.io/badge/docs-mkdocs%20material-blue.svg?style=flat" />
  </a>
  <a href="https://pypi.org/project/arrakis-backend-kafka/">
    <img alt="pypi version" src="https://img.shields.io/pypi/v/arrakis-backend-kafka.svg" />
  </a>
</p>

---

Kafka backend for the [Arrakis server](https://git.ligo.org/ngdd/arrakis-server),
part of the LIGO next-generation data delivery (NGDD) project.  Serves live,
low-latency detector data over Apache Arrow Flight from timeseries published to
Kafka topics, along with the recent past held in Kafka's retention window, and
manages the publishers that write to those topics.

## Resources

* [Documentation](https://ngdd.docs.ligo.org/backends/arrakis-backend-kafka/)
* [Source Code](https://git.ligo.org/ngdd/backends/arrakis-backend-kafka)
* [Issue Tracker](https://git.ligo.org/ngdd/backends/arrakis-backend-kafka/-/issues)
* [Arrakis server](https://git.ligo.org/ngdd/arrakis-server)
* [Arrakis Python client](https://git.ligo.org/ngdd/arrakis-python)

## Installation

With `pip`:

```
pip install arrakis-backend-kafka
```

With Docker:

```
docker run --net=host -it containers.ligo.org/ngdd/backends/arrakis-backend-kafka:main
```

## Features

* Serves live timeseries from Kafka topics over Apache Arrow Flight, with
  historical requests answered from Kafka's retention window
* Contiguous transition from historical to live data when a request spans
  the present
* Pooled Kafka consumers, so many clients subscribed to the same channels
  share a single set of readers and one polling thread
* Fast-forwards to the head of the stream when a subscription falls behind,
  logging the range that was skipped
* Publisher management: partition assignment, topic creation and channel
  registration for publishers defined in TOML configuration, reloaded on
  `SIGHUP`
* Channel metadata kept in compacted Kafka topics and synchronized from
  remote publishers, so replicas serve a consistent channel table
* Gap-aware responses that return masked data for missing samples
