Metadata-Version: 2.4
Name: beavers
Version: 0.14.0
Summary: Python stream processing
Project-URL: Bug Tracker, https://github.com/tradewelltech/beavers/issues
Project-URL: Changelog, https://github.com/tradewelltech/beavers/blob/main/CHANGELOG.md
Author-email: Tradewell Tech <engineering@tradewelltech.co>
Maintainer-email: 0x26res <0x26res@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: apache-arrow,data,streaming
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <4,>=3.11
Requires-Dist: confluent-kafka>=2.1.1
Requires-Dist: pandas
Requires-Dist: perspective-python>=3.0.0
Requires-Dist: polars
Requires-Dist: pyarrow
Requires-Dist: tornado
Provides-Extra: confluent-kafka
Requires-Dist: confluent-kafka; extra == 'confluent-kafka'
Provides-Extra: perspective-python
Requires-Dist: perspective-python; extra == 'perspective-python'
Requires-Dist: tornado; extra == 'perspective-python'
Provides-Extra: polars
Requires-Dist: polars; extra == 'polars'
Provides-Extra: pyarrow
Requires-Dist: pyarrow; extra == 'pyarrow'
Description-Content-Type: text/markdown


[![PyPI Version][pypi-image]][pypi-url]
[![Python Version][versions-image]][versions-url]
[![Github Stars][stars-image]][stars-url]
[![codecov][codecov-image]][codecov-url]
[![Build Status][build-image]][build-url]
[![Documentation][doc-image]][doc-url]
[![License][license-image]][license-url]
[![Downloads][downloads-image]][downloads-url]
[![Downloads][downloads-month-image]][downloads-month-url]
[![Ruff][ruff-image]][ruff-url]
[![prek][prek-image]][prek-url]
[![snyk][snyk-image]][snyk-url]
<a href="https://trackgit.com">
<img src="https://us-central1-trackgit-analytics.cloudfunctions.net/token/ping/m7c1fpbnueo78pkcetcm" alt="trackgit-views" />
</a>

![Beavers Logo][5]

# Beavers

[Documentation][6] / [Installation][7] / [Repository][1] / [PyPI][8]

[Beavers][1] is a python library for stream processing, optimized for analytics.

It is used at [Tradewell Technologies][2],
to calculate analytics and serve model predictions,
for both realtime and batch jobs.

## Key Features

- Works in **real time** (eg: reading from Kafka) and **replay mode** (eg: reading from Parquet files).
- Optimized for analytics, using micro-batches (instead of processing records one by one).
- Similar to [incremental][3], it updates nodes in a dag incrementally.
- Taking inspiration from [kafka streams][4], there are two types of nodes in the dag:
    - **Stream**: ephemeral micro-batches of events (cleared after every cycle).
    - **State**: durable state derived from streams.
- Clear separation between the business logic and the IO.
  So the same dag can be used in real time mode, replay mode or can be easily tested.
- Functional interface: no inheritance or decorator required.
- Support for complicated joins, not just "linear" data flow.

## Limitations

- No concurrency support.
  To speed up calculation use libraries like pandas, pyarrow or polars.
- No async code.
  To speed up IO use kafka driver native thread or parquet IO thread pool.
- No support for persistent state.
  Instead of saving state, replay historic data from kafka to prime stateful nodes.

## Talks

- [Unified batch and stream processing in python | PyData Global 2023][9]

[1]: https://github.com/tradewelltech/beavers
[2]: https://www.tradewelltech.co/
[3]: https://github.com/janestreet/incremental
[4]: https://www.confluent.io/blog/kafka-streams-tables-part-1-event-streaming/
[5]: https://raw.githubusercontent.com/tradewelltech/beavers/master/docs/static/icons/beavers/logo.svg
[6]: https://beavers.readthedocs.io/en/latest/
[7]: https://beavers.readthedocs.io/en/latest/install/
[8]: https://pypi.org/project/beavers/
[9]: https://www.youtube.com/watch?v=8pUwsGA8SQM

[pypi-image]: https://img.shields.io/pypi/v/beavers
[pypi-url]: https://pypi.org/project/beavers/
[build-image]: https://github.com/tradewelltech/beavers/actions/workflows/ci.yaml/badge.svg
[build-url]: https://github.com/tradewelltech/beavers/actions/workflows/ci.yaml
[stars-image]: https://img.shields.io/github/stars/tradewelltech/beavers
[stars-url]: https://github.com/tradewelltech/beavers
[versions-image]: https://img.shields.io/pypi/pyversions/beavers
[versions-url]: https://pypi.org/project/beavers/
[doc-image]: https://readthedocs.org/projects/beavers/badge/?version=latest
[doc-url]: https://beavers.readthedocs.io/en/latest/?badge=latest
[license-image]: http://img.shields.io/:license-Apache%202-blue.svg
[license-url]: https://github.com/tradewelltech/beavers/blob/main/LICENSE
[codecov-image]: https://codecov.io/gh/tradewelltech/beavers/branch/main/graph/badge.svg?token=GY6KL7NT1Q
[codecov-url]: https://codecov.io/gh/tradewelltech/beavers
[downloads-image]: https://pepy.tech/badge/beavers
[downloads-url]: https://static.pepy.tech/badge/beavers
[downloads-month-image]: https://pepy.tech/badge/beavers/month
[downloads-month-url]: https://static.pepy.tech/badge/beavers/month
[ruff-image]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
[ruff-url]: https://github.com/astral-sh/ruff
[prek-image]: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit
[prek-url]: https://github.com/pre-commit/pre-commit
[snyk-image]: https://snyk.io/test/github/tradewelltech/beavers/badge.svg
[snyk-url]: https://snyk.io/advisor/python/beavers
