Metadata-Version: 2.4
Name: mx-multiplexer
Version: 3.0.0
Summary: Multiplexer, a stateless TCP message broker between services: the broker and its Python libraries
Home-page: https://github.com/kulewski/multiplexer
Author: Krzysztof Kulewski
License: MIT
Project-URL: Documentation, https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/README.md
Project-URL: Source, https://github.com/kulewski/multiplexer
Project-URL: Releases, https://github.com/kulewski/multiplexer/releases
Project-URL: Issues, https://github.com/kulewski/multiplexer/issues
Keywords: message broker,request reply,publish subscribe,rpc,microservices,asio,protobuf
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Networking
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES
Requires-Dist: protobuf>=3.20
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<p align="center"><img src="https://raw.githubusercontent.com/kulewski/multiplexer/v3.0.0/docs/assets/logo.svg" alt="Multiplexer" width="420"></p>

# Multiplexer

**A lightweight TCP message broker for service-to-service messaging.**
Not a socket multiplexer, not a WebSocket server: it is the broker your
services talk through, on a trusted network, for request/reply against a
pool of backends and publish/subscribe of events.

Peers connect over TCP, announce their peer type, and exchange typed
messages with low latency. A declarative rules file defines the routing for
each message type: round-robin load balancing across a pool of backends,
fan-out to every subscriber, or direct addressing by instance id. It is
stateless by design: no queues, no persistence, no authentication, and no
coordination between instances. It terminates no HTTP or WebSocket
connections and manages no client sessions; a web server is one kind of
peer among others.

`pip install mx-multiplexer` installs the Python package and the
multiplexer itself, `mxcontrol`; a release also ships `mxcontrol` as one
binary for any Linux with glibc 2.28 or newer and as a container image, and the C++ library as a Debian or
Ubuntu package, all for Linux x86_64, and the Rust library is the crate
`mx-multiplexer`. Nothing needs building; the C++17 source builds with
Bazel or `make` when you want to. In Docker a
multiplexer is one `docker run`, and on Kubernetes a StatefulSet behind a
headless Service, with no operator, no leader and no state to keep
([in Docker and on Kubernetes](#in-docker-and-on-kubernetes)).
Deploy several instances active-active: there is no single point of
failure, every client and backend connects to all of them, and failover
between them is automatic and transparent.

Multiplexer has been used in production in a number of projects since 2008.
This repository adds what publication calls for:
complete documentation, a test for every failure mode, and static
thread-safety analysis.

## Quick start

On Linux x86_64 with Python 3.10 or newer: the package from PyPI, which
brings the multiplexer with it, a rules file, two multiplexers, a backend
and a client. Nothing to build.

```
python3 -m venv mx && . mx/bin/activate
pip install mx-multiplexer
mxcontrol generate_rules multiplexer.rules          # the system rules every rules file starts from
cat >> multiplexer.rules <<'END'
peer { type: 200 name: "WEB" }
peer { type: 201 name: "COMPUTE" }
type { type: 301 name: "COMPUTE_REQUEST" to { peer: "COMPUTE" whom: ANY } }
type { type: 302 name: "COMPUTE_RESPONSE" }
END
mxcontrol generate_constants multiplexer.rules --python multiplexer_constants.py
mxcontrol run_multiplexer --address 127.0.0.1:1980 2> mx1.log &
mxcontrol run_multiplexer --address 127.0.0.1:1981 2> mx2.log &
```

`backend.py` answers each request with its payload upper-cased:

```python
from multiplexer.servers import BaseMultiplexerServer
from multiplexer_constants import peers, types


class Compute(BaseMultiplexerServer):
    def handle_message(self, mxmsg):
        self.send_message(message=mxmsg.message.upper(), type=types.COMPUTE_RESPONSE)


Compute([("127.0.0.1", 1980), ("127.0.0.1", 1981)], type=peers.COMPUTE).serve_forever()
```

`client.py` asks:

```python
from multiplexer.threaded_client import ThreadedClient
from multiplexer_constants import peers, types

client = ThreadedClient([("127.0.0.1", 1980), ("127.0.0.1", 1981)], type=peers.WEB)
print(client.query(b"hello", type=types.COMPUTE_REQUEST, timeout=10).message)
client.shutdown()
```

Start the backend, then ask, take a multiplexer away and ask again:

```
$ python backend.py 2> backend.log &
$ python client.py 2> client.log
b'HELLO'
$ kill %1                       # the multiplexer on 1980
$ python client.py 2> client.log
b'HELLO'
```

Both peers are connected to both multiplexers, so the second query goes
through the one left, and nothing waited or failed. `kill %2 %3` ends the
rest. The libraries log their connections to stderr, which the commands
above send to files.

## In Docker and on Kubernetes

A multiplexer is one process, one rules file and nothing to keep, so a
container is all it takes. The image, `ghcr.io/kulewski/multiplexer`, is
32 MB, an 8 MB download: `mxcontrol` and the system rules on distroless's
glibc base, running as `nonroot`, listening on 1980 and reading
`/etc/mx/multiplexer.rules`.
Mount the quick start's directory there, and these two lines replace the
quick start's two `mxcontrol run_multiplexer` lines:

```
docker run -d --name mx1 -p 1980:1980 -u "$(id -u)" -v "$PWD:/etc/mx:ro" ghcr.io/kulewski/multiplexer:latest
docker run -d --name mx2 -p 1981:1980 -u "$(id -u)" -v "$PWD:/etc/mx:ro" ghcr.io/kulewski/multiplexer:latest
```

The backend and the client stay as they are, and `docker stop mx1` is the
quick start's `kill %1`; `-u` runs the containers as you, so that they can
read your directory. An edit of `multiplexer.rules` is put in use by both
within seconds, with no restart; the directory is mounted rather than the
file so that an editor's save reaches them
([changing the rules](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/operations.md#changing-the-rules)).

On Kubernetes there is nothing to operate: no operator, no leader, no
shared state. A StatefulSet behind a headless Service gives each
multiplexer a name of its own, `mx-0.mx`, `mx-1.mx`, `mx-2.mx`, every peer
is given all of them, and the rules file is a ConfigMap:

```
kubectl create configmap mx-rules --from-file=multiplexer.rules
kubectl apply -f mx.yaml
```

```yaml
# mx.yaml
apiVersion: v1
kind: Service
metadata: {name: mx}
spec:
  clusterIP: None              # headless: a DNS name per pod, no load balancing
  selector: {app: mx}
  ports: [{port: 1980}]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: mx}
spec:
  serviceName: mx
  replicas: 3
  selector: {matchLabels: {app: mx}}
  template:
    metadata: {labels: {app: mx}}
    spec:
      containers:
        - name: mx
          image: ghcr.io/kulewski/multiplexer:latest   # or a version, to pin it
          ports: [{containerPort: 1980}]
          readinessProbe: {tcpSocket: {port: 1980}, periodSeconds: 5}
          volumeMounts: [{name: rules, mountPath: /etc/mx}]
      volumes: [{name: rules, configMap: {name: mx-rules}}]
```

The peers get `mx-0.mx:1980,mx-1.mx:1980,mx-2.mx:1980`. A rolling update
restarts the multiplexers one at a time, which costs a request nothing
while the peers are on the others, and an edit of the ConfigMap needs no
rollout at all. [docs/operations.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/operations.md#on-kubernetes) has
the complete manifest, with the pods spread over nodes and a
PodDisruptionBudget, and the inference example's `compose.yaml` runs a
whole stack with `docker compose up`: two multiplexers, the workers and
the web app ([examples/inference](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/inference)).

## Internal service-to-service messaging

An example: web servers on the left, two multiplexers in the middle, a
pool of C++ backends doing the expensive computation on the right. The web
servers are peers like any other, here the ones that ask; their own HTTP
or WebSocket traffic with browsers is theirs and never reaches the
multiplexer. Every peer is connected to both multiplexers; a request from
any web server reaches one backend, chosen round robin, and the answer
comes back to the server that asked. Lose either multiplexer, or any
backend, and nothing stops.

![Web servers on the left, two multiplexers in the middle, compute backends on the right, every peer connected to both multiplexers](https://raw.githubusercontent.com/kulewski/multiplexer/v3.0.0/docs/assets/deployment.svg)

The rules behind the picture, in the format [rules.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/rules.md)
describes, are the quick start's: the system rules every rules file starts
from, left out here, then the four entries it appended, written out:

```
# ... the system rules, as `mxcontrol generate_rules` writes them ...

peer {
    type: 200
    name: "WEB"
}

peer {
    type: 201
    name: "COMPUTE"
}

type {
    type: 301
    name: "COMPUTE_REQUEST"
    to {
        peer: "COMPUTE"
        whom: ANY
    }
}

type {
    type: 302
    name: "COMPUTE_RESPONSE"
}
```

Every peer has a peer type and every message a message type, numbers the
rules file names. A type's rule says where its messages go: to `ANY` one
peer of a type, round robin, or to `ALL` of them, and several rules fan
one message out to several types. A message with `to` set goes to that
one instance whatever the rules say, which is how a reply finds the peer
that asked, so `COMPUTE_RESPONSE` needs no rule. A deployment keeps its
own rules file, the system rules and its own types after them, and the
multiplexers put a changed file in use without a restart ([changing the
rules](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/operations.md#changing-the-rules)); the constants a program
imports are generated from it, by `mxcontrol generate_constants` or a
Bazel build.
[rules.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/rules.md) describes every field and
[docs/README.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/README.md) the words.

## Highlights

- **Highly available, no single point of failure.** Run several multiplexers
  active-active; clients and backends connect to all of them, fail over
  automatically, and reconnect on their own. A rolling restart of three
  multiplexers under traffic costs no request more than a millisecond
  ([rolling_restart](https://github.com/kulewski/multiplexer/blob/v3.0.0/tests/scenarios/rolling_restart/README.md)).
- **One `docker run`, or a StatefulSet on Kubernetes.** The image is 32 MB,
  `mxcontrol` and the system rules on distroless, running as `nonroot`;
  mount a rules file over its own and it is a multiplexer. On Kubernetes a
  StatefulSet behind a headless Service runs as many as you want, with no
  operator, no leader and no shared state, and an edit of the rules'
  ConfigMap is put in use without a rollout
  ([in Docker and on Kubernetes](#in-docker-and-on-kubernetes)).
- **Fault-tolerant requests.** A request whose backend dies, or whose
  connection dies, is retried once, through the backend a search across
  every multiplexer finds, all inside one call
  ([how a query is answered](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/query.md)).
- **Zero-downtime deployments.** A backend asked to leave drains: it tells
  every multiplexer to route it nothing new, finishes what it holds, and
  leaves as soon as they have confirmed, so a Kubernetes rolling restart
  with a preStop hook costs nobody a timeout, or a retry
  ([backend_drains](https://github.com/kulewski/multiplexer/blob/v3.0.0/tests/scenarios/backend_drains/README.md)).
- **Load balancing and fan-out from one rules file.** Round-robin across a
  backend pool, publish/subscribe to every subscriber, or direct addressing,
  chosen per message type, no code change ([rules](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/rules.md)); the
  file is edited under running multiplexers, which put it in use without a
  restart ([changing the rules](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/operations.md#changing-the-rules)).
- **Pools within a type, by key.** A message may say what it is about, a
  tenant, a model, a group, and each backend which keys of a type it
  takes: a tenant's requests reach only the backends that take its key,
  an event only the subscribers of its group, and a failover's retry
  stays with the backends that take the key ([keys](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/keys.md),
  [examples/models](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/models/)).
- **One instance, one path, when it matters.** A request addressed to one
  backend instance reaches it or fails, never another instance, and is
  found again behind another multiplexer; a lane keeps a stream of
  messages on one connection, in order, following a failover or pinned
  ([lanes, pinning and addressed queries](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_python.md#lanes-pinning-and-addressed-queries)).
- **High performance, low latency.** One thread and one event loop per multiplexer, frames forwarded
  as they arrived, never re-serialized, one write per frame with Nagle's
  algorithm off on every connection: over 120,000 request/reply round
  trips per second through one multiplexer on 70% of one core, 17 µs per
  round trip when idle, 7 MB resident, measured with
  [tests/bench.sh](https://github.com/kulewski/multiplexer/blob/v3.0.0/tests/bench.sh) on a Threadripper PRO 9965WX, and as
  fast with Rust peers as with C++ ones.
- **Stateless and lightweight.** No external coordination service, no
  persistence, one binary and one rules file; scale horizontally by adding
  instances and backends. A documented binary protocol over TCP with
  Protocol Buffers ([wire format](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/wire_format.md)).
- **Connections for C++, Python and Rust, and asyncio.** `SyncClient` for a
  program on one thread; `ThreadedClient`, with callback-based
  asynchronous queries, for servers, safe to share between threads; and
  `AsyncClient` for asyncio programs, which awaits queries and sends and
  delivers events to coroutines on the loop; all fork-aware and clean at
  interpreter exit ([Python API](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_python.md), [C++ API](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_cpp.md)).
  In Rust the same classes, `ThreadedClient`'s calls awaited on any
  executor, tokio's included ([Rust API](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_rust.md)).
- **Two server classes that run the loop for you.** `BaseMultiplexerServer`
  runs the loop and your `handle_message()` on one thread, for quick
  handlers; `BaseThreadedMultiplexerServer` runs handlers on worker threads
  behind a heartbeating io thread, for requests that take minutes, several
  at once, or a handler that blocks on a query of its own. Either is the
  usual base of a backend; a backend can also be built on `ThreadedClient`
  or `AsyncClient` ([which to use](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/README.md#which-class-to-build-on)).
- **Tested for every failure mode.** One documented integration scenario per
  failure, AddressSanitizer, ThreadSanitizer, LeakSanitizer, clang thread-safety
  analysis and a soak test ([scenarios](https://github.com/kulewski/multiplexer/blob/v3.0.0/tests/scenarios/README.md)).
- **Test infrastructure you can import.** The harness those scenarios run on
  is a public package: real multiplexers on ephemeral ports in one line,
  scripted backends and clients in-process for unit tests, peers as
  processes for end-to-end tests, and a Bazel macro that runs a scenario
  against the shipped roles or a binary of your own
  ([testing](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_python.md#testing)).
- **See what was routed, when asked.** A peers file lists who is
  connected. A recording holds every routed message, with who sent it, who
  received it or why nobody did; switched on at start, or started, stopped
  and tapped live on a running cluster from one command, with the
  operator's consent given as a flag. A session replays offline and scripts
  test peers from real traffic ([operations](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/operations.md#recording)).

## How it compares

- **RabbitMQ, Kafka:** those persist and queue; the multiplexer does neither.
  A message goes to whoever is connected right now or is dropped, at most
  once per multiplexer, which is what makes it stateless and simple to run.
- **ZeroMQ:** a library without a broker, so peers need each other's
  addresses; here every peer knows only the multiplexers, and routing is
  configuration.
- **gRPC:** point-to-point calls with generated stubs per service; here one
  rules file routes typed messages to pools, subscribers or instances, and a
  request finds a live backend on its own.
- **NATS:** the closest in spirit. NATS servers form a cluster; multiplexers
  do not know each other, and the clients provide the redundancy by
  connecting to all of them.

[Semantics: delivery, failures and defaults](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/semantics.md) states
exactly what is promised, for the deployment the multiplexer is designed for:
several of them, every peer connected to all.

## Examples

Complete programs under [examples/](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/), each with a README that
runs it and a walkthrough that builds it from nothing, echo's apart, which
runs it step by step. Four small ones first, then building blocks, then
what the broker's latency makes possible:

- [echo](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/echo/): a backend and a client, in Python and in C++,
  and threads sharing one client.
- [echo-rust](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/echo-rust/): the same backend and client in Rust,
  the backend draining on `SIGTERM`.
- [aio](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/aio/): an asyncio TCP gateway in front of a chat backend,
  a reply to the one who asked and a broadcast to everyone.
- [health](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/health/): a backend's health check that sees its
  serve loop, a Kubernetes liveness probe beside the drain on a preStop
  hook.
- [cache](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/cache/): a replicated cache as a Django cache backend, a
  write to every replica and a read from any, and a journal added by
  changing the rules under the running multiplexers.
- [channels](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/channels/): a Django Channels channel layer, Redis
  replaced by one setting.
- [inference](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/inference/): a web app asking a pool of PyTorch
  workers, with a worker killed, a rolling restart and a multiplexer
  killed, in an executed notebook.
- [models](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/models/): workers that each hold some of the
  models, a request's key naming its model and each worker's key filter
  the models it holds, and a model loaded, and unloaded while its
  requests keep coming.
- [stream](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/stream/): an answer that arrives token by token, as
  a language model's does, one request and one reply with the pieces
  addressed and numbered between them, a multiplexer killed under it
  and the lost pieces asked for again, and the answer as Server-Sent
  Events.
- [audio](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/audio/): an audio room, a query per 10 ms frame to a
  C++ worker and the answer heard by everyone, the worker replaced under
  the stream.

## Using it from Python

`pip install mx-multiplexer` installs the package, its compiled extension
included, and the `mxcontrol` command, which runs the multiplexer and its
tools, on Linux x86_64 for CPython 3.10 and newer; the import is
`multiplexer`, since `multiplexer` on PyPI is an unrelated package.
`mxcontrol generate_rules your.rules` writes the system rules every rules
file starts from; add your own types after them, and `mxcontrol
generate_constants your.rules --python multiplexer_constants.py --pyi
multiplexer_constants.pyi` writes their constants, once and again whenever
the file changes, for `from multiplexer_constants import peers, types`. A
Bazel build generates them on its own.

The quick start's two classes are the usual start: a backend built on
`BaseMultiplexerServer`, whose `serve_forever()` runs the loop and calls
`handle_message` for each request, `send_message` replying to the peer
that asked, and a `ThreadedClient`, safe to share between threads, whose
`query()` sends a request and returns the answer. An event is sent the
same way without waiting for an answer, `client.send_message(b"payload",
type=types.SOME_EVENT)`, and `to=<an instance id>` addresses one peer,
bypassing the routing rules. `SyncClient` (named `Client` before 2.4.0),
`AsyncClient` for asyncio and `BaseThreadedMultiplexerServer`, whose
handlers run on worker threads, are in the [Python API](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_python.md),
and [which class to build on](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/README.md#which-class-to-build-on)
compares them all. A test starts real multiplexers with
`multiplexer.testing`, which runs the package's `mxcontrol`
([testing](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_python.md#testing)).

## Using it from C++

pip installs no C++ library: the Debian and Ubuntu packages on the
release page hold it, its headers and a pkg-config file, besides
`mxcontrol`, one package per release, because protobuf's C++ library promises no compatibility between
versions and the library is built against each release's own. With the
package installed, and `libprotobuf-dev` and `libasio-dev`:

```
sudo apt install ./multiplexer_<version>~<codename>_amd64.deb
g++ -std=c++17 backend.cc $(pkg-config --cflags --libs multiplexer) -o backend
```

The C++ library has the Python package's classes but the asyncio one:
`SyncClient` with `query`, `schedule_one` and `schedule_all`,
`ThreadedClient`, `BaseMultiplexerServer` and
`BaseThreadedMultiplexerServer`; [examples/echo/backend.cc](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/echo/backend.cc)
is a complete backend and the [C++ API](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_cpp.md) describes them. The constants of your own
rules file come from `mxcontrol generate_constants your.rules --cxx
multiplexer/multiplexer.constants.h`, placed on the include path before
the package's copy ([packaging](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/packaging.md#the-debian-packages)).

## Using it from Rust

The crate `mx-multiplexer` is the Rust library, written in Rust on mio
rather than bound to the C++ one, with the classes of the other two and
their behaviour: `SyncClient`, `ThreadedClient`, whose calls are also
awaited on any executor, tokio's included, with no runtime of its own,
and the server classes `BaseMultiplexerServer` and
`BaseThreadedMultiplexerServer`, which serve a `Handler`. `cargo add
mx-multiplexer` adds it, Rust 1.85 or newer on Linux; the constants of
your own rules file come from `mxcontrol generate_constants your.rules
--rust src/multiplexer_constants.rs`, written once and again whenever the
file changes. The [Rust API](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_rust.md) describes the classes, and
[examples/echo-rust](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/echo-rust/) is a complete backend and client.

## Getting it

Two ways, and the choice is only whether you build it:

- **A release.** `pip install mx-multiplexer` fetches the Python package
  from PyPI, `mxcontrol` inside, one manylinux wheel per CPython from 3.10.
  Every release on GitHub ships the same set, built from one commit and
  stripped: those wheels, `mxcontrol` for any x86_64 Linux with glibc 2.28
  or newer, the
  container image `ghcr.io/kulewski/multiplexer:<version>`, also tagged
  `latest`, with nothing in it but `mxcontrol` and the system rules, a
  Debian package per Debian and Ubuntu release with `mxcontrol`, the C++
  library, its headers and a pkg-config file, and the Rust crate, which
  crates.io serves as `mx-multiplexer`.
  [docs/packaging.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/packaging.md) says how to use each;
  [in Docker and on Kubernetes](#in-docker-and-on-kubernetes) runs the
  image in both, and [docs/operations.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/operations.md#on-kubernetes)
  has the complete Kubernetes manifest.
- **From source**, with Bazel, which is how the repository is developed
  and tested and how another Bazel workspace consumes it, or with `make`
  from the distribution's own packages: [building from
  source](#building-from-source).

## Building from source

### Building with Bazel

Prerequisites, verified on clean Debian 12, Ubuntu 24.04 and Debian 13
installations: Bazel 6.2 through Bazelisk, a C++17 compiler,
`protobuf-compiler` with `libprotobuf-dev`, Python 3.10 or newer with
`python3-dev` and `python3-protobuf`. Asio and pybind11 are fetched by
Bazel. [docs/building.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/building.md) has the exact packages, how to
bring your own protobuf, and `docker/check.sh`, which proves the list on a
clean container.

```
bazel build //...
bazel test //...                              # unit and integration tests
bazel test --test_tag_filters=-slow //...     # the fast ones, a few seconds
bazel run //mxcontrol run_multiplexer
cd examples/echo && bazel test //...          # a backend and a client, built the way your code will be
```

The integration tests under [tests/](https://github.com/kulewski/multiplexer/blob/v3.0.0/tests/) start real multiplexers and play
peers in Python, C++ and Rust; see [tests/README.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/tests/README.md). `mxcontrol
run_multiplexer` accepts `--address host:0 --port-file PATH` to pick a free
port and report it, and on SIGTERM sends what it holds, closes its
connections and exits with 0.

`--config=debug` keeps debug info and frame pointers; `--config=release` builds optimized, stripped binaries. Run the server inside `screen` or `tmux` if you want to get back to the session later.

#### Your own peer and message types

`multiplexer.rules` in this repository is the system rules, which every rules file starts from. Keep your deployment's rules file in your own repository and point the build at it:

```
bazel build --//:multiplexer_rules=//your/pkg:multiplexer.rules //...
```

When this repository is consumed as an external Bazel repository named `mx`, the flag is `--@mx//:multiplexer_rules=...`. At run time `mxcontrol run_multiplexer` reads `multiplexer.rules` from the current directory unless `--rules` points elsewhere.

### Building without Bazel

For a machine that will not have Bazel, a `Makefile` builds the same things
from the distribution's own compiler, protobuf, Asio and pybind11:

```
make -j                      # build/bin/mxcontrol, build/libmultiplexer.a with headers, build/python/
make check                   # the C++ and Python unit tests, against what was built
make wheel                   # a pip wheel of the Python package, mxcontrol inside
sudo make install            # mxcontrol, the library and the headers under /usr/local
make RULES=your.rules -j     # the constants from your rules file
```

[docs/building.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/building.md#without-bazel) lists the packages and
what a program links and imports. The test roles, the scenarios, the
examples and the sanitizer builds stay with Bazel.

### Using it from another Bazel workspace

Declare this repository as `mx`, then two calls in your `WORKSPACE`:

```
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "mx",
    commit = "<commit sha>",
    remote = "https://github.com/kulewski/multiplexer.git",
)

load("@mx//bazel:deps.bzl", "mx_dependencies")
mx_dependencies()
load("@mx//bazel:setup.bzl", "mx_setup")
mx_setup()
```

Then depend on `@mx//multiplexer:clients` (`SyncClient`) or
`@mx//multiplexer:servers` (`BaseMultiplexerServer`) from Python,
`@mx//multiplexer:client` from C++; import and include paths are unchanged.
A workspace that uses C++ alone calls `mx_dependencies(python = False)`
and no `mx_setup()`, and fetches no Python rules. [examples/](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/) holds
complete workspaces built that way, starting with [examples/echo](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/echo).
Your tests get `@mx//multiplexer/testing`: real multiplexers on ephemeral
ports, scripted peers, and the macro this repository's own scenarios use
([docs/api_python.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/api_python.md#testing)).

## Where to read next

[docs/README.md](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/README.md) defines the terminology, shows the
reference deployment and traces a query and an event step by step. The
[examples](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/) carry on from the quick start:
[cache](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/cache/walkthrough.md) builds a pip project from nothing,
and [echo](https://github.com/kulewski/multiplexer/blob/v3.0.0/examples/echo/walkthrough.md) runs the smallest program with
Bazel and reads what each side prints. The reference pages
under [docs/](https://github.com/kulewski/multiplexer/blob/v3.0.0/docs/) cover the rules file, the client libraries,
`mxcontrol`, the wire format and operations.

## History

Multiplexer was built at Azouk Network Ltd. in 2008-2009, first as a Python
prototype by Krzysztof Kulewski and then as the C++ implementation this
repository grew out of, by Piotr Findeisen. Azouk released it under the MIT
license, and Krzysztof Kulewski has owned and developed it since 2010, in
production use throughout. See [AUTHORS](https://github.com/kulewski/multiplexer/blob/v3.0.0/AUTHORS).

## License

MIT, see [LICENSE](https://github.com/kulewski/multiplexer/blob/v3.0.0/LICENSE). What the binaries hold of other projects,
compiled in, is in [THIRD_PARTY_NOTICES](https://github.com/kulewski/multiplexer/blob/v3.0.0/THIRD_PARTY_NOTICES).
