Metadata-Version: 2.4
Name: ds-service-client
Version: 5.2.0
Summary: Python client for DS service.
Author-email: Parantapa Bhattacharya <pb@parantapa.net>
License-Expression: MIT
Project-URL: Homepage, https://github.com/parantapa/ds-service
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: grpcio>=1.75.1
Requires-Dist: protobuf>=6.31.1
Requires-Dist: ifaddr>=0.2
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# ds-service: An in-memory data structure server

![Futuristic banner image.](misc/banner-image.png "Futuristic banner image.")

`ds-service` is a small, in-memory data structure server
that is accessible via [gRPC](https://grpc.io/).

`ds-service` runs a single server process
that holds shared state in memory
and lets many distributed clients and workers coordinate using it.
Use it when several processes need to hand work to each other.
The processes run on one machine or across a cluster.
It covers four cases:

- The processes hand work to each other.
- The processes share intermediate results.
- The processes take turns on a resource.
- The processes agree on a number.

The state only has to live as long as the run does.

Presently, it provides six data structures:
a key-value store, a task queue, a journal store, a time series store,
named mutexes, and counters.
Each is a separate key space with its own set of RPCs,
described in the [data structure reference](docs/data-structure-reference.md).

## Installation

The server is a single statically linked binary.
Download the latest release, make it executable,
and put it somewhere on your `PATH`:

```sh
curl -sSL -o ds-service \
    https://github.com/parantapa/ds-service/releases/latest/download/ds-service
chmod +x ds-service
```

It links against musl with no dynamic dependencies,
so it runs on any x86-64 Linux host.

The Python client comes from PyPI:

```sh
pip install ds-service-client
```

To build the server from source instead,
see [how to build the server](docs/howto-build-the-server.md).

## Usage

Start a server:

```sh
ds-service --address 127.0.0.1:5051
```

Then, from any process that can reach it:

```python
from ds_service_client import DsServiceClient

with DsServiceClient("127.0.0.1:5051") as client:
    client.map_set("greeting", b"hello")
    assert client.map_get("greeting") == b"hello"

    client.task_add("job-1", queue="work", priority=1.0, function=b"greet", input=b"world")

    task = client.task_get(worker_id="worker-a", queue="work")
    client.task_done(task.task_id, worker_id="worker-a", output=b"hello world")

    assert client.task_get_output("job-1") == b"hello world"
```

## Documentation

| Document | What it covers |
| --- | --- |
| [Tutorial: run your first tasks through ds-service](docs/tutorial-your-first-tasks.md) | Start a server, store a value, and take a task from `Ready` to `Complete`. Start here. |
| [How to build the ds-service server](docs/howto-build-the-server.md) | Requirements, the Conan and CMake build, how to install and run the binary, and the static musl build. |
| [How to write a worker](docs/howto-write-a-worker.md) | The claim-work-report loop, mutexes around shared resources, progress reporting, and the asyncio variant. |
| [Data structure reference](docs/data-structure-reference.md) | Every RPC, its arguments and error statuses, and the exact semantics of each data structure. |
| [Python client reference](docs/python-client-reference.md) | `DsServiceClient` and `DsServiceClientAsync`: constructors, method names, the mapping from a gRPC status to an exception, and examples. |
| [Server helper reference](docs/server-helper-reference.md) | `DsServiceServer`, which runs a private `ds-service` process for the life of the object. |
| [About the architecture](docs/about-the-architecture.md) | The three pieces, why the server does not persist state, one lock per structure, and why there are two Python clients. |
| [About the task queue](docs/about-the-task-queue.md) | Task ownership, what canceling does and does not do, and why there is no fault tolerance. |
| [About the static musl build](docs/about-the-static-musl-build.md) | Why the static image exists and why its Conan profile differs. |

- [Developer notes](docs/developer-notes.md)
- [Report a bug](https://github.com/parantapa/ds-service/issues)

## License

MIT. See [LICENSE](LICENSE).
