Metadata-Version: 2.5
Name: highstorm
Version: 0.1.0
Summary: Replay historical records into Kafka at controlled speed
Project-URL: Documentation, https://github.com/RohanDalton/highstorm/tree/main/docs
Project-URL: Home, https://github.com/RohanDalton/highstorm
Project-URL: Issues, https://github.com/RohanDalton/highstorm/issues
Author-email: "Rohan B. Dalton" <rohan@rohandalton.io>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# Highstorm

Highstorm stress-tests Kafka and Redpanda deployments by replaying historical
records at a controlled speed. A Rust core does all per-record work; a thin
CLI and a thin Python package drive the same engine, so a run configured the
same way behaves the same way from a shell or a script.

- **Sources:** a finite snapshot of a Kafka/Redpanda topic, or local Parquet
  files in a message-envelope layout or a structured (one column per field)
  archive layout.
- **Destination:** one Kafka/Redpanda topic, produced with `acks=all` and
  idempotence by default, with partition preservation or key partitioning.
- **Timing:** records are read ahead as fast as bounded buffers allow and sent
  on the historical schedule scaled by `--speed` (default `1.0`), or as fast
  as possible with `--unpaced`. Every deadline derives from one anchor, so
  overhead never accumulates into drift, and lateness is measured rather than
  hidden.
- **Range:** `[start, end)` in RFC 3339 with an explicit timezone.
- **Configuration:** a `kubectl`-style TOML file with named clusters,
  registries, sources, destinations, and contexts, each with its own
  credentials; environment overrides; or a fully environment-based setup.
  See [docs/configuration.md](docs/configuration.md).
- **Reports:** every run is a stress-test result with delivery outcomes,
  scheduling lateness, acknowledgement latency, buffer and queue pressure,
  starvation, and the redacted effective configuration.
  See [docs/reports.md](docs/reports.md).

## Quick start

Install the Python package and its `highstorm` command with Python 3.9 or newer:

```console
python -m pip install highstorm
```

Wheels contain the Rust extension. If a wheel is unavailable for your platform,
installation builds from source and requires Rust 1.85+, a C/C++ compiler,
CMake, pkg-config, and the development libraries for OpenSSL, SASL, curl, and
zlib. For a checkout, Pixi provides these dependencies:

```console
pixi run develop
```

```console
highstorm config init                                  # writes highstorm.toml
highstorm config set source kholinar
highstorm config set destination urithiru
highstorm config view --effective

highstorm inspect --source kholinar                    # metadata, timestamp samples, mapping issues
highstorm replay --dry-run                             # connectivity and compatibility, no records
highstorm replay --dry-run --scan                      # also read the whole source through the engine

highstorm replay \
  --start 2026-04-09T09:00:00Z \
  --end 2026-04-09T10:00:00Z \
  --speed 10

highstorm replay --unpaced --report run.json
```

Ctrl-C stops reading, drains outstanding deliveries within the delivery
timeout, and still prints the report with `status = "cancelled"`. Progress
lines go to standard error every `--progress-seconds` (`--quiet` disables
them); the JSON report goes to standard output and to `--report <file>`.

The same workflow from Python (see [docs/python.md](docs/python.md)):

```python
import highstorm

configuration = highstorm.Config.load("highstorm.toml")
report = highstorm.replay(
    configuration,
    source="kholinar",
    destination="urithiru",
    start="2026-04-09T09:00:00Z",
    end="2026-04-09T10:00:00Z",
    speed=10.0,
)
print(report.status, report.records_sent, report.scheduling_lateness.p99_seconds)
```

`highstorm.start_replay` returns a handle with `cancel()`, `statistics()`, and
`wait(timeout)`; `highstorm.inspect` and `highstorm.dry_run` mirror the CLI.

## Replay contract

| Concern | Behaviour |
|---|---|
| Source consumption | Read ahead as fast as the record and byte limits allow |
| Kafka snapshot | Partition end offsets are captured before reading; later records belong to a later run |
| Isolation | `read_committed` by default (end = last stable offset), `read_uncommitted` optional |
| Scheduling timestamp | Kafka record time by default; header, DBN `ts_event`, or Avro field on request |
| Ordering | Source order within each partition or file; timestamp regressions are clamped and counted, or rejected with `--strict-timestamps` |
| Record contents | Keys, values, nulls, headers, and bytes preserved unless a mapping requires conversion |
| Outgoing timestamp | Historical by default; `kafka_timestamp = "send_time"` optional |
| Failure handling | Stop on invalid records or permanent delivery errors; report partial results |
| Completion | All selected records processed and all outstanding deliveries resolved or reported unresolved |

Exactly-once replay across clusters or restarts is not promised: a restarted
run can produce duplicates even with idempotence enabled.

## Development

Pixi manages the toolchain and tasks:

| Task | Purpose |
|---|---|
| `pixi run test` | Format check, pedantic clippy on all targets, Rust suites, Python suite |
| `pixi run test-brokers` | Broker suite against disposable Redpanda and Apache Kafka containers |
| `pixi run develop` | Build and install the Python extension in place |
| `pixi run build-wheel` / `pixi run build-binary` | Release wheel / standalone executable |
| `pixi run bench` | Criterion engine benchmarks with in-memory adapters |
| `pixi run benchmark-brokers` | Throughput baseline through a real Redpanda |
| `pixi run smoke-release` | Clean-install smoke test of the binary and the wheel |
| `pixi run fixtures` | Regenerate the golden Parquet and DBN fixtures |

## Python releases

Hatchling is the Python build backend. Its
[Maturin hook](https://github.com/cjames23/hatch-maturin-build) compiles the Rust
extension and preserves its native wheel tags and bundled libraries. The
Python distribution version is read dynamically from
`highstorm/__version__.py` and exposed as `highstorm.__version__`.
`highstorm.core_version()` and the CLI's `--version` report the Rust workspace
version in `Cargo.toml`; update that separately when releasing core changes.

With [just](https://just.systems) and Pixi installed:

```console
pixi run -e build hatch version patch  # or an explicit version, e.g. 0.2.0
just build                            # builds dist/*.tar.gz and dist/*.whl, then checks metadata
```

The source archive includes the Rust workspace and lockfile, so it can rebuild
the extension. Each wheel targets the build machine's OS and architecture;
build on each platform you intend to provide wheels for. Linux builds must
produce a PyPI-compatible manylinux or musllinux wheel; the build rejects plain
`linux` wheel tags.

To publish, configure Twine with your PyPI API token (for example, set
`TWINE_USERNAME=__token__` and `TWINE_PASSWORD` in your shell), then run:

```console
just release
```

This rebuilds `dist/`, runs `twine check --strict`, and uploads both artifacts to
PyPI. For TestPyPI, set `TWINE_REPOSITORY=testpypi` and use a TestPyPI token.

## Repository layout

`docker build -t highstorm .` produces a standalone image whose entrypoint is
the executable. Benchmark results and how to reproduce them are in
[docs/benchmarks.md](docs/benchmarks.md).

Layout:

```text
crates/highstorm-core/     configuration, sources, Avro, engine, delivery, runner, inspection
crates/highstorm-cli/      argument parsing and terminal output
crates/highstorm-python/   PyO3 bindings
highstorm/                 Python package: typed API, exceptions, reports, console entry point
tests/fixtures/            golden DBN, Avro, and Parquet fixtures
examples/                  configuration files and a Python script
docs/                      configuration, Python, reports, benchmarks
```

Out of scope for this release: distributed workers, live tailing, resumable
checkpoints, direct S3 reads, and arbitrary Python record transforms.
