Metadata-Version: 2.4
Name: onestep-elasticsearch
Version: 0.1.1
Summary: Elasticsearch and OpenSearch connector plugin for onestep.
License: MIT
Requires-Python: >=3.9
Requires-Dist: httpx>=0.27
Requires-Dist: onestep>=1.7.1
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'test'
Requires-Dist: pytest>=8.0.0; extra == 'test'
Description-Content-Type: text/markdown

# onestep-elasticsearch

`onestep-elasticsearch` provides one asynchronous bulk sink for the common
Elasticsearch and OpenSearch HTTP API surface.

## Install

```bash
pip install onestep-elasticsearch
```

## Python

```python
from onestep_elasticsearch import ElasticsearchConnector

search = ElasticsearchConnector(
    ["https://search-1:9200", "https://search-2:9200"],
    distribution="auto",
    username="ingest",
    password="secret",
    verify_certs=True,
    ca_certs="/etc/ssl/search-ca.pem",
    request_timeout_s=30.0,
)
sink = search.bulk_sink(
    index="events-v1", operation="index", id_field="event_id",
    chunk_size=500, max_chunk_bytes=5_000_000, refresh=False,
)
```

## YAML

```yaml
resources:
  search:
    type: elasticsearch
    hosts: ["${SEARCH_URL}"]
    distribution: auto
    username: "${SEARCH_USERNAME}"
    password: "${SEARCH_PASSWORD}"
    verify_certs: true
    ca_certs: "${SEARCH_CA_FILE:-/etc/ssl/certs/ca-certificates.crt}"
    request_timeout_s: 30
  events:
    type: elasticsearch_bulk_sink
    connector: search
    index: events-v1
    operation: index
    id_field: event_id
    chunk_size: 500
    max_chunk_bytes: 5000000
    refresh: false
```

## Payloads and transport

`send()` accepts one mapping or one non-empty sequence of mappings. Each mapping
is the complete document `_source`; `id_field` also supplies `_id` while remaining
in `_source`. The sink chunks sequentially by action count and serialized NDJSON
bytes, and rejects invalid or oversized payloads before submission.

The common boundary uses HTTP(S), `GET /`, and `POST /_bulk`. `distribution` is
`auto`, `elasticsearch`, or `opensearch`. Configure exactly one authentication
mode: Basic `username`/`password`, `api_key`, or `bearer_token`. Custom `headers`,
`verify_certs`, `ca_certs`, `client_cert`, and `client_key` cover the supported TLS
and proxy boundary. Clients are created lazily; an injected client remains caller
owned, while a connector-owned client is closed by `await connector.close()`.

## Compatibility

The supported release matrix is Elasticsearch 8.x and 9.x plus OpenSearch 2.x and
3.x. Compatibility is defined by the shared HTTP bulk behavior, not a vendor Python
client version. The environment-gated live suite uses `ONESTEP_ELASTICSEARCH_URL`
or `ONESTEP_OPENSEARCH_URL`.

## Delivery semantics

`send()` returns only after every bulk item in every chunk is acknowledged. onestep
acknowledges the source after sink sends, so a crash between the bulk acknowledgement
and source acknowledgement can duplicate output. Multi-sink fan-out is not
transactional. Use `operation: index` with a stable `id_field` when replay must
converge; auto-generated IDs and `create` are not replay-safe.

Bulk chunks are not transactional. Item failures retain a redacted typed cause with
the item index, status, identifier, and normalized reason. A partial commit is
reported as `UNCERTAIN` unless `index` with a stable `id_field` makes replay
deterministic. `create` conflicts and malformed documents are permanent failures.

## Deferred features

Version 1 intentionally excludes Cloud ID, sniffing, SigV4, data streams,
administration APIs, dynamic actions, update/delete, PIT, SQL,
`elasticsearch_search_after`, and `elasticsearch_scroll`.
