Metadata-Version: 2.4
Name: sse-lib
Version: 0.1.2
Summary: Thin Python client for the Semantic Search Engine (SSE) REST API
Author: RadLab
License: Apache-2.0
Project-URL: Homepage, https://github.com/radlab-dev-group/semantic-search-engine
Project-URL: Repository, https://github.com/radlab-dev-group/semantic-search-engine
Project-URL: Issues, https://github.com/radlab-dev-group/semantic-search-engine/issues
Keywords: semantic-search,rag,rest,client,sse,search-engine
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: requests<3.0,>=2.31
Provides-Extra: dev
Requires-Dist: pytest<10.0,>=8.0; extra == "dev"
Requires-Dist: black<26.0,>=24.1; extra == "dev"
Requires-Dist: flake8<8.0,>=7.0; extra == "dev"

# sse_lib

Thin Python client for the **Semantic Search Engine** (SSE) REST API
(`src/sse_api/` in this repository).

`sse_lib` is a client-side layer only: it maps Python calls onto the HTTP
endpoints of a running SSE backend and turns the JSON answers into small
dataclasses. It does **not** import Django, `torch`, `pymilvus` or any other
server dependency — the single runtime requirement is `requests`.

```python
from sse_lib import SSEClient, SearchOptions, GenerativeOptions

client = SSEClient("http://localhost:8271", token="...")

client.create_collection(
    "my_docs",
    "My Documents",
    "Test collection",
    embedder="radlab/polish-bi-encoder-mean",
    reranker="radlab/polish-cross-encoder",
    index_type="HNSW",
    chunking={"max_tokens_in_chunk": 128, "number_of_overlap_tokens": 0},
)
client.upload_files("my_docs", ["invoice.pdf", "regulations.zip"])
client.index_documents("my_docs", ["Some text to index."], category="Notes")

found = client.search("my_docs", "policy on data retention",
                      SearchOptions(max_results=20, rerank_results=True))
answer = client.generate_answer(
    found.query_response_id,
    GenerativeOptions(
        generative_model="radlab/pLLama-3-8B-DPO-L",
        percentage_rank_mass=40,
    ),
)
print(answer.answer)
```

Chunking belongs to the **collection**, not to a single call: declare it with
`create_collection(..., chunking={...})` (or `strict_chunking=True` to forbid any
deviation) and the indexing calls above may leave both keys out — the server takes
the settings of the collection. `client.embedders_info()` returns
`max_model_length` of every embedder, the largest chunk its model can take in
(`client.rerankers_info()` the pair budget of the rerankers). Passing
`max_tokens_in_chunk` / `number_of_overlap_tokens` per call, or once per client
with `SSEClient(..., default_indexing_options={...})`, still works; a split which
cannot exist (`overlap >= chunk`) raises `SSEValueError` before the request goes
out. `percentage_rank_mass` of `generate_answer` is required by the backend and
has no library-wide default on purpose.

## Installation

From a checkout:

```bash
pip install ./sse_lib            # or: pip install -e ./sse_lib  (development)
```

Once published to a registry, `pip install sse-lib` is all that is needed — the
package is standalone and versioned independently (`sse_lib.__version__`).

Python 3.9+, `requests` as the only runtime dependency.

## Authentication

| Mode | How |
|------|-----|
| Pre-existing token (default DRF token auth) | `SSEClient(host, token="...")` — sent as `Authorization: Token <value>` |
| Login with user and password | `SSEClient(host, username="user", password="...")` — `POST login` runs lazily, before the first request |
| Keycloak / OAuth (backend started with `ENV_USE_KC_AUTH=1`) | `client.login_url()` → redirect the user → `client.login_with_code(code, state)`; later `client.refresh_access_token()` and `client.logout()` |
| Custom scheme | `SSEClient(host, token="...", token_type="Bearer")` |

## Configuration

| Constructor argument | Environment variable | Default | Meaning |
|----------------------|----------------------|---------|---------|
| `api_host` | `SSE_API_HOST` | – | `http://localhost:8271`; the scheme may be omitted (`localhost` → `http`, anything else → `https`) |
| `api_prefix` | `SSE_API_PREFIX` | `api` | Path between the host and the endpoints; set `api/v1` for versioned deployments (`main.api` in `django-config.json`) |
| `token` | `SSE_API_TOKEN` | – | Value of the `Authorization` header |
| `language` | `SSE_API_LANGUAGE` | `pl` | Sent as `lang` on every call (`pl` / `en`) |
| `timeout` | `SSE_API_TIMEOUT` | `60.0` | Per-request timeout in seconds |
| `verify`, `session`, `headers`, `max_retries` | – | `True`, new session, `{}`, `2` | TLS verification, shared `requests.Session`, extra headers, GET retries |
| `default_search_options`, `default_indexing_options` | – | – | Client-wide option defaults merged into every search / indexing call |

`SSEClient` is a context manager; leaving the block closes the HTTP session.

### Per-file upload metadata

Upload one whole document per file; SSE performs chunking using the same values
as `create_collection(..., chunking=...)`:

```python
name = "doc:20261022:abc123.txt"
result = client.upload_files(
    "my_docs",
    [(name, "Whole document text".encode("utf-8"), "text/plain")],
    indexing_options={"max_tokens_in_chunk": 400, "number_of_overlap_tokens": 60},
    file_metadata={name: {
        "category": "doc",
        "options": {"source_file": "docs/guide.md", "scope": "project",
                    "doc_type": "markdown"},
    }},
)
```

`file_metadata` is a separate multipart JSON string, not an indexing option.
When supplied, it must cover every non-ZIP file exactly once. Each entry requires
`category` (string, including an empty string) and `options` (JSON object); other
entry fields are rejected. Exact names are preferred; unique basename aliases
are accepted. Missing/unknown/ambiguous names, duplicate aliases or JSON keys,
invalid shapes and non-finite numbers are rejected. ZIP archives may be included
in the batch, but neither archives nor their members may have metadata entries.

`category` is stored in `Document.category`; `options` is merged into
`Document.metadata_json`, with supplied values winning over reader values.
It is not stored under an extra `options` wrapper. Nested JSON values are
preserved. Omit `file_metadata` (or pass `None`) for legacy behavior, including ZIP
directory categories. The constructor and upload response are unchanged.

Both the API and Celery workers must run the updated server code. No schema
migration or new dependency is required: metadata is persisted in the existing
`IndexingJob.request_payload` before dispatch. Update/reinstall `sse_lib` in client
environments as well; an older server does not support this contract.

For a no-server transport test, see
`tests/test_client.py::IndexingTest.test_worker_upload_bytes_with_file_metadata`
and `tests/support.py`: `build_client` constructs the real
`SSEClient(host, token=..., transport=Transport(host, session=FakeSession(...)))`.
The fake session returns real `requests.Response` objects with the envelope
`{"status": true, "body": {"dir_hash": "h", "is_indexed": true}}`.

## API coverage

| Client method | REST endpoint |
|---------------|---------------|
| `create_collection(...)` | `POST new_collection` |
| `list_collections()`, `get_collection(name)`, `collection_exists(name)` | `GET collections` |
| `list_categories(collection)` | `GET categories` |
| `list_documents(collection, with_disabled=False)`, `list_document_details(collection)` | `GET documents` |
| `upload_files(collection, files, ...)` | `POST upload_and_index_files` |
| `document_content(collection, document=None, from_page=None, max_pages=None, chunk_type=None, text_id=None)` | `GET document_content` (`text_id` alone is enough and returns the chunk's page with `.highlight`) |
| `index_documents(collection, documents, ...)` | `POST add_and_index_texts` |
| `indexing_status(collection=None, job_id=None)` | `GET indexing_status` |
| `delete_texts(collection, chunk_ids)` | `POST delete_texts` |
| `delete_documents(collection, documents=None, document_ids=None)` | `POST delete_documents` |
| `delete_collection(collection)` | `POST delete_collection` |
| `set_documents_use_in_search(collection, documents=None, document_ids=None, use_in_search=False)`, `deactivate_documents(...)`, `activate_documents(...)` | `POST set_documents_use_in_search` |
| `list_question_templates()` | `GET question_templates` |
| `list_filter_options()` | `GET filter_options` |
| `search(collection, query, options, ...)` | `POST search_with_options` |
| `generate_answer(query_response_id, options, ...)` | `POST generative_answer` |
| `generative_answer_with_citations(query_response_id, options, ...)` | `POST generative_answer` with numbered sources enabled |
| `rewrite_query(query, generative_model=..., historical_messages=None)` | `POST rewrite_query` |
| `rate_answer(answer_response_id, value, value_max, comment=None)` | `POST rate_generative_answer` |
| `list_search_history(collection=None, query_response_id=None)`, `get_search_history(query_response_id)`, `delete_search_history(query_ids=None, query_response_ids=None)` | `GET search_history`, `GET get_search_history`, `POST delete_search_history` |
| `list_generative_models()`, `list_embedders()`, `list_rerankers()` | `GET generative_models`, `GET embedders`, `GET rerankers` |
| `embedders_info()`, `rerankers_info()` | `GET embedders`, `GET rerankers` — `models_info` with `vector_size`, `device` and `max_model_length` |
| `new_chat(...)`, `send_chat_message(...)`, `save_chat(...)`, `get_chat_by_hash(...)`, `list_chats()`, `delete_chat(chat_ids)` | `POST new_chat`, `POST add_user_message`, `POST save_chat`, `GET get_chat_by_hash`, `GET chats`, `POST delete_chat` |
| `login()`, `login_with_code()`, `refresh_access_token()`, `logout()`, `login_url()` | `POST login`, `POST refresh_token`, `POST logout`, `POST login_url` |
| `health()`, `is_healthy()` | `GET healthz` (public — no token, no login) |
| `request(method, endpoint, ...)`, `get(...)`, `post(...)` | any endpoint, including ones this release does not wrap |

Collections may be passed as a name, a `Collection` returned by
`create_collection()` / `list_collections()`, or any mapping with a `name` key.
A `Collection` also carries the settings of the collection: `metadata_json`,
`chunking` (`{}` when none was declared) and `strict_chunking`.

Rewrite a follow-up before searching, then request an answer with citations:

```python
query = client.rewrite_query(
    "When?",
    generative_model="my-model",
    historical_messages=[{"user": "Who founded Acme?", "assistant": "Ada."}],
)
found = client.search("my_docs", query)
answer = client.generative_answer_with_citations(
    found.query_response_id,
    GenerativeOptions(generative_model="my-model", percentage_rank_mass=40),
)
print(answer.answer, answer.citations)
```

`rewrite_query()` returns a string and does not search or save a conversation.
History turns are chronological and contain exactly `user` and `assistant`
strings; the client's `language` is forwarded to SSE. This method requires
the SSE `rewrite_query` endpoint from this release, not just an updated router.
`generative_answer_with_citations()` accepts the same arguments as
`generate_answer()` and always enables numbered sources, even when options
explicitly disable them. It returns an `Answer` with `.citations` and delegates
through SSE to the router's dedicated citation endpoint; it does not call the
router directly or accept arbitrary document fragments.

The answer of `GET collections` also carries the counters of the collection:
`document_count` (every document of the collection) and
`active_document_count` (the documents which take part in the search,
`use_in_search=True`).  A `Collection` exposes both — `None` when the server
did not send them — and `document_count_or_zero` / `active_document_count_or_zero`
turn the missing counters into `0`.

`GET documents` has two views.  The default one lists the names of the
documents which take part in the search (`list_documents()`); with
`with_disabled=True` — and always in `list_document_details()` — the server
adds the deactivated documents together with their `id` and `use_in_search`
flag.  Those identifiers are what `document_ids[]` of `delete_documents` and
`set_documents_use_in_search` expects.

`send_chat_message(..., collection="my_docs")` requires an explicit collection,
even if the chat was created with one: the current server resolves the collection
from each message request. Omitting it raises `SSEValueError` before sending.

Set `api_prefix=""` for deployments without an API path prefix; an explicitly
provided empty prefix overrides `SSE_API_PREFIX`.

## Options objects

| Class | Used by | Notes |
|-------|---------|-------|
| `SearchOptions` | `search()` | Only the fields that are set are sent; `extra` merges unknown keys |
| `IndexingOptions` | `upload_files()`, `index_documents()` | Serialises the five cleaning keys the server reads with `options["..."]`; the two chunk keys and `delete_files_after_indexing` appear only when set, so the collection and the engine decide |
| `GenerativeOptions` | `generate_answer()`, `send_chat_message()` | `None` fields are dropped; carries `use_doc_names_in_response`, the flag the generator reads from the same `query_options` |
| `ChatOptions` | `new_chat()`, `send_chat_message()` (`chat_options=`) | The pipeline switches (`use_rag_supervisor`, `use_content_supervisor`, `use_intelligent_query_rewrite`, `query_instruction`, `user_msg_is_rag_question`) and the RAG memory settings (`system_prompt_with_enumerated_documents`, `query_rewrite_model`, `rag_history_max_turns`, `rag_history_max_chars`, `history_only`) merged into the single `options` object the chat endpoints read |

Single options may also be passed as keyword arguments — the two calls below are
equivalent:

```python
client.search("my_docs", "query", SearchOptions(max_results=20, hybrid_search=True))
client.search("my_docs", "query", max_results=20, hybrid_search=True)
```

Unknown keys in search and generation dictionaries are preserved in `extra`;
unknown indexing keys raise `SSEValueError`. Per-call dictionaries override only
the supplied client defaults, while `IndexingOptions` instances supply a complete
configuration (including their dataclass defaults). Keyword arguments win over
both.

### Wire-format quirks handled by the client

The backend is not uniform, and these traps are the reason the wrapper exists:

* `options` of `search_with_options`, `query_options` of `generative_answer` and
  `indexing_options` of `upload_and_index_files` are read with `json.loads()`,
  so they are sent as **JSON strings**;
* `indexing_options` of `add_and_index_texts` and `options`/`search_options` of
  the chat endpoints are read as objects, so they are sent as **JSON objects**;
* uploaded files use the field name `files[]` and input texts `texts[]`;
* endpoint paths have **no trailing slash** (that is how Django registers them);
* business errors come back as HTTP 200 with `{"status": false, "errors": [...]}`.

## Return values

Methods return dataclasses (`sse_lib.models`): `Collection`, `SearchResponse`
(with `.hits` of `SearchHit`), `Answer`, `UploadResult`, `IndexingResult`,
`DocumentRef`, `Chat`, `ChatReply`, `ChatMessage`, `ChatHistory`. Every one of them keeps the
untouched server payload in `.raw`, and
`client.request("GET", "some_endpoint")` returns the unwrapped `body` of any
endpoint for fields that are not modelled yet.

The places where a citation leads back to the text are modelled directly:
`SearchHit.text_id` / `.document_id` and `.char_start` / `.char_end` (the
position of the chunk in the preview page, `None` when it cannot be
reproduced), `DocumentPreview.highlight` for the page returned by
`document_content(text_id=...)`, `Answer.citations` for the numbered sources of
an answer, and `ChatReply.citations` / `.rewritten_query` / `.answer_source`
(`"rag"`, `"history"` or `"llm_fallback"`) for a reply of a chat.

## Errors

| Exception | Raised when |
|-----------|-------------|
| `SSEValueError` | An argument is rejected before the request is sent (empty query, unknown index type, ...) |
| `SSEConfigError` | Invalid configuration (missing host, non-positive timeout, login without credentials) |
| `SSEAPIError` | The API answered `{"status": false, "errors": [...]}` — inspect `.error_names`, `.not_given_params` |
| `SSEAuthenticationError` | HTTP 401/403 |
| `SSEHTTPError` | Any other non-2xx status |
| `SSETransportError` | No response at all (DNS, connection refused, timeout) |

All of them derive from `SSEError`, so a single `except SSEError` is enough for
a coarse handler. Idempotent `GET` calls are retried on connection errors and
on 429/502/503/504; `POST` is never replayed, because indexing twice is a real
side effect.

## Development

```bash
cd sse_lib
PYTHONPATH=src python -m unittest discover -s tests -v   # no installation needed
python -m pytest .                                        # pytest, if available
black --check . && flake8 .
```

The tests never open a socket: `tests/support.py` replaces the HTTP session
with a recording double, which pins the exact payloads sent to the API.
