Metadata-Version: 2.4
Name: rail-engine
Version: 0.6.1
Summary: Python SDK for Railtown AI Railengine - Retrieval
Author-email: Railtown AI <support@railtown.ai>
License: MIT
Project-URL: Homepage, https://railengine.ai
Keywords: railtown,rail-engine,retrieval,sdk
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: certifi>=2026.10.19
Requires-Dist: httpx>=0.24.0
Requires-Dist: pydantic>=1.10.0
Dynamic: license-file

# Railengine Retrieval SDK

Python SDK for retrieving and searching data from Railtown AI Railengine - handles embeddings, storage documents, and indexed content.

## Overview

The `rail-engine` package provides a Pythonic interface for retrieving and searching data from Railengine. It supports async/await patterns, client-side filtering, explicit storage list paging via `StorageListPage`, and Pydantic model deserialization.

## Installation

```bash
uv pip install rail-engine
```

## Quick Start

```python
import asyncio
from railtown.engine import Railengine

async def main():
    # Initialize client (reads from ENGINE_PAT and ENGINE_ID env vars)
    async with Railengine() as client:
        page = await client.list_storage_documents(
            page_number=1,
            page_size=25,
        )
        print(f"total_count={page.total_count} total_pages={page.total_pages}")
        for doc in page.items:
            print(doc)

asyncio.run(main())
```

## Configuration

### Environment Variables

- `ENGINE_PAT` (required) - Personal Access Token
- `ENGINE_ID` (required) - Engine ID (can also be passed to constructor)
- `RAILTOWN_API_URL` (optional) - Base API URL (defaults to `https://cndr.railtown.ai/api`)

### Constructor Parameters

- `pat` (optional) - PAT token (if not provided, reads from `ENGINE_PAT` env)
- `engine_id` (optional) - Engine ID (if not provided, reads from `ENGINE_ID` env, required if not in env)
- `api_url` (optional) - Base API URL (if not provided, reads from `RAILTOWN_API_URL` env or defaults to production)
- `model` (optional) - Pydantic model type for deserializing retrieved data
- `timeout` (optional) - Request timeout in seconds or `httpx.Timeout` (default `30`). Applied to the SDK-created client and to each request, including vector search.
- `verify` (optional) - TLS verification. Defaults to `True`. Pass a CA bundle path or `ssl.SSLContext` for custom trust. `False` disables verification and must be set explicitly.
- `http_client` (optional) - Caller-owned `httpx.AsyncClient` for proxies, custom transports, or tests. The SDK does not close this client.
- `retrieval_error_mode` (optional) - `"raise"` (default) or `"fallback"`; see the error contract below.
- `logging_policy` (optional) - `LoggingPolicy` for SDK logs. Default logs operation metadata only. Search text, JSONPath, and response bodies are not logged unless `log_content=True`.

Earlier SDK versions disabled TLS verification. 0.4.0 verifies certificates by default. Use `verify=` with your CA bundle if the API host uses a private certificate.

## Logging and sensitive data

SDK loggers use the `railtown.engine.*` namespace. The host application controls logging handlers and levels.

- With the default `LoggingPolicy(log_content=False)`, SDK logs contain operation metadata such as engine id, status, and duration. Queries, request payloads, and response bodies are excluded, including at DEBUG. Raising the log level alone does not enable content logging.
- Explicitly setting `LoggingPolicy(log_content=True)` allows queries and request/response content in SDK logs. Request content can appear at INFO, and request/response content can appear at ERROR on failures; content logging is not restricted to DEBUG.
- Opt-in content logging applies secret redaction and truncates content to 512 characters by default. A custom `redactor` and `content_max_chars` can adjust this behavior. Redaction and truncation do not guarantee that all sensitive data is removed. Leave `log_content=False` in production.
- `httpx` logs independently of the SDK policy. Its INFO logs can include full URLs with query strings, which may contain sensitive queries. Set the `httpx` logger to WARNING in production:

```python
import logging

logging.getLogger("httpx").setLevel(logging.WARNING)
```

## Features

- **Multiple retrieval methods**:
  - `search_vector_store()` - Semantic search in vector stores
  - `search_vector_store_filtered()` - Semantic search restricted to the documents an index query matches (`search_query` takes the same `IndexingSearchQuery`/dict as `search_index`; requires Indexing and Embeddings enabled on the engine)
  - `get_storage_document_by_event_id()` - Get document by EventId
  - `get_storage_document_by_customer_key()` - Same paging as `list_storage_documents` (`page_number`, `page_size`) but scoped to a CustomerKey; returns `StorageListPage` with `items`, `total_pages`, and `total_count`
  - `query_storage_by_jsonpath()` - Query using JSONPath
  - `list_storage_documents()` - Use `page_number` and `page_size` to paginate; returns `StorageListPage` with `items`, `total_pages`, and `total_count`
  - `search_index()` - Index search (default `raw=True`: each hit is the API dict; use `raw=False` to parse each hit's `body` with `model` or the client default); returns `IndexingSearchResult` with `items` and optional `total_count`
  - `delete_document()` - Delete all document versions by customer key from storage, embeddings, and indexing (raises on API errors)
- **Client-side filtering on storage reads** - Optional `filter_fn` on storage/list/query helpers (not on `search_vector_store` or `search_vector_store_filtered`; filter vector hits in your loop)
- **Storage list pagination** - One HTTP GET per page; use `page_number` / `page_size` on the request and `total_pages` / `total_count` on `StorageListPage` when looping
- **Model deserialization** - Optional Pydantic model support with per-call override
- **Actionable error handling** - Retrieval failures raise by default; explicit fallback mode supports legacy integrations
- **Async/await support** - Built for modern async Python applications

## Vector search: document models and hit metadata

### Migration to 0.6.1

Version 0.6.1 intentionally changes both vector search methods. Every normal result
is a `VectorSearchResult`: `result.hit` contains the outer search metadata and
`result.document` contains the separately validated document. Document fields can
never replace the hit identifier, score, or other metadata.

```python
from pydantic import BaseModel
from railtown.engine import Railengine, RailtownSearchParsingError

class Food(BaseModel):
    name: str

async def search_documents():
    async with Railengine(model=Food) as client:
        try:
            results = await client.search_vector_store(query="apple")
            for result in results:
                assert result.document is not None
                print(result.hit.event_id, result.hit.score, result.document.name)
        except RailtownSearchParsingError as error:
            # Zero-based response row, outer event ID, metadata/document phase,
            # and a fixed failure category (no document values or field paths).
            print(error.row_index, error.event_id, error.phase, error.problem)
```

A per-call `model` overrides the client default. Omitting it or passing `model=None`
keeps the default. Models validate only the JSON in `hit.content`; there is no
whole-hit fallback. Invalid JSON or a model mismatch raises
`RailtownSearchParsingError`, including when `retrieval_error_mode="fallback"`.
One invalid hit fails the call: no partial list, skipped hit, or mixed dictionary
fallback is returned. Invalid outer hit metadata also raises this error.

Without any model, results still have typed `hit` metadata; `document` is `None`
and the original content remains available as `result.hit.content`. Pass a domain
model to parse it. Passing `VectorStoreSearchHit` as the document model only parses
a nested document of that shape; it never requests or replaces outer metadata.

To request unparsed hit dictionaries, explicitly pass `raw=True`. This bypasses
both per-call and client models, including document and hit-field validation.
Response envelope validation and HTTP error handling still apply.

```python
async def search_raw():
    async with Railengine(model=Food) as client:
        rows = await client.search_vector_store(query="apple", raw=True)
        for row in rows:
            print(row["eventId"], row["score"], row["content"])
```

Upgrade existing callers as follows:

- Replace dictionary access such as `row["eventId"]` with `result.hit.event_id`,
  or explicitly select `raw=True` to retain dictionary access.
- Replace direct document access such as `item.name` with `result.document.name`.
- Replace `model=VectorStoreSearchHit` used for metadata with no document model
  and read `result.hit`. With a configured default, use raw mode if parsing is unwanted.
- Catch `RailtownSearchParsingError` and correct invalid documents or choose raw
  mode deliberately. Do not rely on mixed result types or whole-hit fallback.
- Apply the same migration to `search_vector_store_filtered`; its `search_query`
  and ranking behavior are unchanged.

This patch release includes breaking vector search changes. Upgrade and test callers
before moving to 0.6.1; pin `<0.6.1` temporarily if migration must wait.
There is no implicit legacy parsing mode. Storage and index parsing contracts are unchanged.

## Error Handling

The SDK provides custom exception classes:

- `RailtownSearchParsingError` - Invalid vector hit metadata or document content
- `RailtownError` - Base exception
- `RailtownBadRequestError` - 400 Bad Request
- `RailtownUnauthorizedError` - 401 Unauthorized
- `RailtownNotFoundError` - 404 Not Found
- `RailtownConflictError` - 409 Conflict
- `RailtownServerError` - 5xx Server errors

### Error contract and migration to 0.5.0

Version 0.5.0 changes retrieval failures to **raise by default**. Successful empty queries keep their existing result types and pagination metadata. To retain the previous retrieval fallback behavior, explicitly construct `Railengine(retrieval_error_mode="fallback")`. Fallback logs the failure and returns an empty list, empty result/page, or `None` for an event lookup. It does not suppress ingestion or deletion failures, invalid arguments, cancellation, or application callback errors. Storage and index model parsing are unchanged. Vector search uses the strict 0.6.1 contract above.

A 404 raises `RailtownNotFoundError` by default, including event lookups. The current API contract does not establish a reliable discriminator between a missing document and another missing resource; the SDK does not infer which one is absent or perform another request to guess.

Catch the common base class for all request/response errors:

```python
from railtown.engine import Railengine, RailtownError

async def search():
    async with Railengine() as client:
        try:
            return await client.search_index(query={"search": "example"})
        except RailtownError as error:
            print(error.operation, error.status_code, error.request_id)
            raise
```

HTTP 400, 401, 403, 404, 409, and 429 map to `RailtownBadRequestError`, `RailtownUnauthorizedError`, `RailtownForbiddenError`, `RailtownNotFoundError`, `RailtownConflictError`, and `RailtownRateLimitError`. Other 4xx responses use `RailtownError`; 5xx responses use `RailtownServerError`. Transport failures use `RailtownTransportError`, timeouts its subclass `RailtownTimeoutError`, and invalid response JSON/envelopes use `RailtownResponseError`. For retrieval (`rail-engine`), import these exceptions from `railtown.engine`. For ingestion (`rail-engine-ingest`), import them from `railtown.engine.ingest`. The `railtown.engine.ingest` namespace requires `rail-engine-ingest`; it is not included in the retrieval wheel.

Request/response errors expose `status_code` when available, `operation`, the effective `engine_id`, `request_id` when available, and `retryable`. Retryable is true for timeouts, connection failures, and HTTP 408, 429, 502, 503, and 504. It describes a potentially transient failure, not whether replaying a write is safe. The SDK does not retry automatically.

**Diagnostic compatibility change:** SDK-generated errors now leave `response_text` as `None`. The constructor still accepts an explicitly supplied value. Messages exclude query strings and payloads, but the preserved original exception cause can retain sensitive data. Full traceback logging is not guaranteed to be sanitized. Explicit content logging remains governed by `LoggingPolicy` and does not populate error attributes.

## Requirements

- uv
- Python 3.11+
- httpx >= 0.24.0
- pydantic >= 1.10.0

## Related package

For ingesting data into Railengine, install and use [`rail-engine-ingest`](https://pypi.org/project/rail-engine-ingest/).

## License

MIT

### Document deletion migration (0.6.0)

`delete_event` has been removed. Use `await client.delete_document(customer_key, engine_id=None)` with the original customer key, not an event ID. This deletes all existing versions. Requires the updated Document API and a ProjectReadAdmin PAT. Coordinate API and SDK releases.

202 indicates an index deletion was accepted. 204 indicates storage deletion succeeded but may include index failure. Index cleanup is best effort and eventually consistent. In-flight processing or later ingestion can recreate data. API and transport errors raise SDK exceptions.
