Metadata-Version: 2.5
Name: sophons
Version: 0.1.0
Summary: Agent memory and retrieval SDK
Requires-Python: >=3.11
Requires-Dist: lazy-loader>=0.4
Requires-Dist: opentelemetry-api>=1.26
Requires-Dist: pydantic>=2.0
Provides-Extra: all
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: boto3>=1.35; extra == 'all'
Requires-Dist: chromadb>=1.0; extra == 'all'
Requires-Dist: numpy>=1.26; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.26; extra == 'all'
Requires-Dist: opentelemetry-sdk>=1.26; extra == 'all'
Requires-Dist: prompt-toolkit>=3; extra == 'all'
Requires-Dist: pydantic>=2.0; extra == 'all'
Requires-Dist: rich>=13; extra == 'all'
Requires-Dist: sentence-transformers>=3.0; extra == 'all'
Requires-Dist: tavily-python>=0.3; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: bedrock
Requires-Dist: boto3>=1.35; extra == 'bedrock'
Provides-Extra: chroma
Requires-Dist: chromadb>=1.0; extra == 'chroma'
Provides-Extra: cli
Requires-Dist: prompt-toolkit>=3; extra == 'cli'
Requires-Dist: rich>=13; extra == 'cli'
Provides-Extra: deepseek
Requires-Dist: openai>=1.0; extra == 'deepseek'
Provides-Extra: docling
Requires-Dist: docling>=2.50; extra == 'docling'
Provides-Extra: numpy
Requires-Dist: numpy>=1.26; extra == 'numpy'
Provides-Extra: ocr
Requires-Dist: pillow>=10; extra == 'ocr'
Requires-Dist: pypdf>=4; extra == 'ocr'
Requires-Dist: pypdfium2>=4; extra == 'ocr'
Requires-Dist: pytesseract>=0.3.10; extra == 'ocr'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.26; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.26; extra == 'otel'
Provides-Extra: pdf
Requires-Dist: pypdf>=4; extra == 'pdf'
Provides-Extra: pdf-structure
Requires-Dist: pymupdf4llm>=0.3; extra == 'pdf-structure'
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers>=3.0; extra == 'sentence-transformers'
Provides-Extra: tavily
Requires-Dist: tavily-python>=0.3; extra == 'tavily'
Description-Content-Type: text/markdown

# Sophons

Sophons is a Python SDK for agent memory, retrieval, and RAG.

The goal is to provide small, reusable building blocks that can be used across different agent SDKs. Sophons should not be tied to one framework, one model provider, or one memory implementation.

## Status

Sophons is early-stage. The current repo contains the package skeleton, shared data types, and base interfaces for the first SDK layers.

Implemented so far:

- `Document`
- `Message`
- `Retriever` / `AsyncRetriever`
- `Loader` / `AsyncLoader`
- `Splitter`
- `Tool` / `AsyncTool`
- `BM25Retriever`, `SemanticRetriever`, `HybridRetriever`, `MultiQueryRetriever`

## What Sophons Provides

- Short-term conversation memory
- Long-term memory
- Generic retrievers for RAG, memory, and code/docs search
- RAG workflows built on retrievers
- Agent-callable tools
- Integrations with agent SDKs

## Package Shape

```text
sophons/
  documents/       shared document schema
  models/          messages and model interfaces
  loaders/         source -> documents
  splitters/       documents -> chunks
  retrieval/       generic retrievers: vector, BM25, hybrid, rerank
  rag/             RAG workflows built on retrievers
  memory/          short-term and long-term memory
  tools/           expose retrievers and memory as agent tools
  integrations/    adapters for agent SDKs
```

## Install For Local Development

```bash
uv sync --group dev
```

Run tests:

```bash
uv run python -m pytest
```

Install an optional embedding provider when needed:

```bash
uv sync --group dev --extra openai
# or
uv sync --group dev --extra sentence-transformers
```

## Core Types

```python
from sophons import Document, Message

document = Document(
    id="auth.md#chunk_1",
    content="Token refresh happens every 60 minutes.",
    metadata={"source": "auth.md"},
)

message = Message(
    role="user",
    content="How does token refresh work?",
)
```

`Document` is for knowledge/context.

`Message` is for model and agent conversation.

## Embeddings

`sophons.embeddings` is the stable public embedding API. Core protocols are
always available, while provider integrations and their SDKs are loaded only
when requested.

```python
from sophons.embeddings import EmbeddingModel, OpenAIEmbeddings

embedder: EmbeddingModel = OpenAIEmbeddings(
    api_key="...",
    model="text-embedding-3-small",
)

query_vector = embedder.embed_query("How does token refresh work?")
document_vectors = embedder.embed_documents([
    "Token refresh happens every 60 minutes.",
    "Sessions expire after 24 hours.",
])
```

For asynchronous applications:

```python
from sophons.embeddings import AsyncOpenAIEmbeddings

embedder = AsyncOpenAIEmbeddings(api_key="...")
query_vector = await embedder.embed_query("How does token refresh work?")
```

Applications should import from `sophons.embeddings`, not from internal
`sophons.integrations` modules. Provider packages remain optional: importing
the public namespace does not import OpenAI or Sentence Transformers.

Sentence Transformers can return L2-normalized vectors when the consuming
application uses cosine similarity or dot product over unit vectors:

```python
from sophons.embeddings import SentenceTransformerEmbeddings

embedder = SentenceTransformerEmbeddings(
    model="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    normalize=True,
)
```

Normalization is explicit and disabled by default so Sophons does not impose a
similarity policy on every consumer.

## Models

`sophons.models` is also the stable public API for chat-model contracts,
messages, settings, and provider implementations. Provider SDKs are loaded only
when their model is constructed.

Install DeepSeek support:

```bash
uv sync --group dev --extra deepseek
```

Then create the model through the public API:

```python
from sophons.models import DeepSeekModel, Message

model = DeepSeekModel(
    model="deepseek-chat",
    api_key="...",
)

response = model.invoke([
    Message(role="user", content="Explain Article 43."),
])
```

The provider-independent `ChatModel` protocol remains available from the same
namespace:

```python
from sophons.models import ChatModel, DeepSeekModel

model: ChatModel = DeepSeekModel(
    model="deepseek-chat",
    api_key="...",
)
```

Applications should prefer `sophons.models` over internal
`sophons.integrations.models` imports.

## Vector Stores

`sophons.stores` is the stable public API for vector-store contracts and
implementations. The in-memory store has no optional dependency:

```python
from sophons.stores import InMemoryVectorStore, VectorStore

store: VectorStore = InMemoryVectorStore()
```

Chroma remains optional and is loaded only when constructed:

```bash
uv sync --extra chroma
```

```python
from sophons.stores import ChromaVectorStore

store = ChromaVectorStore(
    collection="katiba",
    path="./katiba_index",
)
```

Applications should prefer `sophons.stores` over internal
`sophons.integrations.vector_stores` imports.

For exact cosine search over an existing NumPy matrix:

```bash
uv sync --extra numpy
```

```python
from sophons.stores import NumPyVectorStore

store = NumPyVectorStore()
store.add_matrix(documents, vectors)
results = store.search(query_vector, limit=10)
```

`NumPyVectorStore` keeps vectors as a `float32` matrix and performs bulk cosine
search without converting the matrix to nested Python lists.

## Retriever Pattern

Retrievers follow one simple contract:

```text
query -> documents
```

That means the same retriever can be used in different places:

- A RAG app can retrieve documents before answering.
- A memory system can retrieve relevant user memories.
- An agent can expose a retriever as a tool.
- A framework adapter can wrap the retriever for OpenAI, Strands, LangChain-style APIs, or another SDK.

The first retriever layer includes the common foundation:

- Vector retrieval
- BM25 lexical retrieval
- Hybrid retrieval
- Multi-query retrieval

More advanced retrievers can wrap or combine those foundations:

- Contextual compression
- Self-query retrieval
- Reranking
- Routing
- Ensemble retrieval

## Loader And Splitter Pattern

Loaders ingest external sources:

```text
source -> documents
```

Splitters prepare documents for indexing and retrieval:

```text
documents -> chunks
```

Together:

```text
source -> Loader -> Documents -> Splitter -> Chunks -> Retriever
```

To watch a run as it happens, stream it:

```python
from sophons.agents import Agent
from sophons.models import TextDelta

async for event in Agent(model=model, tools=tools).stream("What is 17 plus 25?"):
    if isinstance(event, TextDelta):
        print(event.text, end="")
```

`stream()` yields model events (text, tool-call fragments, usage), tool start
and finish events, and finally the same result `run()` returns. A model without
native streaming is delivered as a single piece. Streamed text has not been
through output validation; treat it as provisional until the result arrives.

To read a file as structured elements instead of flat text, parse it:

```python
from sophons.parsers import Blob, default_registry
from sophons.splitters import StructureAwareSplitter

parsed = default_registry().parse(Blob.from_path("contract.docx"))
for element in parsed.leaves():
    print(element.kind, parsed.text_of(element)[:60])
chunks = StructureAwareSplitter(max_chunk_size=1000).split_parsed(parsed)
```

A parser takes bytes and returns one `ParsedDocument`: the extracted text plus
typed elements (headings, paragraphs, list items, tables with rows and cells)
recorded as offsets into that text. Built-in parsers cover plain text, Markdown,
PDF text layers (`sophons[pdf]`) and Word. Chunks keep their offsets, pages and
heading path, so a quotation can be traced to its place in the source.

Scanned PDFs and images are read with `sophons[ocr]` (Tesseract). For layout
analysis that finds unnumbered headings, lists and tables, install
`sophons[docling]` and use `default_registry(layout=True)`. Either way,
`clean(parsed)` then removes page numbers and running headers from the content,
separates numbered clauses, sets heading depth from numbering and rejoins
sentences split by a page break, without altering the extracted text.

For structure-aware PDFs, install `sophons[pdf-structure]` and use:

```python
from sophons.loaders import PDFStructureLoader
from sophons.splitters import StructureAwareSplitter

document = PDFStructureLoader("contract.pdf").load()[0]
chunks = StructureAwareSplitter(max_chunk_size=1000).split(document)
```

The loader converts PDF layout to Markdown with PyMuPDF4LLM. The splitter
preserves detected sections and blocks, with source page metadata. See
[the walkthrough](docs/structure_chunking.md) for inspecting intermediate
structure, OCR options, and limitations.

## Tool Pattern

Tools follow a simple agent-facing contract:

```text
structured args -> structured result
```

Tools are how agents will call Sophons capabilities such as memory search, document retrieval, or RAG context retrieval.

## Roadmap

Next small implementation steps:

1. Text loading and recursive splitting
2. BM25 lexical retrieval
3. DeepSeek chat integration
4. Basic BM25-based RAG pipeline
5. Hugging Face embeddings and vector retrieval
