Metadata-Version: 2.5
Name: langchain-xns
Version: 0.1.0
Summary: LangChain document loaders and byte store for XNS — S3-compatible object storage with $0 egress
Project-URL: Homepage, https://xns.tech
Project-URL: Documentation, https://github.com/xns-cloud/langchain-xns#readme
Project-URL: Source, https://github.com/xns-cloud/langchain-xns
Project-URL: Issues, https://github.com/xns-cloud/langchain-xns/issues
Author-email: XNS Cloud Services <support@xns.tech>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: bytestore,document-loader,langchain,object-storage,rag,s3,xns,zero-egress
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Archiving :: Backup
Requires-Python: >=3.10
Requires-Dist: boto3>=1.34
Requires-Dist: langchain-core<2,>=0.3
Provides-Extra: test
Requires-Dist: langchain-classic<2,>=1; extra == 'test'
Requires-Dist: langchain-text-splitters>=0.3; extra == 'test'
Requires-Dist: numpy>=1.26; extra == 'test'
Requires-Dist: pytest-asyncio>=0.24; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# langchain-xns

LangChain document loaders and a `ByteStore` for **XNS** — S3-compatible, distributed object
storage with **$0 egress**. Reading your data back is free, at any volume, so the retrieval
side of a RAG or agent pipeline costs nothing to run repeatedly.

```bash
pip install langchain-xns
```

```python
from langchain_xns import XNSLoader

docs = XNSLoader("corporate-strategy", suffixes=(".md",)).load()
```

That is the whole setup. No endpoint, no keys — see [Zero-config](#zero-config-credentials).

## Why not the generic S3 loader

You *can* point a generic S3 loader at XNS by overriding `endpoint_url`. This package exists
because three things then still fall to you:

- **Credentials.** XNS writes `~/.xns/credentials` during setup. This package reads it, so
  nothing has to be threaded through your app config or environment.
- **Addressing.** A self-hosted gateway is often reached by host:port during bring-up, where
  virtual-hosted addressing cannot work. Path-style is the default here.
- **Cost of a bad list.** Filtering happens on the `ListObjectsV2` response, so a key excluded
  by `suffixes`, `glob`, or `max_bytes` is never downloaded at all.

## Zero-config credentials

XNS has no shared hostname and no vendor API key. Every customer runs their own **Relayer**
(the S3 gateway) and reaches it at a domain they control, with their own TLS certificate. A
connection is therefore always *endpoint + access key + secret*.

Configuration is resolved per field, highest wins:

1. Arguments you pass to the loader or store.
2. `XNS_ENDPOINT`, `XNS_ACCESS_KEY_ID`, `XNS_SECRET_ACCESS_KEY`, `XNS_REGION`, `XNS_PROFILE`.
3. `~/.xns/credentials` — written by the XNS MCP server or by `xns auth login`.
   Override the path with `XNS_CREDENTIALS_FILE`, pick a profile with `profile=`.

Step 3 is why the example above takes no arguments. If you have not set XNS up yet, an agent
can do the whole install for you:

```bash
claude mcp add relayer -- npx @xns-cloud/relayer-mcp@latest
```

Then ask it to "set up XNS storage". It checks prerequisites, registers the account, installs
and starts the Relayer on your Docker host, provisions S3 credentials, and writes the file this
package reads.

## `XNSLoader` — text objects as Documents

```python
from langchain_xns import XNSLoader

loader = XNSLoader(
    "corporate-strategy",
    prefix="2026/",              # pushed to the server, not filtered client-side
    suffixes=(".md", ".txt"),
    glob="*/notes/*",            # optional fnmatch over the full key
)

for doc in loader.lazy_load():   # streams; never lists the whole bucket into memory
    print(doc.metadata["source"], len(doc.page_content))
```

Bodies are decoded incrementally as they stream, so a multi-byte character split across a chunk
boundary is handled correctly and the object is never held in memory twice.

Each `Document` carries `id="s3://bucket/key"` — stable across re-indexing, so vector stores
that support upsert will replace rather than duplicate.

| Metadata key | Value |
|---|---|
| `source` | `s3://bucket/key` |
| `bucket`, `key` | as given |
| `size` | bytes, from the listing |
| `etag` | quotes stripped |
| `last_modified` | ISO 8601 |
| `content_type` | as stored |

Pass `metadata_fn=lambda meta, entry: {...}` to add your own.

**Large objects.** `max_bytes` defaults to 64 MiB. Over that, `XNSLoader` raises
`XNSObjectTooLargeError` *before* fetching the body; pass `on_oversize="skip"` to ignore them,
or `max_bytes=None` to disable the check. A text document that trips this is usually a media
file — use `XNSBlobLoader` instead.

## `XNSBlobLoader` — binary objects for parsers

```python
from langchain_xns import XNSBlobLoader
from langchain_community.document_loaders.parsers import PyPDFParser

for blob in XNSBlobLoader("research", suffixes=(".pdf",)).yield_blobs():
    docs = list(PyPDFParser().lazy_parse(blob))
```

Yields `langchain_core` `Blob` objects with `mimetype` set from the object's `Content-Type`, so
bytes reach a parser without a decode step.

## `XNSByteStore` — a ByteStore for caches and multi-vector retrieval

```python
from langchain_classic.embeddings import CacheBackedEmbeddings
from langchain_xns import XNSByteStore

store = XNSByteStore("agent-cache", prefix="embeddings/")
embedder = CacheBackedEmbeddings.from_bytes_store(
    underlying_embeddings, store, namespace="text-embedding-3-small", key_encoder="sha256"
)
```

As of LangChain 1.x, `CacheBackedEmbeddings` ships in `langchain-classic`, not `langchain`.
`key_encoder` defaults to SHA-1 and warns; pass `"sha256"` to silence it.

It also works as the document store behind a retriever:

```python
from langchain_classic.retrievers.multi_vector import MultiVectorRetriever

retriever = MultiVectorRetriever(
    vectorstore=vectorstore,
    byte_store=XNSByteStore("parents", prefix="mv/"),
    id_key="doc_id",
)
```

Implements the full `BaseStore[str, bytes]` contract — `mget`, `mset`, `mdelete`, `yield_keys`.
`mget` returns `None` for absent keys (it does not raise); other errors propagate. `mdelete`
batches at S3's 1000-key limit. `prefix` namespaces a store so several can share one bucket.

Because reads are unmetered on XNS, there is no cost argument for stacking a local cache in
front of this store — the usual reason to do that is egress billing.

## Multimodal and checkpoint workloads

The pipelines this is built for re-read the same bytes many times: a video re-processed into
transcription and then embedding models, a checkpoint pulled to a fresh GPU host on every run,
an eval suite replayed against a growing corpus. On egress-billed storage, each pass is charged.
On XNS it is not, which is why `XNSByteStore` deliberately has no cache layer and `XNSLoader`
re-reads rather than mirroring to local disk.

## Compatibility

- Python 3.10+
- `langchain-core >= 0.3, < 2` — uses `BaseLoader`, `BlobLoader`, and `BaseStore` only
- `boto3 >= 1.34`
- XNS Relayer, or any S3-compatible endpoint. Both path-style (default) and virtual-hosted
  addressing are supported; pass `addressing_style="virtual"` for the latter.

XNS is validated against the ceph/s3-tests conformance suite on real hardware; per-capability
results are published at <https://xns.tech/s3-compatibility>.

## Links

- XNS — <https://xns.tech>
- Set up with an AI agent — <https://xns.tech/setup-with-ai>
- S3 compatibility matrix — <https://xns.tech/s3-compatibility>
- MCP server — [`@xns-cloud/relayer-mcp`](https://github.com/xns-cloud/relayer-mcp)

## License

Apache-2.0
