Metadata-Version: 2.4
Name: dataoil
Version: 0.1.5
Summary: The Data Oil client for Python: one statement over rows, vectors, keywords and relationships, streamed as NDJSON, with the two tenancy headers, typed errors and the retry a lost race needs built in. Standard library only.
Author-email: MaxDragon Technology <info@dataoil.tech>
License-Expression: LicenseRef-DataOil-Proprietary
Project-URL: Documentation, https://documents.dataoil.tech/tenant/
Keywords: dataoil,database,sql,vector,search,graph,ndjson,retriever
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Classifier: Intended Audience :: Developers
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: langchain
Requires-Dist: langchain-core>=1.3.3; extra == "langchain"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.13.0; extra == "llamaindex"
Provides-Extra: test
Requires-Dist: pytest>=9.0.3; extra == "test"
Dynamic: license-file

# dataoil

The Data Oil client for Python. One statement over rows, vectors, keywords and relationships,
streamed as NDJSON, with the two tenancy headers, typed errors and the retry a lost race needs
built in. Standard library only; Python 3.10+.

```
pip install dataoil
pip install "dataoil[langchain]"     # + a LangChain retriever
pip install "dataoil[llamaindex]"    # + a LlamaIndex retriever
```

## Sixty seconds

```python
from dataoil import DataOilClient

client = DataOilClient("https://acme.dataoil.tech", "bookshop", username="your-account", password="...")
# or, with an API key:  DataOilClient("https://acme.dataoil.tech", "bookshop", token="dot_...")

client.statement("INSERT INTO Book SET isbn = :isbn, title = :title", {"isbn": "978-py", "title": "From Python"})

for row in client.statement("SELECT isbn, title FROM Book ORDER BY isbn"):
    print(row["isbn"], row["title"])

# A large scan streams, and follows the cursor across pages.
for row in client.query("SELECT FROM Report WHERE filedAt > :since", {"since": "2026-01-01"}, page_size=2000):
    handle(row)

# One statement over keywords, meaning and a column.
hits = client.statement("""
    SELECT title, VECTOR_DISTANCE(embedding, EMBED(:q), 'cosine') AS distance, $score AS keywordScore
    FROM Report
    WHERE SEARCH_INDEX('Report[body]', :q) = true AND filedAt > date('2026-01-01')
    ORDER BY distance LIMIT 10
""", {"q": "casing failure"})
```

## What it does for you

| | |
| --- | --- |
| **The two headers** | `X-Job-ID` and `X-Database` on every call. The engine refuses a call without either, one at a time. `client.with_job("nightly-42")` names a batch. |
| **NDJSON, streamed** | `query()` yields rows as they arrive and sends the cursor back for the next page. The trailer line is never a row. `statement()` reads one page into a list and raises `IncompleteResultError` rather than pretend a page that did not finish was the whole answer. |
| **Typed errors** | Every refusal is a `DataOilError` with `status`, `error_code` (the stable code, e.g. `TypeNotFound`, `RequestQuotaExceeded`), `detail`, `correlation_id`, `arguments` (a quota's `used`, `allowed`, `resetsAt`) and `retryable`. Four subclasses: `DataOilAuthorizationError` (401/403), `DataOilNotFoundError` (404), `DataOilConflictError` (409/412), `DataOilQuotaError` (429/507, with `retry_after`). |
| **The 409 retry** | A lost race is answered `409` with `X-Retryable: true`; nothing was written, and the same request a moment later succeeds. It is retried with full jitter on its own budget (`RetryPolicy(attempts=5, base_delay=0.1, max_delay=5.0)`). A duplicate key (`X-Retryable: false`) is never retried. `429` waits what `Retry-After` says when that is inside the budget; `503` is retried like a race. Nothing else is. |

## Retrievers: the AI project that does not start with an export

```python
from dataoil import DataOilClient, Search
from dataoil.langchain import DataOilRetriever          # or dataoil.llamaindex

client = DataOilClient("https://acme.dataoil.tech", "kb", token="dot_...")
search = Search(client, "customer_docs", embed=my_embedding_model)   # the same model the ingestion used
retriever = DataOilRetriever(search=search)

docs = retriever.invoke("what changed in the March release?")
```

`Search` asks the database for the passages nearest a question: by keyword through the full-text
index (`SEARCH_INDEX`, ranked by BM25), by meaning through the vector index (`vector.neighbors`),
or both fused by rank (`mode="hybrid"`, the default when an `embed` function is given). Every hit
carries the record it came from, so a chain cites the row rather than a copy of it - and every
fence the engine applies to a statement applies here, because a retrieval *is* a statement.
Nothing is copied out.

## The rest of the surface

```python
record = client.create("Book", {"isbn": "1", "title": "A"})     # {rid, type, properties}
client.get(record["rid"]); client.delete(record["rid"])
client.batch([BatchOperation("INSERT INTO Book SET isbn = :i", {"i": "1"}), BatchOperation("INSERT INTO Book SET isbn = :i", {"i": "2"})])
tx = client.begin(); client.statement("INSERT INTO Book SET isbn = '3'", transaction=tx); client.commit(tx)   # or rollback
client.schema(); client.schema_narrative()
client.mint_token("the reporting job", rights=["read"], lifetime_seconds=30 * 86400)
client.ask("which reports mention casing?", rows, statement=sql)     # the tenant's own model, sent only the rows you read
client.request("GET", "/api/v2/databases/bookshop/statistics")
```

`ca_file=` names a certificate to trust for a deployment on its own authority.

## Tests

`pytest` runs 28 checks against a stub that answers exactly what the engine answers - NDJSON with
its trailer, problem documents with `X-Retryable`, a 429 with `Retry-After`. Set `DATAOIL_URL`,
`DATAOIL_DATABASE` and a credential (and `DATAOIL_CA` for a self-signed deployment) and one more
creates a type, writes, streams, searches, refuses and drops against a running engine.

The reference manual is at <https://documents.dataoil.tech/tenant/>.

## Source and licensing

Questions, issues and releases: <https://documents.dataoil.tech/tenant/> and info@dataoil.tech. The packages are
published to the public registries as compiled packages; their source is not published.

This library is proprietary and licensed only for legitimate use with Data Oil, under the Data Oil Software
Licence (see `LICENSE`): copying, modifying, redistributing or reverse engineering it, and any attack on
or unauthorised access to the platform, are prohibited. Data Oil itself is a commercial service of MaxDragon
Technology, and using it needs a credential issued under the Data Oil customer agreement. The
library grants no right to the service, and the service applies its own rights, budgets and audit
to every credential. The `NOTICE` file in the package has the full statement, the trademark note
and the third-party components.
