Metadata-Version: 2.4
Name: pyds-ai
Version: 0.2.0
Summary: Data science and LLM work from a notebook, without writing much code. The LLM writes the query and the words; a deterministic engine computes the answer.
Project-URL: Homepage, https://github.com/pyds-ai/pyds-ai
Project-URL: Documentation, https://github.com/pyds-ai/pyds-ai#readme
Project-URL: Issues, https://github.com/pyds-ai/pyds-ai/issues
Author: pyds-ai contributors
License: Apache-2.0
License-File: LICENSE
Keywords: agents,automl,data science,duckdb,llm,notebook,pandas,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Jupyter
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: duckdb>=0.10
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: openpyxl>=3.1
Requires-Dist: pandas>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: tomli-w>=1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: all
Requires-Dist: anthropic>=0.34; extra == 'all'
Requires-Dist: beautifulsoup4>=4.12; extra == 'all'
Requires-Dist: chromadb>=0.5; extra == 'all'
Requires-Dist: google-genai>=0.3; extra == 'all'
Requires-Dist: joblib>=1.3; extra == 'all'
Requires-Dist: keyring>=24; extra == 'all'
Requires-Dist: markdown-it-py>=3.0; extra == 'all'
Requires-Dist: openai>=1.40; extra == 'all'
Requires-Dist: pdfplumber>=0.11; extra == 'all'
Requires-Dist: pypdf>=4.0; extra == 'all'
Requires-Dist: python-docx>=1.1; extra == 'all'
Requires-Dist: python-pptx>=0.6.23; extra == 'all'
Requires-Dist: scikit-learn>=1.3; extra == 'all'
Requires-Dist: shap>=0.44; extra == 'all'
Requires-Dist: sqlalchemy>=2.0; extra == 'all'
Requires-Dist: weasyprint>=61; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.34; extra == 'anthropic'
Provides-Extra: db
Requires-Dist: sqlalchemy>=2.0; extra == 'db'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.25; extra == 'docs'
Provides-Extra: gemini
Requires-Dist: google-genai>=0.3; extra == 'gemini'
Provides-Extra: keyring
Requires-Dist: keyring>=24; extra == 'keyring'
Provides-Extra: ml
Requires-Dist: joblib>=1.3; extra == 'ml'
Requires-Dist: scikit-learn>=1.3; extra == 'ml'
Requires-Dist: shap>=0.44; extra == 'ml'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Provides-Extra: pdf-tables
Requires-Dist: pdfplumber>=0.11; extra == 'pdf-tables'
Provides-Extra: rag
Requires-Dist: beautifulsoup4>=4.12; extra == 'rag'
Requires-Dist: markdown-it-py>=3.0; extra == 'rag'
Requires-Dist: pypdf>=4.0; extra == 'rag'
Requires-Dist: python-docx>=1.1; extra == 'rag'
Provides-Extra: rag-chroma
Requires-Dist: chromadb>=0.5; extra == 'rag-chroma'
Provides-Extra: rag-faiss
Requires-Dist: faiss-cpu>=1.8; extra == 'rag-faiss'
Provides-Extra: rag-pgvector
Requires-Dist: pgvector>=0.2; extra == 'rag-pgvector'
Requires-Dist: psycopg[binary]>=3.1; extra == 'rag-pgvector'
Provides-Extra: report
Requires-Dist: python-docx>=1.1; extra == 'report'
Requires-Dist: python-pptx>=0.6.23; extra == 'report'
Requires-Dist: weasyprint>=61; extra == 'report'
Provides-Extra: test
Requires-Dist: ipykernel>=6.29; extra == 'test'
Requires-Dist: nbmake>=1.5; extra == 'test'
Requires-Dist: pytest-cov>=5.0; extra == 'test'
Requires-Dist: pytest>=8.0; extra == 'test'
Requires-Dist: reportlab>=4.0; extra == 'test'
Description-Content-Type: text/markdown

# pyds-ai

[![CI](https://github.com/pyds-ai/pyds-ai/actions/workflows/ci.yml/badge.svg)](https://github.com/pyds-ai/pyds-ai/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/pyds-ai/pyds-ai/branch/main/graph/badge.svg)](https://codecov.io/gh/pyds-ai/pyds-ai)
[![PyPI](https://img.shields.io/pypi/v/pyds-ai.svg)](https://pypi.org/project/pyds-ai/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

Data science and LLM work from a notebook, without writing much code.

```bash
pip install pyds-ai
```
```python
import pyds_ai as pyds
```
`pyds-ai` is the package name; `pyds_ai` is the import — they don't
match on purpose, because the short names `pyds` and `pyds-cli` are
already taken by unrelated packages on PyPI. Every example on this page
shows both lines together so this is never a guessing game.

**Your data doesn't leave the building by default.** The model gets
column names, types, and summary statistics — never your raw rows —
unless you explicitly turn `send_data_values` on. See
[Privacy default](#privacy-default) below.

<p align="center">
  <img src="https://raw.githubusercontent.com/pyds-ai/pyds-ai/main/docs/images/result-card.png" alt="A real d.drivers() result card, rendered in a notebook" width="640">
  <br><em>A real result card — the finding, the chart, and (expanded) the data, code, and trace.</em>
</p>
<p align="center">
  <img src="https://raw.githubusercontent.com/pyds-ai/pyds-ai/main/docs/images/failure-card.png" alt="A real Failure card" width="640">
  <br><em>A failed call renders as a card, not a traceback.</em>
</p>

**The one rule that decides everything: the LLM never produces a number.
It writes the query and it writes the words. A deterministic engine
(DuckDB) computes the answer.** Every result can show you the exact code
that produced it via `result.code()`.

## Try it with no data and no key

```bash
pip install pyds-ai
```
```python
import pyds_ai as pyds

pyds.demo()  # lists the bundled datasets
pyds.demo("product", offline=True)  # a real card, 5 canned answers, no key needed
```

`pyds.demo("product")` / `pyds.demo("revenue")` load the same bundled
datasets as a normal `DataSet` — everything below works on them exactly
as it would on your own data, once you're ready for `pyds.setup()`.

## Quickstart

```bash
pip install pyds-ai[anthropic]  # or [openai], [gemini] -- Ollama needs no extra
```
```python
import pyds_ai as pyds

pyds.setup()  # one-time key wizard, writes to ~/.pyds_ai/config.toml

d = pyds.load("sales.csv")
d.suggest()  # ten business-language questions this data can answer — no model call
d.ask("which region grew fastest last quarter?")
d.chart("revenue by month, split by channel")

m = d.model(target="churn")
m.explain()
m.what_if(tenure=12, plan="premium")

k = pyds.knowledge("policies/")
k.ask("what is our refund window for enterprise?")

a = pyds.agent(uses=[d, k])
a.ask("did churn spike in any region where we changed the refund policy?")

e = pyds.evaluate(a, cases="qa_cases.csv")
e.watch()
```

`d.suggest()` generates its ten questions from the shape of the data
alone (dates, categories, missing values) — no model call — and the top
three already show up on `d`'s own card the moment it loads, before you
ask anything.

## Real-world files, not clean CSVs

Data reaches this audience as an Excel export from someone else's system
— a merged title row, blank rows, the real header on row 4, six tabs
with the actual data buried on the fifth one, dates that mean something
different depending on where the file came from:

```python
pyds.sheets("book.xlsx")  # lists every sheet's size, no loading
d = pyds.load("book.xlsx")  # finds the real header row automatically,
# loads the sheet that looks like the data,
# and says exactly what it assumed
d2 = pyds.load_clipboard()  # pasted a range from Excel? this reads it directly
d3 = pyds.load("export.txt")  # sniffs the delimiter (comma/tab/pipe/semicolon)
pyds.pdf_tables("report.pdf")  # lists every table pdfplumber finds, before you load one
d4 = pyds.load("report.pdf")  # loads the first table found; table=1 picks another
```
`d.caveats` (shown right on the card) says which row it used as the
header, how many sheets the workbook had and which one it loaded, and —
for a date column like `01/02/2026` that's genuinely ambiguous — that it
assumed month-first (US) and how to say otherwise: `pyds.load("book.xlsx", dayfirst=True)`
for day-first (European) dates. This isn't just a friendlier message: a
date column in a non-ISO format used to come back `NULL` for every row in
every downstream `d.compare()`/`d.trend()`/etc. query, silently — `load()`
now converts it to a real date type before anything else touches it.
A `.pdf`'s tables are detected heuristically (a PDF has no real table
structure the way a spreadsheet does) — every load carries an
unconditional caveat saying so, and `pyds.pdf_tables(path).get(index)`
previews one table's raw extraction before you trust `load()` with it.

## Writing to a real database

The other direction: once a table's loaded, `d.export_to_sql()` writes it
to SQL Server, Oracle, Postgres, MySQL, or SQLite, via SQLAlchemy —
needs `pip install pyds-ai[db]` plus the driver for your target database
(e.g. `pyodbc` for SQL Server, `oracledb` for Oracle). Nothing about this
is reachable from `d.ask()` — it's a completely separate, explicitly-
invoked path, unlike everything else in this README.

```python
d = pyds.load("export.csv")
r = d.export_to_sql(
    "mssql+pyodbc://user:pass@myserver/mydb?driver=ODBC+Driver+17+for+SQL+Server",
    "sales_data",
    if_exists="replace",  # pandas' own vocabulary: "fail" (the default), "replace", or "append"
)
print(r.headline)
# "Wrote 12,483 row(s) to 'sales_data' (replace) at mssql+pyodbc://user:***@myserver/mydb..."
r.teach()  # a walkthrough of the dtype mapping, batching, and why it was written this way
```
The connection string never appears anywhere with credentials — every
headline, caveat, and trace entry shows only the redacted form, and even
a failed connection's own driver exception gets the literal password
scrubbed out before it reaches you. `if_exists` defaults to `"fail"` so a
repeat call never silently overwrites anything.

## Metrics: define it once, everyone gets the same number

Without a definition, "activation rate" gets re-guessed by the model on
every call — two people asking the same question can get two different
numbers. `pyds.define()` fixes a metric's formula once and caches it next
to your data, in a plain YAML file a human can open and edit:

```python
pyds.define(
    "activation rate",
    "users who completed onboarding divided by signups",
    grain="user",
    owner="you@company.com",
)
pyds.metrics()  # lists what's defined for this dataset
pyds.undefine("activation rate")
```

This writes (and reads) `metrics.yaml` next to the dataset's source
file — the handoff artifact between the data person and the product
owner. It ships with a starter template of ~15 common metrics (activation,
retention, churn, conversion, MAU/WAU/DAU, ARPU, CAC, NRR, funnel-step
conversion, LTV, and more) with descriptions filled in but no formula
yet; `pyds.define(name, description)` resolves each one for your real
data. Resolving a definition is the *only* place a model touches a
metric's arithmetic — one call, cached forever after. Every `ask()` about
a defined metric compiles that cached formula, and the card names which
definition it used, in the interpretation line and in `result.why()`. If
a question mentions something close to (but not exactly) a defined name,
you get offered the defined metric instead of a fresh guess:

```python
d.ask("what's our activation this quarter?")
# -> a Clarify card: "Did you mean the defined metric 'activation_rate'?"
```

## Ask the questions leadership actually asks

```python
d.compare("activation_rate", period="quarter", vs="target")
d.trend("mau", period="week", window=12)
d.drivers("revenue", period="quarter")  # "why did it fall" -> a ranked, waterfall answer
d.segment("conversion_rate")  # "which groups are different"
d.funnel(steps=["signed_up", "completed_onboarding", "made_purchase"])  # "where do people drop off"
d.cohort(by="signup_month")  # "is the March cohort better than January"
d.before_after("2026-03-01", "revenue", window="4 weeks")  # "did the launch work"
d.experiment("variant", "converted")  # "did the test win"
```

- **`d.compare(metric, period="quarter", vs=None)`** — current period, prior period, absolute and percentage change, direction. `vs="target"` compares against a `target` column or a fixed target set on the metric.
- **`d.trend(metric, period="week", window=12)`** — a plain-English read on whether the latest move is normal or worth a look ("down 4 percent, which is within the normal week-to-week swing for this metric"), never "standard deviation" or "confidence interval."
- **`d.drivers(metric, period="quarter")`** — decomposes a change across every categorical column, ranks the dimensions and categories that actually explain it, and renders a waterfall chart. An additive metric (a sum or count) gets an exact category-by-category contribution; a defined ratio metric gets a mix-vs-rate split (did the *mix* of segments shift, or did a segment's own rate change).
- **`d.segment(metric, dimensions=None)`** — which groups depart most from the overall average, ranked, with the sample size next to each so nobody acts on a segment of four users.
- **`d.funnel(steps, event_col=None, id_col=None, split_by=None)`** — step-by-step conversion, counting distinct people at each named step, with per-step and overall conversion and the biggest drop-off (by people lost, not percentage) called out in the headline. Renders as a horizontal funnel chart; `split_by` breaks the same funnel out by a dimension (channel, plan, ...).
- **`d.cohort(by="signup_month", metric="retention")`** — the standard cohort retention table (cohorts down the side, periods since signup across the top), rendered as a heatmap, with a plain read on whether recent cohorts retain better or worse than older ones. `by` names a date column plus a period grain (`"signup_week"`, `"signup_quarter"`, ...).
- **`d.before_after(date, metric, window="4 weeks")`** — the window before a date vs. the window after it, both averages and the change — with an explicit, unconditional caveat that this is a before-and-after comparison, not a controlled test.
- **`d.experiment(group_col, metric)`** — an A/B read for someone who's never run one: both group sizes, both metric values, the lift, and a plain-English verdict ("large enough to act on" or "could easily be noise, you'd need roughly N more users per group"), backed by a real statistical test with every piece of statistics vocabulary kept out of the answer (it's still in `result.why()` for anyone who wants it). Refuses, with a clear reason, when the groups are too small or too imbalanced to read.

`metric` accepts a defined metric name or a plain column name either way.

**Accuracy:** *pending first run — see [`benchmark/`](benchmark/) to run
the 200+-case golden set yourself.* It's scored by execution match (does
the generated SQL return the same data as a known-correct query), not by
comparing SQL text.

## Every result is the same shape

Every call returns a `Result` — never a bare DataFrame, never a dict:

```python
result.explain()  # what was done and why, in sentences
result.code()  # the exact SQL/code that produced it
result.data()  # the underlying DataFrame, full precision — never reformatted
result.next()  # up to three suggested follow-up questions
result.why()  # trace: prompt version, model, cost, row counts
result.summary()  # three to five plain-English sentences for an email or speaker notes
result.teach()  # a walkthrough of the SQL and the stats behind this result, for learning
result.save(path)  # .html, .csv, .docx, .pptx, .xlsx, .pdf
```

An `Answer`/`Chart` card also opens with a one-sentence restatement of
how the question was read — which metric (naming the defined metric used,
if any), which filter, which grouping, and which time window — built from
the plan structure, not a second model call. It's how you catch a wrong
reading before it lands in a deck. Every result derived from a dataset —
not just `ask()`, also `.model()`, `.profile()`, `.compare()`, `.trend()`,
and the rest — shows the data's as-of date (`result.as_of`) on its card
and in every export, so a number repeated a week later still says which
snapshot it came from. Numbers on the card, in chart labels, and in
every export are formatted for presentation (thousands separators, `$`
when a column looks like money, `%` when it looks like a rate) — none of
that touches `result.data()` or a CSV export, which stay full precision.

## Learning from a real result

Every card deliberately keeps SQL and statistics out of the headline —
but they're never hidden, and `result.teach()` is built for the person
who actually wants them:

```python
r = d.experiment("variant", "converted")
print(r.headline)  # "A vs B: converted is 19.6% vs 23.2% ... large enough to act on"

r.teach()
# Names the actual technique behind the headline (a two-proportion
# z-test here, because the outcome is yes/no) and why it fits, walks
# through the SQL clause by clause, connects the headline's numbers
# back to result.why()'s z-score and p-value, and ends with one
# concrete "try it yourself" question tied to a real number already on
# the card (e.g. "what happens to required_n_per_group with half the
# sample size?").
```

Grounded in the exact same `result.code()`/`result.why()` any result
already carries — nothing new is computed, and it never describes a
technique that isn't actually reflected in the SQL or the trace. A
paid call like `.summary()`, cached separately after the first one;
`Failure.teach()` never calls the model at all.

## Presentable output

Anything this audience has to edit before showing it is a failure, so the
output is built to go straight into an email, a deck, or a spreadsheet:

```python
result.summary()  # "Revenue grew 12% this quarter, driven mostly by Enterprise/AMER. ..."

pyds.deck(
    [d.compare("revenue"), d.drivers("revenue"), d.segment("revenue")],
    title="Q2 Business Review",
    path="q2_review.pptx",
)

pyds.brand(logo="logo.png", colors=["#8e44ad"], template="corp.potx")

result.save(
    "report.xlsx"
)  # Answer (headline + a live chart), Data (autofiltered), How this was calculated
```

- **`result.summary()`** — three to five plain sentences: the finding, its size, and the one caveat that matters, written by the model from the numbers already on the card — it never computes one itself. A paid call, so it's cached after the first one; `Failure.summary()` never calls the model at all.
- **`pyds.deck(results, title=, path="deck.pptx")`** — a title slide, one large-chart slide per result, a closing summary slide, and an appendix with the full tables, the metric definitions used, and the SQL. Charts are native, editable PowerPoint chart objects built straight from each result's own data — not a picture of the notebook chart — so a `d.cohort()` result becomes a real colored table instead of a chart that doesn't apply to it.
- **`pyds.brand(logo=, colors=, template=)`** — configure once, applied everywhere: every chart's colors, the notebook/HTML card's accent and logo, and the logo/colors on every `.docx`/`.pptx`/`.xlsx` export. `template` is your own `.potx`/`.pptx` file, used as the base for every pptx export and deck instead of the stock layout. A missing logo or template file is skipped quietly — a decorative asset should never block a report.
- **`.xlsx` export** — "Answer" (headline, interpretation, as-of date, caveats, a live chart referencing the Data sheet), "Data" (the full table, real numeric cells with a formatted number style, autofiltered), "How this was calculated" (the metric definition, the SQL, the model call and prompt version — or an honest "no model call, computed deterministically" for the eight verbs that never touch the LLM at all).
- **Shareable `.html`** — already a single, genuinely self-contained file (the chart is inline SVG, there's no external stylesheet, font, or image — a configured logo is embedded as a base64 data URI, never a file path). Includes a print stylesheet, so printing or saving to PDF from a browser shows the full table/code/trace instead of a collapsed disclosure triangle.

## Failures render as a card, not a traceback

A call that hits a real problem — no API key configured, a bad file
path, an unknown target column — returns a `Failure` (also a `Result`)
instead of raising. `Failure` is falsy, so `if not result:` is the check:

```python
d = pyds.load("sales.csv")
if not d:
    print(d.explain())  # what went wrong, and what to try next
```

For a script where nobody is there to read a card, turn this off and get
the old raise-through behavior back:

```python
pyds.config.set_strict(True)
```

## Privacy default

`send_data_values = false` by default: the model gets column names, types
and summary statistics, not your raw rows. Your data does not leave the
building unless you explicitly opt in. Set it via:

```bash
pip install pyds-ai
```
```python
import pyds_ai as pyds

cfg = pyds.config.load_config()
cfg["privacy"]["send_data_values"] = True
pyds.config.save_config(cfg)
```

Your API key is stored in your OS credential store automatically when
`pip install pyds-ai[keyring]` is installed. Without it, the key falls
back to `~/.pyds_ai/config.toml`, written with owner-only file
permissions. Either way, `pyds.setup()` never prints the key back.

## Installing extras

```bash
pip install pyds-ai[ml]          # AutoML: scikit-learn (+ joblib, shap)
pip install pyds-ai[rag]         # RAG: pdf/docx/html parsing
pip install pyds-ai[rag-chroma]  # persistent vector store (Chroma)
pip install pyds-ai[rag-faiss]   # in-process ANN vector store (FAISS)
pip install pyds-ai[rag-pgvector]# Postgres + pgvector vector store
pip install pyds-ai[report]      # result.save(): .pptx, .docx, .pdf; pyds.deck() (.xlsx and Excel loading need no extra — openpyxl is core)
pip install pyds-ai[anthropic]   # or [openai], [gemini] — Ollama needs no extra
pip install pyds-ai[keyring]     # store your API key in the OS credential store
pip install pyds-ai[db]          # d.export_to_sql(): SQLAlchemy (+ your own DB driver, e.g. pyodbc/oracledb)
pip install pyds-ai[pdf-tables]  # pyds.pdf_tables() / loading tables out of a .pdf
pip install pyds-ai[all]
```

Importing a missing extra returns a friendly message with the exact pip
command, never a bare `ImportError`. The default `pyds.knowledge()` vector
store is a zero-dependency in-memory hashing embedder — RAG works the
moment `pyds-ai[rag]` is installed, no embedding API call required.

## Command line

```
$ pyds doctor
[X]  Config file
      Fix: run `pyds setup`.
[X]  API key
      Fix: run `pyds setup` (or set the anthropic provider's API key environment variable).
[OK] DuckDB

Run the fix command(s) above, then run `pyds doctor` again.
```

`pyds doctor --json` prints the same checks as raw JSON for scripts.
`pyds setup` runs the same interactive key wizard as `pyds.setup()`, from
a terminal instead of a notebook. `pyds report <data> "<question>" --out
<path>` and `pyds eval <cases.csv> --docs <folder>` run the `ask` and
`evaluate` pipelines end to end without opening Python.

## Plugins

Internal teams can register a company database reader or a private model
provider without forking, via entry points:

```toml
[project.entry-points."pyds.providers"]
internal = "mycompany.pyds_plugin:InternalLLM"
```

Anything registered under the `pyds.providers` group is picked up by
`pyds.setup(provider="internal")` / `pyds.config.get_llm_client()` the
same way the four built-in providers are.

## Development

```bash
git clone <this repo> && cd pyds-ai
python -m venv .venv && .venv/Scripts/activate   # or source .venv/bin/activate
pip install -e ".[all,test]"
pytest
```

The test suite runs fully offline: a fake LLM client (`tests/support.py`)
stands in for every provider, so no API key or network access is needed
to validate the package.

## Publishing (maintainers)

```bash
python -m pip install --upgrade build twine
python -m build            # writes dist/*.whl and dist/*.tar.gz
twine check dist/*
twine upload dist/*        # or: twine upload --repository testpypi dist/*
```

`pyds-ai/pyds-ai` is a placeholder repo path, used consistently across
`pyproject.toml`, this README's badges and screenshot URLs, `mkdocs.yml`,
`CONTRIBUTING.md`, and `.github/ISSUE_TEMPLATE/config.yml` — a single
find-and-replace for the real org/repo once it exists on GitHub. Two
things depend on it working correctly, not just looking right: the
README's two screenshots are absolute `raw.githubusercontent.com` URLs
(a relative path renders fine on GitHub but shows broken on the PyPI
project page, which doesn't resolve repo-relative links), and the CI/
docs-deploy workflows trigger on push to `main` — confirm the real
repo's default branch is actually named `main` before relying on either.

## License

Apache-2.0 — see [LICENSE](LICENSE).
