Metadata-Version: 2.4
Name: dataiku2onnx
Version: 0.0.1b0
Summary: Convert Dataiku DSS 14 visual ML models to ONNX.
Author: WW Sales Analytics
Maintainer: WW Sales Analytics
License: PROPRIETARY — ALL RIGHTS RESERVED
        
        Copyright (c) 2026 WW Sales Analytics. All rights reserved.
        
        This software and its accompanying documentation are confidential and
        proprietary. No licence, express or implied, is granted to any person or entity
        to use, copy, modify, merge, publish, distribute, sublicense, or sell copies of
        this software, in whole or in part, except under a separate written agreement
        with the copyright holder.
        
        Unauthorised copying, distribution, or disclosure of this software, via any
        medium, is strictly prohibited.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
        FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
        COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
        IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
        CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License-File: LICENSE
Keywords: dataiku,dss,machine-learning,onnx
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: <3.12,>=3.10
Requires-Dist: numpy<2,>=1.23
Requires-Dist: onnx<2.0,>=1.14
Provides-Extra: boosting
Provides-Extra: deeplearning
Provides-Extra: dev
Requires-Dist: dataiku-scoring==14.6.2; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: onnxruntime==1.13.1; (python_version < '3.11') and extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: tomli>=2.0; (python_version < '3.11') and extra == 'dev'
Provides-Extra: sklearn
Description-Content-Type: text/markdown

# dataiku2onnx

Convert **Dataiku DSS 14** visual ML models to **ONNX**. A saved-model version goes in; a single
self-contained `.onnx` file comes out, carrying Dataiku's own feature preprocessing inside the model
rather than assuming a Dataiku runtime at scoring time.

> **This is a first beta — `0.0.1b0`.** The behaviour described here has been measured, but the
> package is early: the API may change, and it has not yet been used outside the project that built
> it.

## About

You train a model in the DSS visual ML lab. Somewhere else — a scoring engine, an application, a
database — you need the same predictions, without a Dataiku runtime. `dataiku2onnx` reads the
saved-model version DSS exports and emits one ONNX file that reproduces the whole pipeline: the
feature preprocessing, the estimator, the calibration, and the predicted label.

What that means in practice:

- **Preprocessing is reproduced in the graph, not around it.** Imputation, rescaling, dummification,
  categorical encoding, derived features and the rest are ONNX nodes. Nothing has to be
  re-implemented by whoever scores the model.
- **Nothing is silently approximated.** A step this package cannot express faithfully is refused by
  name at conversion time, and the refusal says what to change. A model that converts is a model
  whose preprocessing is complete.
- **Every conversion is validated, and the validation cannot be turned off.** There is no
  `validate=False`, no environment variable and no separate validator to remember to call.
- **Nothing is re-fitted, and no estimator library is installed.** Every fitted parameter — for
  scikit-learn, XGBoost and LightGBM alike — is read out of the JSON DSS writes at train time. The
  whole runtime dependency set is `onnx` and `numpy`.
- **Nothing is unpickled.** The loaders read DSS's JSON, never the pickles beside it.
- **It does not connect to DSS.** You fetch the saved-model version with the client you already
  have; this package converts the archive you hand it.
- **It is a Python library used from scripts. There is no command-line interface.**

The emitted model targets **ONNX opset 16, IR version 8** by default, with weights stored inline in
the file. The limitations section below explains why the default is conservative.

### Licence

**`PROPRIETARY — ALL RIGHTS RESERVED`.** **Installing this package does not grant you a licence to
use it.** No licence, express or implied, is granted to use, copy, modify or distribute it, except
under a separate written agreement with the copyright holder. Full terms are in the `LICENSE`
file inside the installed package.

`pip install` normally implies permission to use what you installed. Here it does not, which is why
it is said plainly rather than left in a file inside the package.

## Quickstart

### From an exported saved-model version

In DSS: **Saved model → the version → Actions → Export → Python scoring**. You get a zip.

```python
from dataiku2onnx import FilesystemLoader, convert

model = FilesystemLoader("exports/churn_v3.zip").load()
result = convert(model)
result.save("churn_v3.onnx")

print(result.report.describe())
```

`FilesystemLoader` takes an unpacked version directory, a `model.zip`, or the export zip DSS gives
you, and works out which it was handed.

### Inside a DSS Python recipe

Against a model deployed in your flow, with the client the recipe already holds. Nothing is written
to disk:

```python
from pathlib import Path

import dataiku

from dataiku2onnx import InMemoryLoader, convert

SAVED_MODEL_ID = "YOUR_SAVED_MODEL_ID"

project_key = dataiku.default_project_key()
saved_model = dataiku.api_client().get_project(project_key).get_saved_model(SAVED_MODEL_ID)
version_id = saved_model.get_active_version()["id"]

with saved_model.get_version_details(version_id).get_scoring_python_stream() as stream:
    result = convert(
        InMemoryLoader(
            stream.content,
            project_key=project_key,
            saved_model_id=SAVED_MODEL_ID,
            version_id=version_id,
        ).load()
    )

result.save(Path(dataiku.Folder("model_onnx").get_path()) / "churn_v3.onnx")

print(result.report.describe())
```

The code env needs `dataiku2onnx` and DSS's core packages; nothing else. Add no error handling — if a
model cannot be converted, the exception ends the job and DSS puts the reason in the log, which is
what you want.

The three identifiers are optional and worth passing: an export archive records the saved-model id
nowhere, and they are what let a deployed `.onnx` be traced back to its source months later.

### What comes back

```python
result.report.class_order   # ('No', 'Yes') -- DSS's ordering, read out of the emitted model
result.report.outputs       # ('prediction', 'proba_No', 'proba_Yes')
result.report.warnings      # what to resolve before deploying
result.report.notes         # behaviours that change numbers without changing anything visible
```

## Installation

```bash
pip install dataiku2onnx
```

That is everything a conversion needs. **No model family needs an extra**, because no estimator
library is imported — see the description above.

You also need:

- **Python 3.10 or 3.11.**
- A **Dataiku DSS 14** saved-model version: an exported folder or zip, or the `scoring-python`
  archive fetched with your own DSS client. Verified against **DSS 14.6.0**; the format has been
  read successfully from exports going back to **11.0.3**.

If you fetch versions from a live DSS instance, install `dataiku-api-client` as well — note the
distribution name, which differs from the `dataikuapi` import name. `dataiku2onnx` deliberately does
not declare it: the client is yours, and so is its configuration — the proxy settings and
the TLS trust store included. **Keep certificate verification on.**

## What is supported

### Model types

The set of algorithms DSS 14 can export portably is **closed at nine**, so this table is complete
rather than a snapshot. **Twelve model-and-engine combinations** have been measured against DSS.

| Family | DSS algorithms | Support key | Input types |
| --- | --- | --- | --- |
| Linear models | `LINEAR`, `LOGISTIC` | `LINEAR`, `LOGISTIC` | categorical, float, integer |
| Trees and ensembles | `DECISION_TREE`, `FOREST_CLASSIFIER`, `FOREST_REGRESSOR` | `DECISION_TREE`, `FOREST_CLASSIFIER`, `FOREST_REGRESSOR` | categorical, float, integer |
| Gradient boosting (scikit-learn) | `GRADIENT_BOOSTING_CLASSIFIER`, `GRADIENT_BOOSTING_REGRESSOR` | `GRADIENT_BOOSTING_CLASSIFIER`, `GRADIENT_BOOSTING_REGRESSOR` | categorical, float, integer |
| Gradient boosting (XGBoost engine) | `GRADIENT_BOOSTING_CLASSIFIER`, `GRADIENT_BOOSTING_REGRESSOR` | `GRADIENT_BOOSTING_CLASSIFIER/xgboost`, `GRADIENT_BOOSTING_REGRESSOR/xgboost` | categorical, float, integer |
| Gradient boosting (LightGBM engine) | `GRADIENT_BOOSTING_CLASSIFIER`, `GRADIENT_BOOSTING_REGRESSOR` | `GRADIENT_BOOSTING_CLASSIFIER/lightgbm`, `GRADIENT_BOOSTING_REGRESSOR/lightgbm` | categorical, float, integer |
| **Neural networks** (multi-layer perceptron) | `MLP_CLASSIFIER`, `MLP_REGRESSOR` | `MULTI_LAYER_PERCEPTRON` | categorical, float, integer |

The **support key** is how a family is identified throughout this project's documentation. The three
boosting engines carry their own keys because they do different arithmetic, even though DSS presents
all three as gradient boosting.

**What stands behind that.** Each combination was converted from a real exported model and scored —
28 model runs of 100 rows each — against DSS's own scored output for the same rows. Every predicted
label matched DSS **exactly**, and every probability and regression value landed inside a tolerance
fixed in advance.

Every run exercised categorical, floating-point and integer features **together** in one model. Each
run of 100 rows is **81 ordinary rows, 15 carrying a null in at least one input feature** — 12 of
those in a numeric feature — **and 4 carrying a category the model had never been trained on.**

**What it does not tell you:** that is a measurement on those models, not a guarantee about every
model of the same kind. A model built from a preprocessing step none of them used has not been
measured.

Probability calibration converts for the `SIGMOID` method. `ISOTONIC` is refused, because no real
exported model has ever carried one.

### Preprocessing steps

A DSS visual ML model carries Dataiku's own feature preprocessing, and all of it has to be
reproduced inside the ONNX model, because nothing downstream has a Dataiku runtime to fall back on.
DSS has **21** preprocessing steps in total, and every one of them has a verdict here.

| status | steps | what it means for you |
| --- | --- | --- |
| **converts** | `PrepareInput`, `Impute`, `Binarize`, `Flag`, `Dummify`, `Rescale`, `Derive`, `DeriveRescale`, `CategoricalEncode`, `Selection`, `Calibrator` | Reproduced in the ONNX model and measured against DSS. |
| **not expressible at opset 16** | `Normalize`, `DatetimeCyclical`, `VectorizeWordCount`, `VectorizeTfidf`, `VectorsUnfold` | Refused, with a message naming the step. Each needs a model to look inside a string, which needs ONNX operators well above the emitted opset — see the limitations section below. Do that feature engineering upstream and train on the result. |
| **refused upstream by DSS** | `NumericalNumericalInteractions`, `NumericalCategoricalInteractions`, `CategoricalCategoricalInteractions` | DSS itself refuses to export a model that generated these, so no file exists. Not this library's decision. |
| **structurally impossible** | `DropRows` | Cannot be expressed in a per-row model, at any ONNX version. |
| **unobserved shape** | `RawCategoricalEncode` | A converter exists, and no real exported model has ever used it. Not blocked — unproven, and flagged on every conversion that does. |

Nothing is awaiting implementation. Every one of the 21 steps has a verdict above, and every
refusal names the step it refused and what to do instead.

## Loaders and the convert function

Two loaders, and they produce equal models. Pick the one that matches where the archive is.

| | takes | use it when |
| --- | --- | --- |
| `FilesystemLoader(path)` | an unpacked version directory, a `model.zip`, or DSS's export zip | the archive is on disk |
| `InMemoryLoader(payload, label=...)` | the archive's `bytes`, or a **seekable** binary stream | the archive came over a network, out of a blob store or straight from a DSS client |

Both accept `project_key=`, `saved_model_id=` and `version_id=`, which are stamped into the emitted
model. An export archive records the saved-model id nowhere, so passing them is what lets a deployed
`.onnx` be traced back months later. `InMemoryLoader` writes no temporary file: `bytes` are copied
once, a stream is not copied at all, and a non-seekable stream is refused rather than quietly
buffered — a zip is read from its central directory at the end of the file, so it cannot be read
forwards.

`.load()` gives a `DataikuModel`. Everything else starts from there.

```python
from dataiku2onnx import FilesystemLoader, convert, plan

model = FilesystemLoader("exports/churn_v3.zip").load()

decision = plan(model)          # what would happen, without building anything
result = convert(model)         # the model and its report
path = result.save("churn_v3.onnx")
```

**`convert(model, *, opset=16, numeric_dtype="float64", converted_at=None)`** returns a
`ConversionResult` carrying `model` — the `onnx.ModelProto` — and `report`. `opset` selects the
target `ai.onnx` opset (16 by default, 17 also accepted). `numeric_dtype` is the working element
type, `float64` by default because Dataiku's own feature matrix is `float64`, so the arithmetic
matches without a cast. `converted_at` exists so a build can be made reproducible — **not** so
anything can be skipped.

**`plan(model)`** answers the same question `convert()` would, without building a graph. It reports
*every* refusal a model carries rather than the first, which makes it the cheap way to survey a
directory of exports and find the ones that will not convert.

**`result.report`** is the part to actually read. `describe()` prints a one-screen summary; the
fields behind it are `steps` (one outcome per preprocessing step, in order, so a silently skipped
step is detectable rather than merely discouraged), `estimator`, `inputs` and `input_types`,
`outputs`, `class_order`, `column_order`, `metadata`, `model_bytes` and `diagnostics`.
`report.warnings` are things to resolve before deploying; `report.notes` are behaviours that change
numbers without changing anything visible — a value treated as `NaN`, a float32 tree round trip, an
applied calibrator, a decision threshold that is not 0.5. Every finding carries a `DiagnosticCode`,
so you can match on it without parsing English.

**A refusal is an exception, never a quietly reduced model.** `convert()` raises
`UnsupportedPreprocessingError`, naming every step it cannot express, or `UnsupportedModelError` for
an estimator it does not handle, rather than handing back something that fails later. The validation
runs on every call and there is no flag that skips it.

## Limitations

Some models cannot be converted, and the reason tells you whether waiting helps.

| | what | does waiting help? |
|---|---|---|
| **DSS does not export it** | Keras / TensorFlow / PyTorch models; text handled by hashing, which is the DSS default for a text column; combination (interaction) features; custom preprocessing code | **No, and it never will.** DSS's own portable export refuses these, so no file exists for any converter to read. Neural networks *are* covered, through the multi-layer perceptron, which is what DSS does export. |
| **Not expressible at the emitted opset** | text vectorisation, date parsing, cyclical date encoding, vector unfolding | **Not by upgrading anything of yours.** Each needs a model to look inside a string, which needs ONNX string operators introduced at opset 19–20 — well above the conservative opset this package emits at. That is our default, not your runtime's limit, and it is deliberate. See below. |
| **No ONNX model can ever do it** | dropping rows; returning a blank prediction | **No.** An ONNX model scores one row at a time and returns a value for every row it is given. Where DSS declines to predict and returns a blank, a converted model returns a number. |

### Why the emitted opset is conservative

Opsets and IR versions are backward compatible: a runtime that accepts opset 20 accepts 16, and one
that accepts IR 10 accepts IR 8. Emitting at **opset 16 / IR 8** therefore loads on essentially
every ONNX runtime from roughly 2022 onward, current ones included, without asking you which one you
have.

Raising the default would buy the string operators the second row above needs, and would pay for
them by turning a refusal into a load failure on older runtimes. **A load failure is the worse of
the two.** A refusal happens at conversion time, on your machine, in our words, names the step and
comes with a workaround. A load failure happens at deployment time, in the runtime's words, and says
nothing about which step caused it. The conservative default turns an unknown into a known.

The target opset is a parameter of `convert()`, so this is a default rather than a wall.

### Keras, TensorFlow and PyTorch

**DSS has no portable export for these frameworks, so there is no file to convert.** The path ends
at *export*, before any question about ONNX arises — Dataiku's own portable scoring library supports
the nine algorithms above and no deep-learning framework.

**This is a boundary of the DSS export format, not of this library.** Neural networks are covered, by
`MLP_CLASSIFIER` / `MLP_REGRESSOR`, and they are among the twelve combinations measured above.

### Also not covered

**Custom user-written Python models, clustering, time-series forecasting, models stacked or
ensembled out of other models, and partitioned models.** All are detected and refused with a clear
message — never silently mis-converted.

### The evidence is one environment

Everything measured here was measured against **DSS 14.6.0** exports. Another DSS version has not
been measured, and a model whose preprocessing none of the measured models used has not been
measured either. A conversion that merely runs without error proves that the model was built, not
that it agrees with DSS.
