Metadata-Version: 2.5
Name: convalesce-emit-gx
Version: 0.3.0
Summary: Forward Great Expectations validation results to Convalesce.
Author: Convalesce
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: convalesce-emit<1,>=0.3.0
Description-Content-Type: text/markdown

# convalesce-emit-gx

Forward Great Expectations validation results to Convalesce: every
expectation's outcome, its counts and statistics, and the failing values
Great Expectations sampled. One setting keeps the values where they are.

## Install

```sh
pip install convalesce-emit-gx
```

## Wire it up

One class path serves both majors. `convalesce_emit_gx.action` picks the
implementation for the installed Great Expectations.

On 1.x, list the action on the checkpoint:

```python
from great_expectations import Checkpoint
from convalesce_emit_gx.action import ConvalesceValidationAction

checkpoint = Checkpoint(
    name="orders",
    validation_definitions=[...],
    actions=[ConvalesceValidationAction()],
)
```

On 0.x, list it in the checkpoint's action list:

```python
action_list=[
    {
        "name": "convalesce",
        "action": {
            "module_name": "convalesce_emit_gx.action",
            "class_name": "ConvalesceValidationAction",
        },
    }
]
```

Set `CONVALESCE_INGEST_KEY` where the checkpoint runs.

### Validating outside a checkpoint

Great Expectations runs actions only from a checkpoint.
`ValidationDefinition.run()`, `Batch.validate()` and a 0.x
`Validator.validate()` fire nothing, so send their result yourself:

```python
from convalesce_emit_gx import forward_validation_result

result = validation_definition.run()
forward_validation_result(result)
```

On 0.x, pass the validator too, `forward_validation_result(result,
validator=validator)`, so the platform is named from its engine.

### Naming the table a checkpoint validates

A table or query asset is matched to its table in the catalogue from the
datasource. A dataframe has no table that Great Expectations knows of, so
name it on the action:

```python
ConvalesceValidationAction(
    platform="postgres",
    dataset_name="my_db.my_schema.orders",
)
```

On 0.x the same fields go beside `class_name` in the action's entry:

```python
"action": {
    "module_name": "convalesce_emit_gx.action",
    "class_name": "ConvalesceValidationAction",
    "platform": "postgres",
    "dataset_name": "my_db.my_schema.orders",
},
```

| Field | What it names |
|---|---|
| `platform` | The platform the table lives on, as the catalogue names it: `postgres`, `snowflake`, `bigquery`. |
| `dataset_name` | The table, as the catalogue names it: `my_db.my_schema.orders`. Every result of the checkpoint is filed under it, so set it on a checkpoint that validates one table. |
| `platform_instance` | The instance of the platform, where your catalogue tells several apart. |

Each is optional, and what is set applies to that checkpoint alone.
`forward_validation_result` takes the same three as keyword arguments.

## What crosses

- Every field of the checkpoint result, each validation result and each
  expectation result, under every result format.
- Each datasource's platform, database and default schema. Never its
  connection string; a password inside a batch spec is masked.
- The query a query asset's batch is read with, up to 20,000 characters,
  so the tables and columns behind the batch can be read from it. Set
  `CONVALESCE_SEND_SOURCE=false` to keep it where it is.
- Each result's GX Cloud page, when GX Cloud stored it.
- The `platform`, `dataset_name` and `platform_instance` set on the action.

### Row values

A validation result holds some of the values of the table it checked, and
by default they are sent:

- Failing sample values. For a check that fails row by row, Great
  Expectations keeps a sample of the values that failed, 20 by default, and
  how often each occurred (`partial_unexpected_list`,
  `partial_unexpected_counts`).
- What a fuller result format adds. Under `COMPLETE`, every failing
  value and where its row is (`unexpected_list`, `unexpected_index_list`,
  `unexpected_index_query`). Where the result format asks for the failing
  rows themselves, those rows (`unexpected_rows`).
- Observed column values and their counts. A distinct-values or
  most-common-value expectation reports the values it found in the column
  (`observed_value`), and on 0.x how many rows hold each
  (`details.value_counts`).

They are what tells an investigation which rows broke a check. The rows
that passed, and the frame that was validated, stay where they are.

To send none of them, set this where the checkpoint runs:

```sh
CONVALESCE_GX_SEND_SAMPLES=false
```

Each value or list of values is then replaced by how many there were.
Counts, percentages and numeric statistics (a mean, a maximum, a row count)
are sent either way.

## Settings

Set `CONVALESCE_SEND_SETTINGS=true` where your checkpoints run and each
validation result also carries the environment variables of the process
that ran it. It is off by default.

A setting whose name or value reads as a credential, and any value longer
than 300 characters, is sent as a `fingerprint`: a keyed hash, sixteen hex
characters long, made in your process. It says that the value changed
between two runs and nothing of what the value is. Every other setting is
sent as its `value`.

The hash is keyed with `CONVALESCE_FINGERPRINT_KEY` where you set one, and
with a key derived from your ingest key otherwise. The fingerprint key is
only ever read in your process and is never sent. `keyed_by` identifies the
key the hashes were made under, so hashes are compared only between runs
that used the same one.

Set `CONVALESCE_SETTINGS_SKIP` to a comma-separated list of names to keep
those settings out in either form, such as
`CONVALESCE_SETTINGS_SKIP=HOSTNAME,INTERNAL_REGION`.

The event then holds:

```json
{
  "settings": {
    "items": [
      {"kind": "environment", "name": "API_KEY", "fingerprint": "5d1c0a9e7b3f2468"},
      {"kind": "environment", "name": "WAREHOUSE", "value": "analytics"}
    ],
    "keyed_by": "9f3b6c1d2e4a5b70"
  }
}
```

## Supported

Great Expectations 0.17 through 1.x, on Python 3.9 and later. Verified on
real installs: see
[version-support.md](https://github.com/convalesce/emit/blob/main/docs/version-support.md).

## Configure

Read from the environment: see
[convalesce-emit](https://pypi.org/project/convalesce-emit/).
