Metadata-Version: 2.4
Name: etlgen-client
Version: 0.1.3
Summary: Python SDK and CLI for Fibtec's ETL Integrator (etlgen.fibtec.co.uk)
Author: Fibtec Limited
License-Expression: Apache-2.0
Project-URL: Homepage, https://etlgen.fibtec.co.uk
Project-URL: Repository, https://github.com/fibtecltd/fibtec-etlgen
Project-URL: Bug Tracker, https://github.com/fibtecltd/fibtec-etlgen/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff>=0.15.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# etlgen-client

Python SDK and command-line client for [Fibtec's ETL Integrator](https://etlgen.fibtec.co.uk):
describe a source and a destination, and get back the Python that connects
them, as an ETL script (optionally with tests), a REST or GraphQL connector, or
a webhook handler. Nothing you upload is stored by the service, and this
package stores and logs nothing either.

**Status: alpha (0.1.x).** Python 3.11+. One runtime dependency, `httpx`.

```bash
pip install etlgen-client
```

## Before you start: authentication

Read this first, because it decides whether this release fits your use.

The API authenticates with a **Cognito ID token** that **expires after one
hour**. There are no API keys yet. So you give the client either a token, or a
function that returns a fresh one:

```python
from etlgen_client import Client

client = Client(token=my_id_token, base_url="https://<your-api-id>.execute-api.<region>.amazonaws.com")
```

- It must be the **ID token**, not the access token: the API reads your
  organisation from the token's `email` claim.
- `base_url` is required. It is the API address your administrator gets from the
  `EtlGenApi` stack's `ApiUrl` output. It must be `https` (plain `http` is only
  accepted for `localhost`).
- **To try it quickly**, copy the ID token from your signed-in portal session
  (browser developer tools, Application, Local Storage, the entry whose name ends in
  `.idToken`). It is valid for an hour.
- **For a script that keeps running**, pass a function. This example uses
  [`pycognito`](https://pypi.org/project/pycognito/), a third-party library (not a
  dependency of this package) that does Cognito's SRP password login and renews
  tokens. It works for password accounts, not Google or Apple sign-in:

  ```python
  from pycognito import Cognito

  user = Cognito(user_pool_id="<pool id>", client_id="<app client id>", username="you@company.com")
  user.authenticate(password="...")

  def fresh_id_token() -> str:
      user.check_token()        # renews the tokens when they have expired
      return user.id_token

  client = Client(token=fresh_id_token, base_url="https://...")
  ```

  The function is called once per request.

**This is not yet suited to unattended CI or server use**, which needs a
long-lived service credential the API does not have. Customer API keys are the
planned follow-up; when they ship, this package gets an `api_key=` argument and
nothing else changes.

## Quick start

```python
from pathlib import Path

from etlgen_client import Client, EtlGenInsufficientCreditError, EtlGenTooLargeError

source = {"format": "xsd", "content": Path("customer.xsd").read_text()}
destination = {"format": "json_schema", "content": Path("customer.schema.json").read_text()}

with Client(token=fresh_id_token, base_url=BASE_URL) as client:
    quote = client.generate.estimate(source, destination)      # free, writes nothing
    print(quote["tier"], quote["price_expected_gbp_pence"], "pence")

    try:
        result = client.generate.etl(source, destination, include_tests=True)
    except EtlGenTooLargeError as exc:
        print(f"{exc.field_count} fields, limit {exc.max_field_count}: split it up")
    except EtlGenInsufficientCreditError as exc:
        print(f"top up first: this job costs {exc.price_gbp_pence} pence")
    else:
        Path("etl_job.py").write_text(result["script"])
        if result["tests"]:
            Path("test_etl_job.py").write_text(result["tests"])
        print(result["unresolved_fields"], result["triangle"]["tes"])
```

The first argument of each call is the dict the API takes: `{"format": ..., "content": ...}`
for a schema pair, an OpenAPI document (or a GraphQL introspection document) for a
connector, a declared webhook surface for a webhook handler. The server validates them.

## Methods

| Call | What it does | Retried automatically? |
|---|---|---|
| `client.generate.etl(source, destination, *, python_version, include_tests, field_mapping_overrides)` | The ETL script, with optional pytest tests | **never** |
| `client.generate.review(source, destination)` | Parse or infer both sides and return them for review, without generating or charging. Put a reviewed schema back as `confirmed_schema` and call `etl` | yes |
| `client.generate.connector(api_surface, *, python_version, platform)` | A REST or GraphQL connector (`files`) | **never** |
| `client.generate.webhook(webhook_surface, *, python_version)` | A webhook handler (`files`) | **never** |
| `client.generate.estimate(source, destination)` | Price quote and developer baseline | yes |
| `client.triangle.rollup()` / `client.triangle.get(job_id)` | The Iron Triangle for your organisation, or for one job | yes |
| `client.billing.get()` | Plan, credit balance, price table, free-plan quota | yes |

**Why the generation calls are never retried.** A paid-plan job deducts credit
before it runs, and the API has no idempotency key, so retrying a call whose
outcome is unknown could charge you twice. Reads and `estimate` are retried on
connection errors, timeouts and 5xx (three retries, with backoff). A timeout on a
generation call raises the underlying `httpx` exception.

The default timeout is 60 seconds, above API Gateway's own 30-second limit.

## Errors

Every API error is an `EtlGenError` subclass carrying `response_status` and
`response_body`. A network failure with no response raises the `httpx` exception.

| Exception | Status | Meaning |
|---|---|---|
| `EtlGenAuthError` | 401 | Token missing, expired (an hour) or not an ID token |
| `EtlGenInsufficientCreditError` | 402 | Balance too low for this job (`.price_gbp_pence`). Nothing ran |
| `EtlGenPlanLimitError` | 403 | The free plan cannot take a pair this complex (`.tier`) |
| `EtlGenValidationError` | 422 | Bad request, or a schema that could not be parsed |
| `EtlGenTooLargeError` | 422 | Over the per-job field limit (`.field_count`, `.max_field_count`). Refused before any charge |
| `EtlGenContentSafetyError` | 422 | The result failed the leak check and was not delivered (`.reason`). The request is refunded (`.refunded` is `True`): the charge goes back to your credit, or a free-plan request is not counted. `.refunded` is `None` against an API from before refunds, where the work was done and **was not refunded** |
| `EtlGenQuotaError` | 429 | The free plan's monthly quota is used (`.monthly_request_cap`) |
| `EtlGenRateLimitError` | 429 | API Gateway throttling (`.retry_after`) |
| `EtlGenNotFoundError` | 404 | For example an unknown job id |
| `EtlGenError` | other | Base class. A 502 means the upstream model call failed, and the charge was refunded |

The two 422 subclasses are also `EtlGenValidationError`.

## Command line

`pip install etlgen-client` installs an `etlgen` command.

```bash
export ETLGEN_TOKEN=...                       # a Cognito ID token (prefer this over --token)
export ETLGEN_BASE_URL=https://<api-id>.execute-api.<region>.amazonaws.com

etlgen billing get
etlgen generate estimate --params params.json
etlgen generate etl --params params.json --out-dir ./generated
etlgen triangle get --params-json '{"job_id": "..."}'
etlgen list-functions --domain generate
```

`params.json` is a JSON object of the method's keyword arguments. Any string
that starts with `@file:` is replaced by that file's contents, so a schema never
has to be pasted into JSON:

```json
{
  "source":      {"format": "xsd",         "content": "@file:customer.xsd"},
  "destination": {"format": "json_schema", "content": "@file:customer.schema.json"},
  "include_tests": true
}
```

A method with required arguments, called with neither `--params` nor
`--params-json`, prints its signature and documentation instead.

`--out-dir DIR` (for `etl`, `connector` and `webhook`) writes the generated files
under `DIR` (`etl_job.py` and `test_etl_job.py` for an ETL job, the portal's own
names), overwriting existing files, and prints a summary without the file
bodies. A path in the result that would escape `DIR` is refused.

Exit codes: `0` ok, `1` other error or bad input (including a network failure),
`2` usage, `3` auth, `4` validation (including too large and content safety),
`5` insufficient credit, `6` free-plan limit or quota, `7` rate limited, `8` not
found.

A content-safety failure (exit 4) prints the check's `reason` on its own line: either
that the script embeds a value from your sample data, or that the check itself gave no
usable verdict. Nothing is delivered either way, and the line after it says
`this request was refunded`. Against an API from before refunds it says instead that the
work was done and is not refunded.

A failure with no response at all (connection refused, DNS, a timeout) prints one
`error:` line, not a traceback. If it happened on `generate etl`, `connector` or
`webhook` after the request may have been sent, a second line says the job may
still have run and been charged: those calls are never retried, so run
`etlgen billing get` before trying again.

## Licence

Apache-2.0. Copyright Fibtec Limited.
