Metadata-Version: 2.4
Name: datus-flink-plugin
Version: 0.1.0
Summary: Datus skill-only plugin for building and operating Flink jobs on Kubernetes Operator
Author: Datus
License-Expression: Apache-2.0
Keywords: datus,flink,kubernetes,operator,plugin,skill
Requires-Python: >=3.12
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

# datus-flink-plugin

A skill-only Datus plugin for creating and operating Apache Flink jobs. It
bundles two skills, one per stage of a job's life:

| Skill | Use it for |
|---|---|
| `flink-local-dev` | Running a Flink SQL job on the current machine, in an in-process MiniCluster, to validate its logic before it goes anywhere |
| `flink-k8s-operator` | Building, deploying, and operating the job on the Apache Flink Kubernetes Operator |

The plugin intentionally declares no `datus flink` CLI and no Flink profiles.
`flink-local-dev` drives the Flink SQL Client in the local shell;
`flink-k8s-operator` builds JVM or PyFlink projects, prepares Operator custom
resources, and delegates every Kubernetes workload operation to the separately
installed `datus k8s` plugin.

Each skill is a **single `SKILL.md`** — Datus discovers the skill file only, so a
skill directory cannot ship an `assets/` or `references/` subdirectory. Every
template the skills hand to a project (Operator manifests, Dockerfiles, the SQL
overlays, the local runner script) is inlined in the skill file under a
`### <filename>` heading, and the test suite extracts those blocks to check them.

## Install

Install and configure the Kubernetes plugin first — it is needed for the
deployment stage, not for local validation:

```bash
datus plugin install src:./datus-k8s-plugin
```

Then install this plugin:

```bash
datus plugin install src:./datus-flink-plugin
```

Both skills appear in the Datus skill catalogue. Invoke `flink-local-dev` when
writing or debugging a Flink SQL job; invoke `flink-k8s-operator` when creating,
upgrading, suspending, resuming, snapshotting, or diagnosing a FlinkDeployment
or FlinkSessionJob.

## Local validation, then production

The intended path for a Flink SQL job:

1. **`flink-local-dev`** — run the script in a MiniCluster inside one JVM. No
   Docker, no Kubernetes, no shared cluster. Sources may be real development
   endpoints; every sink is shadowed with `print`, `blackhole`, or a local
   `file://` table, so the run cannot write to a real system. Judge the output
   rows, changelog kinds, and counts against what the query should produce.
2. **`flink-k8s-operator`** — package the validated script, deliver the image,
   render the FlinkDeployment or FlinkSessionJob, and apply it through
   `datus k8s`.

The production artifact (`sql/job.sql`) is byte-identical in both stages;
everything local lives in a separate, never-shipped overlay:

```
deploy/flink/<name>/
├── sql/job.sql                 # the artifact — unchanged between stages
├── local/                      # flink-local-dev overlay (git-ignore credentials)
│   ├── local-session.sql       # pins execution.target=local, table.dml-sync=true
│   ├── local-sources.sql       # bounded, read-only dev source shadows
│   ├── local-sinks.sql         # print / blackhole / file:// sink shadows
│   └── run-local-sql.sh        # preflight + guards + SQL Client invocation
└── flinkdeployment.yaml        # flink-k8s-operator output
```

`run-local-sql.sh` fails closed: it refuses to run when the session overlay does
not pin the local execution target and synchronous DML, when an `INSERT` target
has no local shadow, when a sink connector or path is not local, or when an
overlay carrying a credential is tracked by git.

Local validation needs a Flink distribution on the machine (`FLINK_HOME`) whose
minor version matches the production `spec.flinkVersion`, and a JDK that release
supports. It does not need Docker or cluster access.

## Runtime boundary

- Flink Operator installation, CRDs, cluster RBAC, and webhooks are managed by
  the Kubernetes administrator.
- Flink workload reads and writes use `datus k8s` and inherit its namespace
  allowlist and confirmation policy.
- A local run never writes to a production sink, consumer group, CDC slot, or
  checkpoint path, and never builds or pushes a production image.
- Application jobs may package a JAR or Python project into a custom Flink
  image.
- Session jobs normally use an Operator-accessible HTTPS, S3, or HDFS
  artifact URI. A `local://` URI refers to the Operator pod filesystem, not
  merely the Session Cluster image.
- The Operator submits jars, not SQL scripts: a validated SQL script reaches
  production through a SQL runner jar or a SQL Gateway, decided per project.

The initial schema guidance targets the stable Operator 1.15 API while
discovering the actual `flink.apache.org` resource version from the target
cluster before generating a manifest.

## Develop

```bash
uv run --package datus-flink-plugin pytest datus-flink-plugin
```

The suite renders every template, checks the documented invariants, and drives
`run-local-sql.sh` against a fake Flink distribution to prove each guard rejects
what it claims to reject. The package contains no runtime Python implementation
and never imports or depends on `datus`.
