Metadata-Version: 2.4
Name: gluekit
Version: 1.2.2
Summary: A developer‑centric Glue CLI built to speed local iteration with git/uv‑style commands, while bridging to highly controlled (eg Fedramped) AWS consoles for PII‑sensitive runs and smoothing onboarding around Glue’s quirks—like the lack of script‑to‑notebook conversion and the friction of refactoring Glue scripts.
Author-email: Mike Kranz <kranz-michael@norc.org>
Keywords: python
Requires-Python: <4.0,>=3.10
Description-Content-Type: text/markdown
Requires-Dist: boto3>=1.28.0
Requires-Dist: python-dotenv>=1.2.1
Requires-Dist: python-slugify>=8.0.4
Requires-Dist: pydantic<3,>=2.11
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rapidfuzz>=3.14.3
Requires-Dist: typer>=0.9.0
Provides-Extra: docs
Requires-Dist: mkdocs>=1.6.1; extra == "docs"
Requires-Dist: mkdocs-material>=9.7.1; extra == "docs"
Requires-Dist: mkdocstrings[python]>=1.0.0; extra == "docs"
Provides-Extra: job-dev
Requires-Dist: moto[s3,ssm]>=5.1.16; extra == "job-dev"
Requires-Dist: sqlframe[duckdb]>=4.3.0; extra == "job-dev"
Provides-Extra: spark-dev
Requires-Dist: moto[s3,ssm]>=5.1.16; extra == "spark-dev"
Requires-Dist: pyspark<4,>=3.3; extra == "spark-dev"

# Gluekit

Gluekit works with one active AWS Glue job in a local workspace. Activate a job, edit its local definition or arguments, convert its script and notebook, then push or pull files through AWS Glue and S3.

```bash
gluekit activate my-job --profile my-aws-profile
gluekit status
gluekit settings show
gluekit settings set --worker-type G.1X --number-of-workers 5
gluekit params set --temp-dir s3://my-bucket/tmp/
gluekit script convert
gluekit notebook convert
gluekit push --to aws --dry-run
gluekit push --to aws
```

Run directly from PyPI with `uvx gluekit --help`, or install the command with
`uv tool install gluekit`. Contributors can run the checkout with
`uv run gluekit`.

## Commands

| Command | Purpose |
| --- | --- |
| `activate <job>` | Select one job and optionally an AWS profile for later commands. `activate --clear` clears the selection. |
| `settings show/set` | Inspect or change local Glue job fields such as `Role`, `WorkerType`, and `Timeout`. |
| `params show/set` | Inspect or change `DefaultArguments` and `NonOverridableArguments`. |
| `params save <key> <value>` | Save a reusable value for the active job; add `--profile` or `--global` for that scope. `params saved [job]` inspects these values. |
| `script convert` / `notebook convert` | Convert to the other representation. |
| `script sync` / `notebook sync` | Refresh generated metadata from the local job definition. |
| `script import-settings` / `notebook import-settings` | Import represented metadata into the local job definition. |
| `push --to aws` / `pull --from aws` | Transfer the full selected job. Pass a job name explicitly to override the selection; pass `'*'` explicitly for all jobs. |
| `script push/pull` / `notebook push/pull` | Transfer only that artifact. |
| `init`, `build`, `status` | Set up profile values, build local dependencies, and inspect the active job. |
| `local`, `run` | Set up local fixtures and run Glue scripts locally. |

The default transfer endpoint is AWS. The `--to` and `--from` flags accept `aws`; a GitHub transfer backend is planned separately. Transfer commands use local files as the other endpoint, and do not need `--include` or `--exclude`. To inspect the available options, use `gluekit <command> --help`.

### Script and notebook conversion

```bash
uvx gluekit notebook convert my-job  # notebook → clean Python script
uvx gluekit script convert my-job    # script → notebook with Glue magics
```

Omit `my-job` to use the active job. `--artifact-only` skips the local job
configuration and generated magics; `--dry-run` writes neither artifacts nor state.

Notebook exports contain Python and commented markdown, with no percent cell
markers, generated Glue configuration header, or commented pyproject/PEP 723
metadata. Existing percent markers are removed as well. Python indentation and
user comments are retained; Gluekit does not reformat executable code. Non-Python
magics such as SQL remain as commented documentation, not executable Python.

Cell boundaries live separately under `notebook_cells["my-job"]` in
`.gluekit/state.json`. The `markers` object maps **1-based script line numbers**
to the marker inserted *before* that line when reconstructing notebook cells:

```json
{"markers": {"1": "#%%", "9": "#%%", "15": "#%%[markdown]"}}
```

Each entry also records script/notebook paths and line hashes. At conversion time,
Python's standard-library `difflib` relocates boundaries after insertions,
replacements, and deletions; `ast` moves unsafe boundaries to the start of the
containing top-level statement. Functions, decorators, classes, loops, `with`,
`try/except/finally`, multiline expressions, and `match` blocks stay intact.
Markdown is restored from its script comments without accumulating comment prefixes.
The map updates on conversion, not continuously while editing; no watcher or extra
dependency is needed. Line-diff matching is best-effort for moved or repeated code,
but AST validation still prevents a boundary from splitting a statement.

For a script without saved boundaries or explicit percent markers, Gluekit groups
global setup and starts cells around top-level definitions and compound statements.
Nested blocks remain in their containing cell. Existing explicit markers take
precedence over saved boundaries and are also checked for scope safety. Invalid
Python causes conversion to fail before writing the notebook or its saved map.

Script-to-notebook conversion generates the existing Glue session magics from the
local job configuration, but omits `%iam_role`. It replaces script-only
`job.commit()` statements with no-ops so otherwise empty suites remain valid.
`notebook sync` also omits the role, and importing a notebook without a role leaves
the configured job role intact. Explicitly authored role magics can still be imported.
See the [AWS interactive-session magic reference](https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions-magics.html).

`script sync` remains an explicit opt-in to the older commented Glue configuration
representation; a clean notebook export does not contain settings for
`script import-settings`. Use `notebook import-settings` or edit the job configuration
instead. Saved cell metadata is local workspace state, never an AWS job parameter.

### Workspace configuration

Gluekit owns three validated files with distinct responsibilities:

- `.gluekit/config.json` stores declarative workspace, job, and profile
  overrides. Values use nested native Glue fields; each narrower scope merges
  over the prior one by mapping key. Lists replace as a whole, and a JSON
  `null` explicitly removes an inherited field.
- `.gluekit/state.json` stores one active job, its profile and configuration
  directory, the independently selected local fixture file, and notebook
  round-trip metadata.
- `.gluekit/local.json` stores local S3 mounts and explicit objects plus SSM
  fixtures.

Activation only selects context; it never materializes profile values into the
base job JSON. Commands that need a runnable definition resolve the effective
configuration in memory as workspace defaults, job defaults, profile defaults,
and profile/job overrides. Explicit job patterns and `"*"` are resolved
independently of the active job. A plain `build` does not rewrite job files,
scripts, notebooks, or cell metadata; use `--package-whl` for explicit package
integration and `--push` for AWS transfer.

Existing `.gluekit/glue_set.json` files are validated and migrated once into the
new files without modifying the legacy file. Invalid JSON, unknown fields in the
new contract, or invalid values fail with their file and field path; Gluekit does
not reset the file. Writes validate the complete candidate and replace the target
atomically. The old `checkout` command and deprecated aliases remain unregistered.

## CLI reference

<!-- BEGIN: CLI_HELP -->
> Auto-generated from live CLI `--help` output.

### `gluekit --help`

```text
Usage: gluekit [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  activate  Replace the active job.
  build     Build local artifacts; only touches AWS when --push is used.
  init      Initialize local repo-level Gluekit settings from AWS CLI...
  pull      Transfer the full active job between local files and AWS.
  push      Transfer the full active job between local files and AWS.
  run       Run a Glue script locally with emulated Glue libraries and...
  status    Show the active job and its local-to-S3 artifact mappings.
  settings  Edit the active Glue job definition.
  params    Edit the active job's Glue arguments.
  script    Work with the active job's script.
  notebook  Work with the active job's notebook.
  local     Local Glue fixtures and setup.
```


### `gluekit activate --help`

```text
Usage: gluekit activate [OPTIONS] [JOB_NAME]

  Replace the active job.

Arguments:
  [JOB_NAME]  Glue job name to activate for later commands.

Options:
  --config-dir PATH   Directory containing Glue job config files.  [default:
                      glue/configs]
  -p, --profile TEXT  AWS CLI credential profile name to store as the active
                      gluekit profile scope for saved params.
  --dry-run           Show gluekit profile-scoped config updates without writing
                      files.
  --clear             Remove the active job selection without deleting saved
                      params or local files.
  --help              Show this message and exit.

  **Examples**

  - `gluekit activate my-job`

  - `gluekit activate --clear`
```


### `gluekit status --help`

```text
Usage: gluekit status [OPTIONS]

  Show the active job and its local-to-S3 artifact mappings.

Options:
  --config-dir PATH  Directory containing Glue job config files.
  --help             Show this message and exit.

  **Examples**

  - `gluekit status`

  - `gluekit status --config-dir glue/configs`
```


### `gluekit settings --help`

```text
Usage: gluekit settings [OPTIONS] COMMAND [ARGS]...

  Edit the active Glue job definition.

Options:
  --help  Show this message and exit.

Commands:
  show  Show the active job's local Glue definition.
  set   Manage local AWS Glue job-definition properties.
```


### `gluekit settings show --help`

```text
Usage: gluekit settings show [OPTIONS] [JOB_NAME]

  Show the active job's local Glue definition.

Arguments:
  [JOB_NAME]

Options:
  --config-dir PATH
  --help             Show this message and exit.
```


### `gluekit settings set --help`

```text
Usage: gluekit settings set [OPTIONS] [JOB_NAME]

  Manage local AWS Glue job-definition properties.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active job selection.

Options:
  --description TEXT              Set the job description.
  --role TEXT                     Set the IAM role for the job.
  --glue-version TEXT             Set the Glue version.
  --worker-type TEXT              Set the worker type.
  --number-of-workers INTEGER     Set the number of workers.
  --timeout INTEGER               Set the job timeout in minutes.
  --max-retries INTEGER           Set the max retry count.
  --execution-class TEXT          Set the execution class.
  --max-concurrent-runs INTEGER   Set the max concurrent runs.
  --job-run-queuing-enabled / --no-job-run-queuing-enabled
                                  Enable or disable job run queuing.
  --security-configuration TEXT   Set the security configuration name.
  --maintenance-window TEXT       Set the maintenance window.
  --log-uri TEXT                  Set the log URI.
  --command-name TEXT             Set Command.Name.
  --python-version TEXT           Set Command.PythonVersion.
  --runtime TEXT                  Set Command.Runtime.
  --remote-script-location TEXT   Set Command.ScriptLocation.
  --local-script-location TEXT    Set SourceControlDetails.ScriptLocation.
  --notebook-location TEXT        Set SourceControlDetails.NotebookLocation and
                                  NotebookPath.
  --source-control-provider TEXT  Set SourceControlDetails.Provider.
  --source-control-owner TEXT     Set SourceControlDetails.Owner.
  --source-control-repository TEXT
                                  Set SourceControlDetails.Repository.
  --source-control-branch TEXT    Set SourceControlDetails.Branch.
  --source-control-folder TEXT    Set SourceControlDetails.Folder.
  --add-connection TEXT           Add one or more Glue connections.
  --remove-connection TEXT        Remove one or more Glue connections.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit settings set --description 'Updated job description' --glue-version
  5.0`

  - `gluekit settings set my-job --worker-type G.1X --number-of-workers 5`
```


### `gluekit params --help`

```text
Usage: gluekit params [OPTIONS] COMMAND [ARGS]...

  Edit the active job's Glue arguments.

Options:
  --help  Show this message and exit.

Commands:
  show   Show the active job's local Glue arguments.
  set    Manage local AWS Glue runtime and job arguments.
  save   Save a reusable value for the active job, profile, or all jobs.
  saved  Show stored local Glue config params.
```


### `gluekit params show --help`

```text
Usage: gluekit params show [OPTIONS] [JOB_NAME]

  Show the active job's local Glue arguments.

Arguments:
  [JOB_NAME]

Options:
  --config-dir PATH
  --help             Show this message and exit.
```


### `gluekit params set --help`

```text
Usage: gluekit params set [OPTIONS] [JOB_NAME]

  Manage local AWS Glue runtime and job arguments.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active job selection.

Options:
  --add-additional-python-modules TEXT
                                  Add one or more values to --additional-python-
                                  modules.
  --remove-additional-python-modules TEXT
                                  Remove one or more values from --additional-
                                  python-modules.
  --add-extra-py-files TEXT       Track local relative Python files in
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --remove-extra-py-files TEXT    Remove local Python files from
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --add-extra-jars TEXT           Add one or more values to --extra-jars.
  --remove-extra-jars TEXT        Remove one or more values from --extra-jars.
  --add-additional-python-files TEXT
                                  Track local paths in
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --remove-additional-python-files TEXT
                                  Remove local paths from
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --add-extra-files TEXT          Track local paths in
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --remove-extra-files TEXT       Remove local paths from
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --add-default-argument TEXT     Set a DefaultArguments entry using KEY=VALUE.
                                  Repeat as needed.
  --remove-default-argument TEXT  Remove a DefaultArguments entry by key. Repeat
                                  as needed.
  --add-non-overridable-argument TEXT
                                  Set a NonOverridableArguments entry using
                                  KEY=VALUE. Repeat as needed.
  --remove-non-overridable-argument TEXT
                                  Remove a NonOverridableArguments entry by key.
                                  Repeat as needed.
  --enable-metrics / --disable-metrics
                                  Set DefaultArguments.--enable-metrics to true
                                  or false.
  --enable-observability-metrics / --disable-observability-metrics
                                  Set DefaultArguments.--enable-observability-
                                  metrics to true or false.
  --enable-glue-datacatalog / --disable-glue-datacatalog
                                  Set DefaultArguments.--enable-glue-datacatalog
                                  to true or false.
  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
                                  Set DefaultArguments.--enable-continuous-
                                  cloudwatch-log to true or false.
  --enable-spark-ui / --disable-spark-ui
                                  Set DefaultArguments.--enable-spark-ui to true
                                  or false.
  --enable-job-insights / --disable-job-insights
                                  Set DefaultArguments.--enable-job-insights to
                                  true or false.
  --enable-auto-scaling / --disable-auto-scaling
                                  Set DefaultArguments.--enable-auto-scaling to
                                  true or false.
  --job-bookmark-option TEXT      Set DefaultArguments.--job-bookmark-option.
  --temp-dir TEXT                 Set DefaultArguments.--TempDir.
  --spark-event-logs-path TEXT    Set DefaultArguments.--spark-event-logs-path.
  --datalake-formats TEXT         Set DefaultArguments.--datalake-formats.
  --spark-conf TEXT               Set DefaultArguments.--conf.
  --continuous-log-log-group TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logGroup.
  --continuous-log-log-stream-prefix TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logStreamPrefix.
  --continuous-log-conversion-pattern TEXT
                                  Set DefaultArguments.--continuous-log-
                                  conversionPattern.
  --executor-cores INTEGER        Set DefaultArguments.--executor-cores.
  --customer-driver-env-vars TEXT
                                  Set DefaultArguments.--customer-driver-env-
                                  vars.
  --customer-executor-env-vars TEXT
                                  Set DefaultArguments.--customer-executor-env-
                                  vars.
  --python-modules-installer-option TEXT
                                  Set DefaultArguments.--python-modules-
                                  installer-option.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit params set --temp-dir s3://bucket/tmp --enable-metrics`

  - `gluekit params set my-job --add-default-argument survey_id=test`
```


### `gluekit params save --help`

```text
Usage: gluekit params save [OPTIONS] KEY VALUE

  Save a reusable value for the active job, profile, or all jobs.

Arguments:
  KEY    Saved parameter name.  [required]
  VALUE  Saved parameter value.  [required]

Options:
  --job TEXT          Override the active job.
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit params save default_arguments.--TempDir s3://my-bucket/tmp/`

  - `gluekit params save additional_python_modules "pydantic==2.11.7" --global`
```


### `gluekit params saved --help`

```text
Usage: gluekit params saved [OPTIONS] [JOB_NAME]

  Show stored local Glue config params.

Arguments:
  [JOB_NAME]  Optional Glue job name to view merged params (global + job).

Options:
  -p, --profile TEXT  Include params from this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit params saved my-job`

  - `gluekit params saved`
```


### `gluekit script --help`

```text
Usage: gluekit script [OPTIONS] COMMAND [ARGS]...

  Work with the active job's script.

Options:
  --help  Show this message and exit.

Commands:
  convert          Convert the script to a notebook.
  sync             Refresh generated script metadata from the local job...
  import-settings  Import represented script metadata into the local job...
  push             Upload the local script to AWS.
  pull             Download the remote script from AWS.
```


### `gluekit script convert --help`

```text
Usage: gluekit script convert [OPTIONS] [JOB_NAME]

  Convert the script to a notebook.

Arguments:
  [JOB_NAME]

Options:
  --use-config / --artifact-only  [default: use-config]
  --dry-run
  --config-dir PATH
  --help                          Show this message and exit.
```


### `gluekit script sync --help`

```text
Usage: gluekit script sync [OPTIONS] [JOB_NAME]

  Refresh generated script metadata from the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.
```


### `gluekit script import-settings --help`

```text
Usage: gluekit script import-settings [OPTIONS] [JOB_NAME]

  Import represented script metadata into the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.
```


### `gluekit script push --help`

```text
Usage: gluekit script push [OPTIONS] [JOB_NAME]

  Upload the local script to AWS.

Arguments:
  [JOB_NAME]

Options:
  --to TEXT                       [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.
```


### `gluekit script pull --help`

```text
Usage: gluekit script pull [OPTIONS] [JOB_NAME]

  Download the remote script from AWS.

Arguments:
  [JOB_NAME]

Options:
  --from TEXT                     [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.
```


### `gluekit notebook --help`

```text
Usage: gluekit notebook [OPTIONS] COMMAND [ARGS]...

  Work with the active job's notebook.

Options:
  --help  Show this message and exit.

Commands:
  convert          Convert the notebook to a script.
  sync             Refresh generated notebook metadata from the local job...
  import-settings  Import represented notebook metadata into the local job...
  push             Upload the local notebook to AWS.
  pull             Download the remote notebook from AWS.
```


### `gluekit notebook convert --help`

```text
Usage: gluekit notebook convert [OPTIONS] [JOB_NAME]

  Convert the notebook to a script.

Arguments:
  [JOB_NAME]

Options:
  --use-config / --artifact-only  [default: use-config]
  --dry-run
  --config-dir PATH
  --help                          Show this message and exit.
```


### `gluekit notebook sync --help`

```text
Usage: gluekit notebook sync [OPTIONS] [JOB_NAME]

  Refresh generated notebook metadata from the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.
```


### `gluekit notebook import-settings --help`

```text
Usage: gluekit notebook import-settings [OPTIONS] [JOB_NAME]

  Import represented notebook metadata into the local job config.

Arguments:
  [JOB_NAME]

Options:
  --dry-run
  --config-dir PATH
  --help             Show this message and exit.
```


### `gluekit notebook push --help`

```text
Usage: gluekit notebook push [OPTIONS] [JOB_NAME]

  Upload the local notebook to AWS.

Arguments:
  [JOB_NAME]

Options:
  --to TEXT                       [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.
```


### `gluekit notebook pull --help`

```text
Usage: gluekit notebook pull [OPTIONS] [JOB_NAME]

  Download the remote notebook from AWS.

Arguments:
  [JOB_NAME]

Options:
  --from TEXT                     [default: aws]
  --dry-run
  --config-dir PATH
  -p, --profile TEXT
  --auto-login / --no-auto-login  [default: auto-login]
  --help                          Show this message and exit.
```


### `gluekit push --help`

```text
Usage: gluekit push [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Transfer the full active job between local files and AWS.

Arguments:
  [JOB_NAME]    Glue job name or pattern to push. Use "*" for all configs.
  [TARGET_JOB]  Optional remote Glue job name to push to.

Options:
  --dry-run                       Show what would be uploaded without writing
                                  files.
  --update-config / --no-update-config
                                  Update Glue job configuration after uploading
                                  files.  [default: update-config]
  --build                         Build local package artifacts in the workspace
                                  root before push.
  --build-tool TEXT               Build frontend for --build: auto, uv, or
                                  build.  [default: auto]
  -v, --verbose                   Print local build command(s) before execution.
  --config-dir PATH               Directory containing Glue job config files.
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --to TEXT                       Transfer endpoint (aws).  [default: aws]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit push my-job --profile my-sso-profile`

  - `gluekit push source-job target-job --profile my-sso-profile`

  - `gluekit push "my-job-\*"`

  - `gluekit push my-job --build --build-tool auto --dry-run`
```


### `gluekit pull --help`

```text
Usage: gluekit pull [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Transfer the full active job between local files and AWS.

Arguments:
  [JOB_NAME]    Glue job name or pattern to pull. Use "*" for all jobs.
  [TARGET_JOB]  Optional local job name to write pulled artifacts to.

Options:
  -f, --force                     Download even if the local config is newer or
                                  equal to LastModifiedOn.
  --dry-run                       Show what would be downloaded without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
  --write-job-list / --no-write-job-list
                                  Write glue/glue_full_job_list.csv during the
                                  run.  [default: write-job-list]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --from TEXT                     Transfer endpoint (aws).  [default: aws]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit pull source-job target-job --profile my-sso-profile`

  - `gluekit pull "my-job-\*" --force`

  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`
```


### `gluekit init --help`

```text
Usage: gluekit init [OPTIONS]

  Initialize local repo-level Gluekit settings from AWS CLI profiles.

Options:
  --aws-config PATH               Local AWS CLI config file to scan for
                                  credential profile names; defaults to
                                  AWS_CONFIG_FILE or ~/.aws/config.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
                                  run 'aws sso login' when credentials are
                                  missing or expired.  [default: auto-login]
  --detect-roles / --no-detect-roles
                                  Fetch the selected real AWS Glue job in each
                                  profile and save its Role as local gluekit
                                  params.  [default: detect-roles]
  --detect-buckets / --no-detect-buckets
                                  Compare S3 URIs in real AWS Glue job configs
                                  across profiles and save local gluekit profile
                                  mappings.  [default: detect-buckets]
  --match-threshold INTEGER       Minimum rapidfuzz score for non-exact S3 key
                                  matches.  [default: 85]
  --baseline-profile TEXT         Selected AWS CLI profile to store as the first
                                  gluekit activate scope and S3 comparison
                                  baseline.
  --profile-job TEXT              Real AWS Glue job to inspect for an AWS CLI
                                  profile, as PROFILE=GLUE_JOB_NAME. May be
                                  repeated.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit init`
```


### `gluekit build --help`

```text
Usage: gluekit build [OPTIONS]

  Build local artifacts; only touches AWS when --push is used.

Options:
  --job-name TEXT                 Active or explicit local Glue job to update
                                  when using --package-whl or --push.
  --dry-run                       Show what would be built and zipped without
                                  running build commands.
  --bump-version, --bump TEXT     Bump pyproject.toml [project].version before
                                  building: major, minor, patch, or X.Y.Z.
  --build-tool TEXT               Build frontend to use: auto, uv, or build.
                                  [default: auto]
  --out-dir PATH                  The output directory to which distributions
                                  should be written.
  --no-version-suffix             Remove the version suffix from generated zip
                                  artifact names.
  -v, --verbose                   Print build commands before execution.
  --package-whl                   After building, replace the tracked dist/*.whl
                                  in the selected Glue config.
  --push                          Push the selected Glue config and artifacts
                                  after building.
  --update-config / --no-update-config
                                  For --push, update Glue job configuration
                                  after uploading files.  [default: update-
                                  config]
  --config-dir PATH               Directory containing Glue job config files.
  -p, --profile TEXT              AWS CLI credential profile for profile-scoped
                                  config params and, with --push, real AWS
                                  Glue/S3 API calls.
  --auto-login / --no-auto-login  For --push with a real AWS profile,
                                  automatically run 'aws sso login' when
                                  credentials are missing or expired.  [default:
                                  auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit build`

  - `gluekit build --build-tool uv`

  - `gluekit build --out-dir build/artifacts --no-version-suffix`

  - `gluekit build --build-tool build --dry-run`
```


### `gluekit local --help`

```text
Usage: gluekit local [OPTIONS] COMMAND [ARGS]...

  Local Glue fixtures and setup.

Options:
  --help  Show this message and exit.

Commands:
  setup   Create or update the local-only development setup.
  status  Show the local development fixture configuration.
  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...
```


### `gluekit run --help`

```text
Usage: gluekit run [OPTIONS] SCRIPT_PATH

  Run a Glue script locally with emulated Glue libraries and mocked AWS
  services.

Arguments:
  SCRIPT_PATH  Path to the Glue Python script to execute locally.  [required]

Options:
  --glue-arg TEXT       Glue script argument in KEY=VALUE form; passed through
                        as --KEY VALUE.
  --job-name TEXT       Default JOB_NAME used when not provided in script args.
                        [default: local-glue-job]
  --create-bucket TEXT  Create a mocked S3 bucket only inside the local run.
                        Repeat as needed.
  --ssm-parameter TEXT  Seed a mocked SSM parameter only inside the local run as
                        NAME=VALUE. Repeat as needed.
  --aws-region TEXT     Mock AWS region used for local boto3 clients. Defaults
                        to us-east-1.
  --config-file FILE    Local fixture config file for mocked S3/SSM. Defaults to
                        .gluekit/local.json when local settings are active.
  --runtime TEXT        Local DataFrame runtime: sqlframe (fast default) or
                        pyspark (real local Spark).  [default: sqlframe]
  --spark-master TEXT   Spark master URL used when --runtime=pyspark.  [default:
                        local[2]]
  --help                Show this message and exit.

  **Examples**

  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
  bucket`

  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
  report_date=2026-06-18`

  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
  --report_date 2026-06-18`
```
<!-- END: CLI_HELP -->

## Package releases

Install the repository task runner with `uv tool install poethepoet==0.48.0`,
then run `poe release-check` for validation and `poe build` for a local wheel
and source distribution. One-off usage is
`uvx --from poethepoet==0.48.0 poe release-check`.

The `publish-to-pypi.yml` workflow has three jobs: **prepare**, **publish**, and
**release**. Pushes to `dev` start or increment a patch development version
(`1.0.0 → 1.0.1.dev1 → 1.0.1.dev2`). Pushes to `main` promote a prerelease to
stable, or increment the patch when the source already has a stable version.
After a stable tag exists, `dev` starts the next patch series. Normal releases
should not edit the version manually. Merge updated `main` back into `dev`
when needed to keep the branches' version baselines aligned.

`poe release --branch dev --source <full-commit-sha>` runs `scripts/release.py`
in a clean checkout of that source. It uses `uv version --no-sync` to update
`pyproject.toml` and `uv.lock`, invokes `poe build`, and creates a local release
commit and annotated tag. This is the CI preparation command: it changes the
local checkout but does not push or publish. The workflow first saves the built
distributions, then atomically pushes the release commit and tag. Both branches
share one release concurrency group; a stale run fails rather than overwriting
newer work. Rapid pushes may supersede pending runs; the latest source should
be released. Workflow pushes use `GITHUB_TOKEN` and do not recursively trigger
another push workflow.

PyPI publication uses the `pypi` GitHub environment and Trusted Publishing bound
to this repository and **`publish-to-pypi.yml`**. No long-lived PyPI token is
needed. Only stable `main` versions get a GitHub Release, after PyPI succeeds.
The branch rules must permit the workflow's version commit; rejected pushes
leave both remote refs unchanged.

For a failed publish, use **Re-run failed jobs** on the original Actions run.
The publish job downloads the saved wheel and source distribution without
rebuilding. Identical PyPI uploads can be retried, including a partially
completed upload. A full rerun recognizes the tagged source/branch before
calculating a version and reuses its original artifact. Existing GitHub Releases
are left intact. If preparation failed before the atomic push, a fresh attempt
can rebuild and replace that run's unpublished artifact. If the original
artifact has expired or been deleted after the push, stop and recover those
exact files; the workflow deliberately does not rebuild a published version.

`poe release-test` exercises the release helper with temporary local Git remotes
and real uv version changes; distribution builds are mocked. No test publishes
packages or contacts cloud providers.

## Documentation development

Install Poe with `uv tool install poethepoet==0.48.0`, or use
`uvx --from poethepoet==0.48.0 poe <task>` for one-off commands.

- `poe docs-update`: regenerate committed documentation; review the diff and commit it.
- `poe docs-check`: detect stale generated documentation without changing it.
- `poe docs-build`: build the site with MkDocs strict mode.
- `poe docs-validate`: run both documentation checks and the strict build.
- `poe docs`: regenerate documentation and build the site for local development.

Documentation tasks use dependencies from `uv.lock` with `--frozen`; update the
lockfile intentionally when changing dependencies. CI calls the same Poe tasks,
never regenerates or auto-commits documentation, and validates docs through
`poe release-check` for both pull requests and package releases. Documentation
hosting remains a separate deployment concern.
