Metadata-Version: 2.4
Name: gluekit
Version: 1.0.1.dev1
Summary: A developer‑centric Glue CLI built to speed local iteration with git/uv‑style commands, while bridging to highly controlled (eg Fedramped) AWS consoles for PII‑sensitive runs and smoothing onboarding around Glue’s quirks—like the lack of script‑to‑notebook conversion and the friction of refactoring Glue scripts.
Author-email: Mike Kranz <kranz-michael@norc.org>
Keywords: python
Requires-Python: <4.0,>=3.10
Description-Content-Type: text/markdown
Requires-Dist: boto3>=1.28.0
Requires-Dist: python-dotenv>=1.2.1
Requires-Dist: python-slugify>=8.0.4
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rapidfuzz>=3.14.3
Requires-Dist: typer>=0.9.0
Provides-Extra: job-dev
Requires-Dist: moto[s3,ssm]>=5.1.16; extra == "job-dev"
Requires-Dist: sqlframe[duckdb]>=4.3.0; extra == "job-dev"

# gluekit

## Introduction

`gluekit` is reusable AWS Glue job tooling for local-first development. It is designed for teams that want Glue job definitions, scripts, notebooks, and supporting artifacts in source control while still being able to pull from AWS, edit safely, validate locally, and push intentional changes back.

The command line is meant to feel closer to `git` and `uv` than to a cloud console: quick local commands, explicit artifact layouts, checkout-style context, and conversion workflows that smooth over AWS Glue quirks without hiding the underlying AWS behavior. A major motivation is making script and notebook work interchangeable enough for day-to-day development while still respecting that Glue scripts and Glue interactive notebooks are not identical runtime surfaces.

**Local Development Emulators:** `gluekit` provides scoped `awsglue` and `pyspark` emulators during local runs, using [SQLFrame](https://sqlframe.readthedocs.io/) when available, so you can test jobs rapidly on your local machine without AWS compute. See the [Local Development](docs/local_development.md) documentation for more details.

## Suggested workflows

For AWS CLI-oriented setup, sync, and deployment recipes, start with the [AWS CLI recipes](docs/aws_cli_recipes/) folder, especially [docs/aws_cli_recipes/workflows.md](docs/aws_cli_recipes/workflows.md).

Common starting points:

- Pull a real Glue job into source control, edit the local script/config, and push selected artifacts back with `gluekit pull`, `gluekit checkout`, `gluekit edit`, and `gluekit push`.
- Use `gluekit local setup`, `gluekit local s3`, `gluekit local ssm`, and `gluekit run` to build a repeatable local test fixture before touching AWS.
- Keep account-specific values such as IAM roles, temp buckets, and script buckets in profile-scoped saved params so local configs can move between dev/prod-like AWS profiles without hand-editing generated artifacts.

## Script-based development with notebook interoperability

`gluekit` supports a script-first workflow that can still interoperate with notebooks. Python scripts can use `#%%` cell tags to mark notebook-like cells. These tags divide a `.py` file into executable notebook cells; Markdown cells can use `#%% [markdown]`. This follows the same family of conventions described by VS Code's [Python Interactive/Jupyter documentation](https://code.visualstudio.com/docs/python/jupyter-support-py#_convert-jupyter-notebooks-to-python-code-file), where Python files can contain Jupyter-style cell delimiters and be converted back into notebooks.

The intent is flexibility: you can develop in a normal Python file, use editor support for cell-by-cell execution, keep clean script diffs in git, and generate a Glue notebook when that is the deployment or debugging surface you need.

If you have already selected the job with checkout:

```bash
gluekit checkout {jobname}
gluekit convert --from script --to notebook
```

If you have not checked out the job, pass it explicitly:

```bash
gluekit convert {jobname} --from script --to notebook
```

## Gluekit vs AWS CLI terminology

Some `gluekit` commands look like AWS CLI commands, but the scope is different:

- `gluekit pull`, `gluekit push`, `gluekit build --push`, and `gluekit init` use real AWS Glue/S3 APIs. On those commands, `--profile` means an AWS CLI credential profile, the same profile name boto3 would use.
- `gluekit checkout --profile` and `gluekit config set --profile` update local gluekit state in `.gluekit/glue_set.json`. The value is a gluekit profile scope for saved params; it usually matches an AWS profile name, but those commands do not contact AWS.
- `gluekit local ...` and `gluekit run` are local test tooling. Their S3, SSM, region, job, and profile values are labels or mocked fixture values stored in JSON and used by local moto/boto3 mocks. They do not create, read, or delete real AWS resources.

## Install

### If using `uv` as a global tool (recommended)

Ensure you have `uv` installed globally (see [uv installation docs](https://uv.pypa.io/en/latest/installation.html#installing-uv-globally)) and then run:


```bash
uv tool install https://github.com/NORCUChicago/gluekit.git
```

### If using `pip`:

```bash
pip install https://github.com/NORCUChicago/gluekit.git

```

## Quick start

1. Install the AWS command line tool and configure your credentials. If using sso, need to log in with 
```bash
aws sso login --profile your-profile-name
```


2. Right now, the best way to get started is to check out the [Command line help documentation](#command-line-help-documentation) below, which is auto-generated from the live CLI `--help` output and includes usage examples for each command.

## Quick start commands

### Glue job management

Initialize a workspace from your AWS CLI credential profiles:

```bash
gluekit init
```

Pull or check out the job you want to work on, then make focused config changes:

```bash
gluekit checkout my-job --profile survey-dev
gluekit edit --role arn:aws:iam::111111111111:role/glue-dev --temp-dir s3://dev-bucket/tmp/
```

Add local artifacts or shared files to the checked-out job:

```bash
gluekit add dist/my_pkg-0.0.1-py3-none-any.whl --package-whl
gluekit add glue/lib/helpers.py --as-path
```

Convert between Glue scripts and notebooks when your workflow changes:

```bash
gluekit convert my-job --from script --to notebook
gluekit convert my-job --from notebook --to script
```

Build Python artifacts and optionally push the checked-out job updates:

```bash
gluekit build
gluekit build --bump-version patch --package-whl --push
```

### Local Glue job development

Create and check out the local setup with mocked AWS resource mappings:

```bash
gluekit local setup \
  --profile local-dev \
  --s3-root sources/s3 \
  --s3-bucket local-dev=tests/fixtures \
  --ssm-param /app/runtime=local
```

Inspect the saved setup, add SSM parameters, and run the Glue script locally:

```bash
gluekit local status
gluekit local ssm put /app/runtime local
gluekit run glue/scripts/my-job.py --glue-arg run_date=2026-06-18
```

Use `--s3-root sources/s3` to map an entire mocked profile/account S3 namespace, where `sources/s3/{bucket}/{prefix/key}` appears as `s3://{bucket}/{prefix/key}`. Use `--s3-bucket` as `{s3 bucket in mocked profile}={local directory}` for a single bucket, and `--ssm-param` as `{ssm parameter in mocked profile}={local parameter value}`.

Local setup writes fixture data to `.gluekit/local.json` and records the active local settings under `checkout.local` in `.gluekit/glue_set.json`. The local `--profile` value is saved in both places as a cross-reference label for comparison and validation; `gluekit local setup` does not authenticate to AWS.

## Generated metadata

gluekit now includes two checked-in metadata artifacts used by notebook conversion and Glue job validation helpers:

- `src/gluekit/resources/glue_job_schema.json` is generated from the local botocore AWS Glue service model.
- `src/gluekit/resources/magic_map.json` is a curated mapping of Glue job config fields to AWS Glue Interactive Sessions magics.

The schema artifact is generated by:

```bash
uv run python scripts/extract_glue_schema.py
```

Rerun that script when changing schema-driven validation logic, when upgrading boto3/botocore in a way that could change the Glue service model, or when you need to refresh the checked-in schema after AWS Glue API changes.

The magic map is intentionally maintained by hand because it reflects documented notebook behavior rather than raw AWS API structure. When updating notebook conversion behavior, review the AWS Glue Interactive Sessions magics docs and keep `src/gluekit/resources/magic_map.json` in sync.

## edit workflow

`gluekit edit` is checkout-first. The default flow is to select a job once with `gluekit checkout my-job` and then run `gluekit edit ...` against that active local context.

Use `gluekit add` when you want the older inferred artifact behavior for local paths, wheels, zip files, and remote module specifiers. Use `gluekit update` when you need a raw property-path setter. Both remain available for compatibility, but `gluekit edit` is the primary mutation surface.

For the common package-wheel flow, run `gluekit add --package-whl` after building the project. It uses the single wheel matching `dist/*.whl`, routes it to `--additional-python-modules`, replaces the previously tracked `dist/*.whl` package wheel, updates `SourceControlDetails.AdditionalPythonFiles`, and refreshes an existing notebook's generated Glue magic cells.

If you need to target a different local config for one command, pass the job name directly to `gluekit edit <job-name> ...`, or keep using `gluekit add --job-name other-job <items...>` for the compatibility workflow. That stays parallel to GitHub CLI's context-first workflow, where a flag like `--repo` overrides the ambient target without changing the saved default context.

## Profile-specific params and build automation

Use profile-scoped saved params for account-dependent values such as IAM roles, script buckets, temp buckets, and other Glue default arguments. These are local gluekit param scopes saved in `.gluekit/glue_set.json`; they are only used as real AWS credential profiles by commands that call AWS, such as `pull`, `push`, `build --push`, and `init`.

```bash
gluekit config set my-job --profile survey-dev --Role=arn:aws:iam::111111111111:role/glue-dev
gluekit config set my-job --profile survey-prod --Role=arn:aws:iam::222222222222:role/glue-prod
gluekit config set my-job --profile survey-prod --command_script_location=s3://prod-bucket/scripts/my-job.py
```

Then check out the job and gluekit profile scope together:

```bash
gluekit checkout my-job --profile survey-prod
```

After a job is checked out, switch only the active profile scope with:

```bash
gluekit checkout --profile survey-dev
```

Checkout stores the active gluekit profile scope in `.gluekit/glue_set.json` and applies the matching saved params to the local config immediately, including mapped Glue config paths such as `Role`, `Command.ScriptLocation`, `DefaultArguments.*`, and `SourceControlDetails.*`. `gluekit pull` validates existing active profile mappings against the pulled config; if the active profile has no mappings yet but another profile exists for the job, pull detects and saves role plus S3/config mappings using the same profile comparison rules as `gluekit init`. `gluekit push` validates that the active profile mappings already align with the local config before uploading, and reports every mismatch instead of rewriting the config at push time. `gluekit pull`, `gluekit push`, and `gluekit build --push` also use the active profile value as the AWS CLI credential profile when `--profile` is not passed.

The common package release loop can now be a single command:

```bash
gluekit build --bump-version patch --package-whl --push
```

That bumps `pyproject.toml`, builds artifacts, refreshes the checked-out job's tracked `dist/*.whl` in `--additional-python-modules`, applies active profile params before deriving S3 paths, and pushes the selected Glue artifacts/config. Use `--dry-run` to preview the build, wheel update, and push steps.

For first-time repository setup, `gluekit init` scans your AWS CLI config profiles, prompts for the Glue job to inspect in each selected profile, can save profile-specific role and S3 params locally, and sets the first checked-out job/profile scope.

### Command line help documentation
<!-- BEGIN: CLI_HELP -->
> Auto-generated from live CLI `--help` output.

### `gluekit --help`

```text
Usage: gluekit [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  build     Build local artifacts; only touches AWS when --push is used.
  checkout  Replace the active local checkout selection.
  edit      Edit one local Glue job config with explicit mutation flags.
  clone     Clone local Glue job configurations and artifacts.
  set       Set reusable local key/value parameters for a Glue job.
  convert   Convert Glue jobs between script and notebook formats.
  init      Initialize local repo-level Gluekit settings from AWS CLI...
  list      List real AWS Glue jobs in the configured AWS account.
  pull      Download real AWS Glue job artifacts and config into the local...
  push      Upload local artifacts to real AWS Glue/S3 and optionally...
  run       Run a Glue script locally with emulated Glue libraries and...
  remove    Remove local Glue job configs and local artifacts.
  status    Show the active local checkout and its local-to-S3 artifact...
  sync      Sync generated Glue config metadata cells in local...
  add       Compatibility command for inferred artifact and module additions.
  update    Compatibility command for generic property updates.
  config    Show and set local reusable Glue config parameters.
  local     Manage local-only Glue development setups and mocked AWS fixtures.
```


### `gluekit pull --help`

```text
Usage: gluekit pull [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Download real AWS Glue job artifacts and config into the local workspace.

Arguments:
  [JOB_NAME]    Glue job name or pattern to pull. Use "*" for all jobs.
  [TARGET_JOB]  Optional local job name to write pulled artifacts to.

Options:
  -f, --force                     Download even if the local config is newer or
                                  equal to LastModifiedOn.
  --dry-run                       Show what would be downloaded without writing
                                  files.
  -i, --include TEXT              Include only specific components (config,
                                  script, notebook, extra-py-files, extra-
                                  files).
  -x, --exclude TEXT              Exclude specific components (config, script,
                                  notebook, extra-py-files, extra-files).
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --write-job-list / --no-write-job-list
                                  Write glue/glue_full_job_list.csv during the
                                  run.  [default: write-job-list]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit pull source-job target-job --profile my-sso-profile`

  - `gluekit pull "my-job-\*" --force --include config,script --exclude
  notebook`

  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`
```


### `gluekit edit --help`

```text
Usage: gluekit edit [OPTIONS] [JOB_NAME]

  Edit one local Glue job config with explicit mutation flags.

Arguments:
  [JOB_NAME]  Glue job name to edit. Defaults to the active checkout selection.

Options:
  --description TEXT              Set the job description.
  --role TEXT                     Set the IAM role for the job.
  --glue-version TEXT             Set the Glue version.
  --worker-type TEXT              Set the worker type.
  --number-of-workers INTEGER     Set the number of workers.
  --timeout INTEGER               Set the job timeout in minutes.
  --max-retries INTEGER           Set the max retry count.
  --execution-class TEXT          Set the execution class.
  --max-concurrent-runs INTEGER   Set the max concurrent runs.
  --job-run-queuing-enabled / --no-job-run-queuing-enabled
                                  Enable or disable job run queuing.
  --security-configuration TEXT   Set the security configuration name.
  --maintenance-window TEXT       Set the maintenance window.
  --log-uri TEXT                  Set the log URI.
  --command-name TEXT             Set Command.Name.
  --python-version TEXT           Set Command.PythonVersion.
  --runtime TEXT                  Set Command.Runtime.
  --remote-script-location TEXT   Set Command.ScriptLocation.
  --local-script-location TEXT    Set SourceControlDetails.ScriptLocation.
  --notebook-location TEXT        Set SourceControlDetails.NotebookLocation and
                                  NotebookPath.
  --source-control-provider TEXT  Set SourceControlDetails.Provider.
  --source-control-owner TEXT     Set SourceControlDetails.Owner.
  --source-control-repository TEXT
                                  Set SourceControlDetails.Repository.
  --source-control-branch TEXT    Set SourceControlDetails.Branch.
  --source-control-folder TEXT    Set SourceControlDetails.Folder.
  --add-connection TEXT           Add one or more Glue connections.
  --remove-connection TEXT        Remove one or more Glue connections.
  --add-additional-python-modules TEXT
                                  Add one or more values to --additional-python-
                                  modules.
  --remove-additional-python-modules TEXT
                                  Remove one or more values from --additional-
                                  python-modules.
  --add-extra-py-files TEXT       Track one or more local relative Python files
                                  in SourceControlDetails.ExtraPyFiles and
                                  --extra-py-files.
  --remove-extra-py-files TEXT    Remove one or more local relative Python files
                                  from SourceControlDetails.ExtraPyFiles and
                                  --extra-py-files.
  --add-extra-jars TEXT           Add one or more values to --extra-jars.
  --remove-extra-jars TEXT        Remove one or more values from --extra-jars.
  --add-additional-python-files TEXT
                                  Track one or more local paths in
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --remove-additional-python-files TEXT
                                  Remove one or more local paths from
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --add-extra-files TEXT          Track one or more local paths in
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --remove-extra-files TEXT       Remove one or more local paths from
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --add-default-argument TEXT     Set a DefaultArguments entry using KEY=VALUE.
                                  Repeat as needed.
  --remove-default-argument TEXT  Remove a DefaultArguments entry by key. Repeat
                                  as needed.
  --add-non-overridable-argument TEXT
                                  Set a NonOverridableArguments entry using
                                  KEY=VALUE. Repeat as needed.
  --remove-non-overridable-argument TEXT
                                  Remove a NonOverridableArguments entry by key.
                                  Repeat as needed.
  --enable-metrics / --disable-metrics
                                  Set DefaultArguments.--enable-metrics to true
                                  or false.
  --enable-observability-metrics / --disable-observability-metrics
                                  Set DefaultArguments.--enable-observability-
                                  metrics to true or false.
  --enable-glue-datacatalog / --disable-glue-datacatalog
                                  Set DefaultArguments.--enable-glue-datacatalog
                                  to true or false.
  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
                                  Set DefaultArguments.--enable-continuous-
                                  cloudwatch-log to true or false.
  --enable-spark-ui / --disable-spark-ui
                                  Set DefaultArguments.--enable-spark-ui to true
                                  or false.
  --enable-job-insights / --disable-job-insights
                                  Set DefaultArguments.--enable-job-insights to
                                  true or false.
  --enable-auto-scaling / --disable-auto-scaling
                                  Set DefaultArguments.--enable-auto-scaling to
                                  true or false.
  --job-bookmark-option TEXT      Set DefaultArguments.--job-bookmark-option.
  --temp-dir TEXT                 Set DefaultArguments.--TempDir.
  --spark-event-logs-path TEXT    Set DefaultArguments.--spark-event-logs-path.
  --datalake-formats TEXT         Set DefaultArguments.--datalake-formats.
  --spark-conf TEXT               Set DefaultArguments.--conf.
  --continuous-log-log-group TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logGroup.
  --continuous-log-log-stream-prefix TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logStreamPrefix.
  --continuous-log-conversion-pattern TEXT
                                  Set DefaultArguments.--continuous-log-
                                  conversionPattern.
  --executor-cores INTEGER        Set DefaultArguments.--executor-cores.
  --customer-driver-env-vars TEXT
                                  Set DefaultArguments.--customer-driver-env-
                                  vars.
  --customer-executor-env-vars TEXT
                                  Set DefaultArguments.--customer-executor-env-
                                  vars.
  --python-modules-installer-option TEXT
                                  Set DefaultArguments.--python-modules-
                                  installer-option.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit edit my-job --description 'Updated job description' --glue-version
  5.0`

  - `gluekit edit --add-additional-python-files glue/libs/helper.py --add-extra-
  files glue/resources/job.json`

  - `gluekit edit my-job --temp-dir s3://bucket/tmp --remove-default-argument
  legacy-arg --enable-metrics`

  - `gluekit edit my-job --enable-auto-scaling --enable-job-insights --executor-
  cores 8`
```


### `gluekit add --help`

```text
Usage: gluekit add [OPTIONS] [ARGS]...

  Compatibility command for inferred artifact and module additions. Prefer edit
  for explicit config changes.

Arguments:
  [ARGS]...  Paths or module specifiers to add to the selected Glue job config.

Options:
  --job-name TEXT    Override the active checked-out job for this command only.
  --as-path          Treat all items as local paths.
  --as-pypi          Treat all items as PyPI modules.
  --package-whl      Use the package wheel from dist/*.whl.
  --dry-run          Show what would be updated without writing files.
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit checkout my-job`

  - `gluekit add dist/my-package.whl`

  - `gluekit add --job-name other-job requests==2.32.3 --as-pypi`
```


### `gluekit convert --help`

```text
Usage: gluekit convert [OPTIONS] [JOB_NAME]

  Convert Glue jobs between script and notebook formats.

Arguments:
  [JOB_NAME]  Glue job name to convert. Defaults to the active checkout
              selection.

Options:
  --from TEXT                   Source format (script or notebook).  [default:
                                script]
  --to TEXT                     Target format (script or notebook).  [default:
                                notebook]
  --use-config / --script-only  Use config metadata (magics and paths) when
                                available.  [default: use-config]
  --dry-run                     Show what would be converted without writing
                                files.
  --config-dir PATH             Directory containing Glue job config files.
                                [default: glue/configs]
  --help                        Show this message and exit.

  **Examples**

  - `gluekit convert my-job --from script --to notebook --use-config`

  - `gluekit convert my-job --from notebook --to script --script-only`
```


### `gluekit sync --help`

```text
Usage: gluekit sync [OPTIONS] [JOB_NAME]

  Sync generated Glue config metadata cells in local script/notebook files.

Arguments:
  [JOB_NAME]  Glue job name to sync. Defaults to the active checkout selection.

Options:
  --dry-run          Show what would be updated without writing files.
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit sync my-job`

  - `gluekit sync my-job --dry-run`
```


### `gluekit status --help`

```text
Usage: gluekit status [OPTIONS]

  Show the active local checkout and its local-to-S3 artifact mappings.

Options:
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit status`

  - `gluekit status --config-dir glue/configs`
```


### `gluekit run --help`

```text
Usage: gluekit run [OPTIONS] SCRIPT_PATH

  Run a Glue script locally with emulated Glue libraries and mocked AWS
  services.

Arguments:
  SCRIPT_PATH  Path to the Glue Python script to execute locally.  [required]

Options:
  --glue-arg TEXT       Glue script argument in KEY=VALUE form; passed through
                        as --KEY VALUE.
  --job-name TEXT       Default JOB_NAME used when not provided in script args.
                        [default: local-glue-job]
  --create-bucket TEXT  Create a mocked S3 bucket only inside the local run.
                        Repeat as needed.
  --ssm-parameter TEXT  Seed a mocked SSM parameter only inside the local run as
                        NAME=VALUE. Repeat as needed.
  --aws-region TEXT     Mock AWS region used for local boto3 clients. Defaults
                        to us-east-1.
  --config-file FILE    Local fixture config file for mocked S3/SSM. Defaults to
                        .gluekit/local.json when local settings are checked out.
  --help                Show this message and exit.

  **Examples**

  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
  bucket`

  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
  report_date=2026-06-18`

  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
  --report_date 2026-06-18`
```


### `gluekit local --help`

```text
Usage: gluekit local [OPTIONS] COMMAND [ARGS]...

  Manage local-only Glue development setups and mocked AWS fixtures.

Options:
  --help  Show this message and exit.

Commands:
  setup   Create or update and check out the local-only development setup.
  status  Show the active local development setup.
  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...
```


### `gluekit local setup --help`

```text
Usage: gluekit local setup [OPTIONS]

  Create or update and check out the local-only development setup.

Options:
  --profile TEXT    Optional gluekit/AWS profile name to associate with this
                    local setup in checkout state and .gluekit/local.json; this
                    command does not authenticate to AWS.
  --s3-bucket TEXT  Map a mocked S3 bucket to a local directory in {s3 bucket in
                    mocked profile}={local directory} form. Repeat as needed.
  --s3-root PATH    Map a local directory as the full mocked S3 namespace for
                    this setup/profile. Expected layout: {local s3
                    root}/{bucket}/{prefix/key}.
  --ssm-param TEXT  Map a mocked SSM parameter to a local value in {ssm
                    parameter in mocked profile}={local parameter value} form.
                    Repeat as needed.
  --help            Show this message and exit.

  **Examples**

  - `gluekit local setup --profile local-dev --s3-root sources/s3`

  - `gluekit local setup --s3-bucket local-dev=tests/fixtures`

  - `gluekit local setup --ssm-param /app/runtime=local`
```


### `gluekit local status --help`

```text
Usage: gluekit local status [OPTIONS]

  Show the active local development setup.

Options:
  --help  Show this message and exit.

  **Examples**

  - `gluekit local status`
```


### `gluekit local s3 --help`

```text
Usage: gluekit local s3 [OPTIONS] COMMAND [ARGS]...

  Manage local mocked S3 fixture mappings; does not call AWS S3.

Options:
  --help  Show this message and exit.

Commands:
  cp       Copy a local file or mocked S3 object in the fixture config;...
  mount    Map a local directory recursively into mocked S3 keys; does not...
  unmount  Remove a recursive local-directory-to-mocked-S3 mapping.
  mv       Move a local file or mocked S3 object in the fixture config;...
  rm       Remove mapped mocked S3 objects from the local fixture config.
  ls       List mocked S3 buckets inferred from mappings, or mapped objects.
```


### `gluekit local s3 cp --help`

```text
Usage: gluekit local s3 cp [OPTIONS] SOURCE DESTINATION

  Copy a local file or mocked S3 object in the fixture config; does not call
  AWS.

Arguments:
  SOURCE       Local path or mocked S3 URI source.  [required]
  DESTINATION  Local path or mocked S3 URI destination.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 cp tests/fixtures/input.json s3://my-bucket/input.json`

  - `gluekit local s3 cp s3://my-bucket/input.json s3://my-bucket/copy.json`

  - `gluekit local s3 cp s3://my-bucket/input.json tmp/input.json`
```


### `gluekit local s3 mount --help`

```text
Usage: gluekit local s3 mount [OPTIONS] SOURCE_DIR DESTINATION

  Map a local directory recursively into mocked S3 keys; does not call AWS.

Arguments:
  SOURCE_DIR   Local directory to map recursively.  [required]
  DESTINATION  Mocked S3 bucket or prefix base for mapped files.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 mount tests/fixtures s3://local-dev/input/`

  - `gluekit local s3 mount data/mock s3://bucket-a`
```


### `gluekit local s3 unmount --help`

```text
Usage: gluekit local s3 unmount [OPTIONS] TARGET

  Remove a recursive local-directory-to-mocked-S3 mapping.

Arguments:
  TARGET  Local source directory or mocked S3 destination to remove.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 unmount tests/fixtures`

  - `gluekit local s3 unmount s3://local-dev/input/`
```


### `gluekit local s3 mv --help`

```text
Usage: gluekit local s3 mv [OPTIONS] SOURCE DESTINATION

  Move a local file or mocked S3 object in the fixture config; does not call
  AWS.

Arguments:
  SOURCE       Local path or mocked S3 URI source.  [required]
  DESTINATION  Local path or mocked S3 URI destination.  [required]

Options:
  --delete-local      Delete the local source file after mapping a LocalPath to
                      S3.
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 mv s3://my-bucket/input.json s3://my-bucket/renamed.json`

  - `gluekit local s3 mv s3://my-bucket/input.json tmp/input.json`
```


### `gluekit local s3 rm --help`

```text
Usage: gluekit local s3 rm [OPTIONS] URI

  Remove mapped mocked S3 objects from the local fixture config.

Arguments:
  URI  Mapped mocked S3 object or prefix to remove.  [required]

Options:
  -r, --recursive     Remove all mapped objects under the S3 prefix.
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 rm s3://my-bucket/input.json`

  - `gluekit local s3 rm s3://my-bucket/prefix/ --recursive`
```


### `gluekit local ssm --help`

```text
Usage: gluekit local ssm [OPTIONS] COMMAND [ARGS]...

  Manage local mocked SSM Parameter Store values; does not call AWS SSM.

Options:
  --help  Show this message and exit.

Commands:
  put  Create or update a mocked SSM parameter with local-only syntax.
  get  Show one mocked SSM parameter with local-only syntax.
  rm   Delete one mocked SSM parameter with local-only syntax.
  ls   List mocked SSM parameters with local-only syntax.
```


### `gluekit local ssm put --help`

```text
Usage: gluekit local ssm put [OPTIONS] NAME VALUE

  Create or update a mocked SSM parameter with local-only syntax.

Arguments:
  NAME   Mocked SSM parameter name.  [required]
  VALUE  Mocked SSM parameter value.  [required]

Options:
  --overwrite / --no-overwrite  Overwrite an existing parameter.  [default:
                                overwrite]
  --config-file PATH            Local fixture config file to update. Defaults to
                                .gluekit/local.json.
  --help                        Show this message and exit.

  **Examples**

  - `gluekit local ssm put /app/env dev`
```


### `gluekit build --help`

```text
Usage: gluekit build [OPTIONS]

  Build local artifacts; only touches AWS when --push is used.

Options:
  --job-name TEXT                 Checked-out or explicit local Glue job to
                                  update when using --package-whl or --push.
  --dry-run                       Show what would be built and zipped without
                                  running build commands.
  --bump-version, --bump TEXT     Bump pyproject.toml [project].version before
                                  building: major, minor, patch, or X.Y.Z.
  --build-tool TEXT               Build frontend to use: auto, uv, or build.
                                  [default: auto]
  --out-dir PATH                  The output directory to which distributions
                                  should be written.
  --no-version-suffix             Remove the version suffix from generated zip
                                  artifact names.
  -v, --verbose                   Print build commands before execution.
  --package-whl                   After building, replace the tracked dist/*.whl
                                  in the selected Glue config.
  --push                          Push the selected Glue config and artifacts
                                  after building.
  -i, --include TEXT              For --push, include only specific components
                                  (script, notebook, additional-python-modules,
                                  extra-files, job-config).
  -x, --exclude TEXT              For --push, exclude specific components
                                  (script, notebook, additional-python-modules,
                                  extra-files, job-config).
  --update-config / --no-update-config
                                  For --push, update Glue job configuration
                                  after uploading files.  [default: update-
                                  config]
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile TEXT              AWS CLI credential profile for profile-scoped
                                  config params and, with --push, real AWS
                                  Glue/S3 API calls.
  --auto-login / --no-auto-login  For --push with a real AWS profile,
                                  automatically run 'aws sso login' when
                                  credentials are missing or expired.  [default:
                                  auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit build`

  - `gluekit build --build-tool uv`

  - `gluekit build --out-dir build/artifacts --no-version-suffix`

  - `gluekit build --build-tool build --dry-run`
```


### `gluekit push --help`

```text
Usage: gluekit push [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Upload local artifacts to real AWS Glue/S3 and optionally update the job
  config.

Arguments:
  [JOB_NAME]    Glue job name or pattern to push. Use "*" for all configs.
  [TARGET_JOB]  Optional remote Glue job name to push to.

Options:
  --dry-run                       Show what would be uploaded without writing
                                  files.
  -i, --include TEXT              Include only specific components (script,
                                  notebook, additional-python-modules, extra-
                                  files, job-config).
  -x, --exclude TEXT              Exclude specific components (script, notebook,
                                  additional-python-modules, extra-files, job-
                                  config).
  --update-config / --no-update-config
                                  Update Glue job configuration after uploading
                                  files.  [default: update-config]
  --build                         Build local package artifacts in the workspace
                                  root before push.
  --build-tool TEXT               Build frontend for --build: auto, uv, or
                                  build.  [default: auto]
  -v, --verbose                   Print local build command(s) before execution.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit push my-job --profile my-sso-profile`

  - `gluekit push source-job target-job --profile my-sso-profile`

  - `gluekit push "my-job-\*" --include script,job-config --exclude notebook`

  - `gluekit push my-job --build --build-tool auto --dry-run`
```


### `gluekit set --help`

```text
Usage: gluekit set [OPTIONS] [JOB_NAME]

  Set reusable local key/value parameters for a Glue job.

Arguments:
  [JOB_NAME]  Glue job name to save params for.

Options:
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit set my-job --script_location glue/scripts/my-job.py --extra_files
  s3://my-bucket/shared/config.json`

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit set --global --additional_python_modules "pydantic==2.11.7"`

  - `gluekit push "my-job-\*" --profile my-sso-profile`

  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
  bucket/tmp/`

  - `gluekit config get my-job`

  - `gluekit config get`

  - `gluekit set my-job --extra_py_files s3://my-
  bucket/dist/gluekit-0.0.1-py3-none-any.whl`

  - `gluekit push my-job --build --build-tool auto --profile my-sso-profile`
```


### `gluekit config --help`

```text
Usage: gluekit config [OPTIONS] COMMAND [ARGS]...

  Show and set local reusable Glue config parameters.

Options:
  --help  Show this message and exit.

Commands:
  set  Set reusable local Glue config params by job or globally.
  get  Show stored local Glue config params.
```


### `gluekit config set --help`

```text
Usage: gluekit config set [OPTIONS] [JOB_NAME]

  Set reusable local Glue config params by job or globally.

Arguments:
  [JOB_NAME]  Glue job name to save params for.

Options:
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
  bucket/tmp/`

  - `gluekit config set --global --additional_python_modules "pydantic==2.11.7"`
```


### `gluekit config get --help`

```text
Usage: gluekit config get [OPTIONS] [JOB_NAME]

  Show stored local Glue config params.

Arguments:
  [JOB_NAME]  Optional Glue job name to view merged params (global + job).

Options:
  -p, --profile TEXT  Include params from this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit config get my-job`

  - `gluekit config get`
```


### `gluekit list --help`

```text
Usage: gluekit list [OPTIONS]

  List real AWS Glue jobs in the configured AWS account.

Options:
  --help  Show this message and exit.

  **Examples**

  - `gluekit list --help`

  - `gluekit init --help`
```


### `gluekit init --help`

```text
Usage: gluekit init [OPTIONS]

  Initialize local repo-level Gluekit settings from AWS CLI profiles.

Options:
  --aws-config PATH               Local AWS CLI config file to scan for
                                  credential profile names; defaults to
                                  AWS_CONFIG_FILE or ~/.aws/config.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
                                  run 'aws sso login' when credentials are
                                  missing or expired.  [default: auto-login]
  --detect-roles / --no-detect-roles
                                  Fetch the selected real AWS Glue job in each
                                  profile and save its Role as local gluekit
                                  params.  [default: detect-roles]
  --detect-buckets / --no-detect-buckets
                                  Compare S3 URIs in real AWS Glue job configs
                                  across profiles and save local gluekit profile
                                  mappings.  [default: detect-buckets]
  --match-threshold INTEGER       Minimum rapidfuzz score for non-exact S3 key
                                  matches.  [default: 85]
  --baseline-profile TEXT         Selected AWS CLI profile to store as the first
                                  gluekit checkout scope and S3 comparison
                                  baseline.
  --profile-job TEXT              Real AWS Glue job to inspect for an AWS CLI
                                  profile, as PROFILE=GLUE_JOB_NAME. May be
                                  repeated.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit init`
```
<!-- END: CLI_HELP -->

## Getting started with your project

### 1. Create a New Repository

First, create a repository on GitHub with the same name as this project, and then run the following commands:

```bash
git init -b main
git add .
git commit -m "init commit"
git remote add origin git@github.com:mbkranz/gluekit.git
git push -u origin main
```

### 2. Set Up Your Development Environment

Then, install the environment and the pre-commit hooks with

```bash
make install
```

This will also generate your `uv.lock` file

### 3. Run the pre-commit hooks

Initially, the CI/CD pipeline might be failing due to formatting issues. To resolve those run:

```bash
uv run pre-commit run -a
```

### 4. Commit the changes

Lastly, commit the changes made by the two steps above to your repository.

```bash
git add .
git commit -m 'Fix formatting issues'
git push origin main
```

You are now ready to start development on your project!
The CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.

To finalize the set-up for publishing to PyPI, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/publishing/#set-up-for-pypi).
For activating the automatic documentation with MkDocs, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/mkdocs/#enabling-the-documentation-on-github).
To enable the code coverage reports, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/codecov/).

## Releasing a new version

- Create an API Token on [PyPI](https://pypi.org/).
- Add the API Token to your projects secrets with the name `PYPI_TOKEN` by visiting [this page](https://github.com/mbkranz/gluekit/settings/secrets/actions/new).
- Create a [new release](https://github.com/mbkranz/gluekit/releases/new) on Github.
- Create a new tag in the form `*.*.*`.

For more details, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/cicd/#how-to-trigger-a-release).

---

Repository initiated with [fpgmaas/cookiecutter-uv](https://github.com/fpgmaas/cookiecutter-uv).
