Metadata-Version: 2.3
Name: pinexq-cli
Version: 0.4.2
Summary: pinexq command-line interface
Author: Sebastian Höfer, Sebastian Boerakker
Author-email: Sebastian Höfer <hoefer@data-cybernetics.com>, Sebastian Boerakker <boerakker@data-cybernetics.com>
License: MIT
Requires-Dist: typer>=0.12.5
Requires-Dist: copier>=9.10.2
Requires-Dist: docker>=7.1.0
Requires-Dist: dotenv>=0.9.9
Requires-Dist: pinexq-client>=1.11.0
Requires-Dist: pydantic>=2.12.0
Requires-Dist: rich>=14.2.0
Requires-Dist: tomli>=2.3.0
Requires-Dist: importlib>=1.0.4
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: plotille>=6.0.5
Maintainer: Sebastian Höfer, Sebastian Boerakker
Maintainer-email: Sebastian Höfer <hoefer@data-cybernetics.com>, Sebastian Boerakker <boerakker@data-cybernetics.com>
Requires-Python: >=3.13
Description-Content-Type: text/markdown

# pinexq CLI

A minimal command-line interface for creating and running pinexq workers.

Install (editable):

- Using uv or pip, from the project root:
  - `uv pip install -e .` or `pip install -e .`

Usage:

- Global options must precede the subcommand.
- Authentication: Provide `--api-key` or set the `PINEXQ_API_KEY` environment variable. The CLI prefers the flag over the env var.

## Reading Resources (`list` and `show`)

The CLI can read every JMA Resource in your Context. Each Resource is a noun with
two verbs — `list` (many) and `show <ref>` (one):

| Noun | Alias | Resource |
| --- | --- | --- |
| `pinexq job` | | Jobs |
| `pinexq workdata` | `wd` | WorkData |
| `pinexq processing-step` | `ps` | ProcessingSteps |
| `pinexq template` | | Templates |
| `pinexq trigger` | | Triggers |
| `pinexq folder` | | Folders |

These commands run from any directory and do not need a Project.

### Connection

Every read command resolves its endpoint and API key in this order:

1. the `--endpoint` / `--api-key` flags,
2. the `PINEXQ_ENDPOINT` / `PINEXQ_API_KEY` environment variables,
3. for the endpoint only, `pinexq_endpoint` in a `pinexq.toml` in the working
   directory (there is no file fallback for the API key).

### Output

`-o, --output` selects `table` (default, human-readable), `json`, or `csv`.
`--wide` adds the extra columns to the table; `-q, --quiet` prints only ids, one
per line, for piping. JSON and CSV always carry the full column set and go to
plain stdout so they can be redirected safely.

### Pagination and sorting

`--limit` (page size, max 1000) and `--offset` page the result; `--all` fetches
every page and overrides both. `--sort FIELD` picks the sort field (the choices
differ per noun) and `--desc` / `--asc` the direction; each noun has a natural
default (Jobs, WorkData, Templates, and Triggers newest-first; ProcessingSteps by
title and Folders by path, ascending).

### Filters

Filter flags are grouped under a **Filters** panel in each command's `--help`.
The families a noun exposes follow its JMA query: tag filters (`--tag`,
`--tag-or`, `--tql`), a folder scope (`--folder`, `--recursive`), and date bounds
(`--created-after`, `--created-before`) where the Resource supports them, plus
noun-specific flags — for example `--state` for Jobs, `--kind` and `--media-type`
for WorkData, `--function` / `--version` / `--deployment-state` for
ProcessingSteps, `--enabled` / `--disabled` for Triggers, and `--path-prefix` for
Folders. Run `pinexq <noun> list --help` for the full set.

### Reference forms for `show`

`show` accepts a Resource's UUID or its full URL for every noun. Two nouns
accept a friendlier reference as well:

- `pinexq ps show <name>` or `pinexq ps show <name>@<version>` resolves a
  ProcessingStep by Function name (latest version, or the pinned one).
- `pinexq folder show </path>` resolves a Folder by its path; `folder show` also
  lists the Folder's direct child folders.

### One example per noun

```bash
pinexq job list --state completed --sort created_at --limit 20
pinexq wd list --kind clientupload --media-type text/csv -o json
pinexq ps show resize@1.2.0
pinexq template list --tag nightly --wide
pinexq trigger list --enabled
pinexq folder show /projects/foo
```

## Acting on Resources (`actions`, `action`, and the Bound verbs)

The JMA advertises every state change as an **Action** on the Resource (or on the
Resource type's root, for creation), and lists an Action only while the Resource's
state and your grants allow it. The CLI discovers and executes those Actions by
name instead of hardcoding a fixed set. Every noun in the table above — `job`,
`workdata`/`wd`, `processing-step`/`ps`, `template`, `trigger`, and `folder` —
has the two generic verbs:

| Verb | What it does |
| --- | --- |
| `pinexq <noun> actions <ref>` | List the Actions the Resource offers right now. |
| `pinexq <noun> actions --root` | List the creation Actions of the Resource type. |
| `pinexq <noun> action <ref> <name> [--set K=V]... [--json BODY]` | Execute one Action by name. |
| `pinexq <noun> action --root <name> [--set K=V]...` | Execute a creation Action on the root. |

```bash
pinexq job actions <ref>                              # what can I do to this Job right now?
pinexq job action <ref> hide                          # execute a parameterless Action
pinexq wd action <ref> rename --set NewName=report    # execute one with parameters
pinexq folder actions --root                          # how do I create a Folder?
pinexq folder action --root create-folder --set Name=reports
pinexq template action <ref> execute-now              # prints the Job it created
pinexq job action <ref> delete --yes                  # Destructive, no prompt
```

`show <ref>` in table output ends with a footer naming the Actions the Resource
offers right now (`Actions: hide, rename, ...`), so the next step is one screen
away. The JSON and CSV output of `show` are unchanged.

### Discovering Actions

`<noun> actions <ref>` lists each available Action with its name in kebab-case, its
HTTP method, its markers (such as `Destructive`, or `FileUploadAction`), and its
parameters as `Name: type, required` read from the schema the JMA serves for it.
`-o json` adds the served name, the URL, the Siren class list, the current
parameter values, the schema URL, and the schema itself; `-q` prints only the
names.

### Executing an Action

`<noun> action <ref> <name>` executes an Action and then prints the result with
the same renderer and `-o table|json|csv` / `-q` options as `show`:

- an Action that changes the Resource prints the Resource, re-fetched;
- an Action that creates a Resource (a creation Action on `--root`, or a
  Template's `execute-now`) prints the **created** Resource, with the view of
  its own type — `template action <ref> execute-now -q` prints the new Job's id;
- an Action that answers with a report body (a Folder's `clear`) prints that
  report;
- an Action that removes the Resource (a `delete`) prints what is left of it: its
  id and url.

The name is matched case-insensitively and ignoring `-` and `_`, so
`StartProcessing`, `start-processing`, and `start_processing` all work.

Parameters are built in three layers, each overriding the one before:

1. the current values the JMA sends with the Action (so you only mention what
   changes),
2. `--json BODY`, `--json @file.json`, or `--json -` (stdin), a JSON object,
3. `--set KEY=VALUE`, repeatable.

Each `--set` value is typed by the schema property: `integer`, `number`,
`boolean` (`true`/`false`, `yes`/`no`, `on`/`off`, `1`/`0`), `string` (kept verbatim, so
`--set NewName=1` is the string `"1"`), `array` (a repeated key appends, and a
JSON list literal is accepted as is), and `object` (a JSON literal). `null` is
accepted for a nullable property. The body is validated against the schema
(Draft 2020-12) before anything is sent: a missing required property, an unknown
property, or a type mismatch is a usage error (exit 2) naming the property.

When the JMA serves no schema for an Action, or the schema cannot be fetched,
the CLI prints a notice on stderr, types each value heuristically (JSON if it
parses, otherwise a string), and lets the JMA validate.

`--dry-run` resolves the target, finds the Action, types and validates the
parameters, then prints the method, URL, markers, and the pretty-printed body
without sending anything or asking for confirmation.

An Action that takes a file upload (marked `FileUploadAction`, such as the
WorkData root's `upload`) cannot be sent by `action`; the CLI refuses it with a
hint to the `wd upload` Bound verb (a separate ticket).

### Destructive Actions and confirmation

The JMA marks an Action whose effect cannot be undone as `Destructive` (for
example `delete`); the CLI never hardcodes that list. Before executing one:

- on a terminal, the CLI asks `... is Destructive and cannot be undone. Execute?`
  and declining exits 1 without an error message;
- `--yes` / `-y` skips the prompt, for scripts;
- without a terminal and without `--yes`, the CLI refuses with exit 1 and a hint
  to pass `--yes`, so a pipeline never hangs on a prompt.

Non-Destructive Actions never prompt. The prompt goes to stderr, so `-o json`
output stays clean. A Bound verb acting on several Resources asks once, naming
the Action and the count.

### Job Bound verbs

A **Bound verb** executes exactly one Action with typed flags. It runs through
the same path as `action` (same typing, validation, confirmation, and rendering)
and adds nothing the JMA does not advertise. `pinexq job --help` lists them in
their own panel:

| Verb | Action | Notes |
| --- | --- | --- |
| `job start <ref>...` | StartProcessing | |
| `job delete <ref>...` | Delete | Destructive: prompts once, `--yes` skips. |
| `job hide <ref>...` / `job unhide <ref>...` | Hide / UnHide | |
| `job rename <ref> <new-name>` | Rename | Prints the Job like `show`. |
| `job tag <ref>... --add T --remove T --clear` | EditTags | Reads the current tags, applies `--clear`, then `--remove`, then `--add`, writes once per Job. |
| `job move <ref>... --to <folder>` / `--root` | MoveToFolder | `--to` takes a Folder `/path`, UUID, or URL; `--root` moves to the Context root. |
| `job set-error <ref> -m MESSAGE` | SetJobToErrorState | Prompts when the JMA marks it Destructive. |

Every Bound verb also accepts `--dry-run`, `-o`, and `-q`.

### WorkData Bound verbs

The same housekeeping verbs exist under `workdata` and its alias `wd`, with
`comment` in place of Job's `start` and `set-error`:

| Verb | Action | Notes |
| --- | --- | --- |
| `wd delete <ref>... [--force]` | Delete | Destructive: prompts once, `--yes` skips. See `--force` below. |
| `wd hide <ref>...` / `wd unhide <ref>...` | Hide / UnHide | |
| `wd rename <ref> <new-name>` | Rename | Prints the WorkData like `show`. |
| `wd comment <ref> <text>` | EditComment | Replaces the comment. |
| `wd tag <ref>... --add T --remove T --clear` | EditTags | As on Job. |
| `wd move <ref>... --to <folder>` / `--root` | MoveToFolder | As on Job. |

A WorkData that a Job produced does not allow deletion until you say so: the JMA
offers `AllowDeletion` instead of `Delete`. Plain `wd delete` fails on such a
WorkData with the usual not-available message, listing `allow-deletion` among
the Actions it does offer, and continues with the rest of the batch. `wd delete
--force` executes `AllowDeletion` first, re-reads the WorkData, and then executes
`Delete`; a WorkData that already allows deletion is deleted directly. The one
confirmation is asked up front, before anything is sent, because the end of the
two-step is the Destructive `Delete`. `--dry-run` prints both steps. Should
`Delete` fail after `AllowDeletion` succeeded, the error is followed by a note
that the WorkData now allows deletion.

```bash
pinexq wd list -q --folder /runs/2024 --unused | pinexq wd delete - --force -y
```

#### Upload and download

| Verb | What it does |
| --- | --- |
| `wd upload <file> [--name NAME] [--media-type TYPE] [--tag T]... [--folder <folder>]` | Creates a WorkData from a local file through the WorkData root's Upload Action (multipart). The name defaults to the file name, the media type is guessed from the extension (else `application/octet-stream`) unless `--media-type` says otherwise. Tags and Folder are set right after creation, one Action each. Prints the WorkData like `show`; `-q` prints its id. |
| `wd download <ref> [-o PATH\|-] [--force]` | Streams the content to a file named after the WorkData (its name reduced to a bare file name, else its id), to `PATH`, or to stdout with `-o -`, byte for byte. The file is written next to its final name and renamed into place only once the stream ended, so a failure leaves nothing behind. An existing file is not overwritten unless `--force`. Note: on `download`, `-o` is the destination path, not an output format — there is no table or JSON to format. |

```bash
pinexq wd upload ./input.csv --tag raw --folder /data/2024
pinexq wd download <ref> -o - | head
```

### ProcessingStep Bound verbs

Under `processing-step` and its alias `ps`; references accept a UUID, URL, `name`, or `name@version` as `ps show` does:

| Verb | Action | Notes |
| --- | --- | --- |
| `ps deprecate <ref>... [--reason TEXT]` | Deprecate | |
| `ps restore <ref>...` | Restore | |
| `ps delete <ref>...` | Delete | Destructive: prompts once, `--yes` skips. |
| `ps hide <ref>...` / `ps unhide <ref>...` | Hide / UnHide | |
| `ps rename <ref> <title>` | EditTitle | Sets the display title; the function name and version are immutable. |
| `ps tag <ref>... --add T --remove T --clear` | EditTags | As on Job. |
| `ps move <ref>... --to <folder>` / `--root` | MoveToFolder | As on Job. |
| `ps set-defaults <ref> --param K=V...` | ConfigureDefaultParameters | Each `--param` is typed and validated against the Function's own parameter schema, which the JMA serves for this ProcessingStep (heuristically when it serves none). The given parameters are merged over the current defaults; a repeated key builds an array, as with `--set`. |
| `ps clear-defaults <ref>` | ClearDefaultParameters | Prompts when the JMA marks it Destructive. |

### Template Bound verbs

| Verb | Action | Notes |
| --- | --- | --- |
| `template run <ref>` | ExecuteNow | Prints the **Job** it created, with the Job view; `-q` prints the Job's id. |
| `template delete <ref>...` | Delete | Destructive: prompts once, `--yes` skips. |
| `template edit <ref> [--name] [--description]` | SetNameAndDescription | Read-modify-write: the omitted field keeps its current value. |
| `template tag <ref>... --add/--remove/--clear [--scope template\|job\|output]` | SetTags / SetJobTags / SetOutputTags | `--scope` picks the Template's own tags (default), the tags for the Jobs it creates, or the tags for their output WorkData. |
| `template set-params <ref> --param K=V...` | ConfigureParameters | Typed against the Template's parameter schema (the Function's own). |
| `template clear-params <ref>` | ClearParameters | |
| `template set-input <ref> INDEX <wd-ref>...` | SelectWorkDataCollection (on the DataSlot) | `INDEX` is the input DataSlot's 0-based position as `show` lists them; several WorkData select a collection. The Template is shown afterwards. |
| `template clear-input <ref> INDEX` | Clear (on the DataSlot) | |
| `template create --name N --description D --processing-step <ps-ref> [--param K=V]... [--tag T]...` | CreateActionTemplate, then ConfigureParameters and SetTags | The ProcessingStep accepts a UUID, URL, `name`, or `name@version`. |

The DataSlot verbs act on a sub-entity of the Template, which the generic
`action` verb cannot reach.

### Trigger Bound verbs

| Verb | Action | Notes |
| --- | --- | --- |
| `trigger enable <ref>...` / `disable <ref>...` | Enable / Disable | |
| `trigger delete <ref>...` | Delete | Destructive: prompts once, `--yes` skips. |
| `trigger edit <ref> [--title] [--description]` | SetTitleAndDescription | Read-modify-write: the omitted field keeps its current value. |
| `trigger set-templates <ref> <template-ref>...` | SetTemplates | Replaces the linked Templates. |
| `trigger set-cron <ref> EXPR [--tz ZONE] [--on-conflict stackup\|skip]` | SetCronExpression | Sends only the given fields; the rest keep their current values (the JMA requires the conflict setting, so the Trigger's current one is sent back when `--on-conflict` is omitted). `--on-conflict` is case-insensitive. |
| `trigger set-window <ref> [--starts-at T] [--expires-at T] [--delete-orphaned/--keep-orphaned]` | SetActivationWindow | Sends only the given fields. |
| `trigger create --title T --description D --cron EXPR [--tz] [--on-conflict] [--enabled/--disabled] [--starts-at] [--expires-at] [--delete-orphaned] [--template <ref>]...` | CreateCronTrigger, then SetTemplates | When `--on-conflict` is omitted the JMA's default applies; if its schema requires one, the CLI reports the missing property (exit 2). |

### Folder Bound verbs

Folder references accept a `/path`, UUID, or URL as `folder show` does.

| Verb | Action | Notes |
| --- | --- | --- |
| `folder create <name> [--parent <folder>]` | CreateFolder | Under an existing parent (no recursive creation; a missing parent is an error), or at the Context root. |
| `folder rename <ref> <new-name>` | Rename | |
| `folder move <ref>... --to <folder>` / `--root` | Move | Sets the parent. |
| `folder delete <ref>...` | Delete | Destructive: prompts once, `--yes` skips. |
| `folder clear <ref>` | Clear | Deletes everything inside the Folder and prints the report the JMA returns (counts and skipped items) instead of a Resource row. Destructive. |

### `job create`

```bash
pinexq job create --name run-1 --processing-step resize@1.2.0 \
  --param quality=90 --input 0=<wd-ref> --input 1=<wd-ref>,<wd-ref> \
  --tag nightly --folder /runs --start
```

Creates a fully configured Job in one root Action (the JMA's RapidSetupJob) and
prints it (`-q` prints its id). The ProcessingStep accepts a UUID, URL, `name`,
or `name@version`; WorkData references accept a UUID or URL, and several on one
0-based DataSlot index assign a collection. `--param` values are typed and
validated against the ProcessingStep's own parameter schema when it carries one
(a type mismatch or unknown parameter exits 2), heuristically otherwise, and are
sent as the `Parameters` JSON text the Action takes (a repeated key builds an
array, as with `--set`). `--start` starts the Job in the same request.
`--folder` is not a RapidSetupJob field, so the created Job is moved afterwards
as a follow-up Action; if that fails the Job exists, unmoved, and the exit code
is 1. `--dry-run` prints the assembled body. WorkData references in `--input`
and Template references elsewhere must be a UUID or URL; anything else is a
usage error before a request is made.

#### Creating verbs and follow-ups

`wd upload`, `template create`, `trigger create`, `job create`, and `folder
create` create a Resource through a root Action and then chain zero or more
follow-up Actions on it (tags, a Folder, parameters, linked Templates). The
creation is planned and validated before anything is sent; `--dry-run` prints
it and names the follow-ups. If a follow-up fails, the failure is reported with
a note naming the created Resource's id and the failed step, the created
Resource is still printed (`-q` its id), and the exit code is 1.

#### Several Resources at once

The multi-ref verbs (`start`, `delete`, `hide`, `unhide`, `tag`, `move`, and
their equivalents on the other nouns: `deprecate`, `restore`, `enable`,
`disable`) accept several references and print one table (or JSON array) with one row per Resource;
`-q` prints the ids. A reference that fails — not found, not in the right state,
a JMA error — is reported on stderr and the rest continue; the exit code is 1 if
any failed. A `<ref>` of `-` reads references from stdin, one per line (blank
lines ignored), which closes the loop with `list -q`:

```bash
pinexq job list -q --state failed | pinexq job delete - -y
pinexq job list -q --tag nightly | pinexq job tag - --add archived
pinexq wd list -q --tag scratch | pinexq wd hide -
```

Reading references from stdin leaves no terminal for a prompt, so a Destructive
batch fed from a pipe needs `--yes`.

### Exit codes

| Code | Meaning |
| --- | --- |
| `0` | Success. |
| `1` | Resource not found; Action not currently available (the message lists the Actions that are); a file-upload Action; authentication, connection, or JMA failure (a `400` validation problem is listed per property; a `423` is reported as a version conflict to re-run); confirmation declined or impossible; any failure in a batch. |
| `2` | Usage error: malformed `--set` or `--param`, unreadable `--json` source, unknown or mistyped property, schema violation, parameters passed to a parameterless Action, `--root` together with a reference (or neither), `--to` together with `--root` (or neither), `tag` without a change, `edit` or `set-window` without a field, a malformed `--input`, a missing upload file, an unknown `--on-conflict`, or an empty stdin. |

Connection resolution and the `<ref>` forms are the same as for `show`.

## Job logs (`pinexq job logs <ref>`)

`pinexq job logs <ref>` prints the Log entries of a Job, oldest first, one per
line. `<ref>` is the Job's UUID or URL, and the connection resolves exactly as for
`list` and `show`. The CLI follows the Job's Logs link to the monitoring service
and pages through the whole log.

### Formats

`-o, --output` selects the line format:

- `text` (default): `[timestamp] LEVEL message`. On a TTY the level is coloured
  (yellow for `WARN`, red for `ERROR` and above); piped output has no escape
  codes. `--no-color` or the `NO_COLOR` environment variable forces plain text.
- `raw`: the stored message only, verbatim.
- `json`: NDJSON, one flat object per line with `timestamp`, `level`, `message`,
  and `metadata` (not an array, so `jq -c .level` works line by line).

`raw` and `json` go to plain stdout and are never coloured. No line is ever
wrapped, whatever the terminal width.

### Filters and `--tail`

The monitoring service has no server-side filters, so these run in the CLI:

- `--level L` keeps entries at that Log level and above. Levels in ascending
  order: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, `CRITICAL` (`FATAL`
  and `CRITICAL` rank equal). The value is case-insensitive and tab-completes; an
  entry with an unknown level always passes.
- `--search TERM` keeps entries whose message contains the term
  (case-insensitive substring, no regex).
- `--level` and `--search` combine with AND.
- `--tail N` prints only the last N matching entries, still oldest first. The
  CLI reads the log newest-first and stops as soon as N entries matched, so a
  tail of a long log is cheap.

```bash
pinexq job logs 11111111-1111-1111-1111-111111111111 --level warn
pinexq job logs <ref> --search timeout --tail 20
pinexq job logs <ref> -o json | jq -c 'select(.level == "ERROR") | .message'
```

### Exit codes

- `0`: the log was read, including when no entry matched (a notice goes to
  stderr and stdout stays empty).
- `1`: the Job has no Logs link (`job <ref> has no logs`), the Job was not
  found, authentication failed, or the endpoint could not be reached. The
  reason is printed on stderr with the same wording as `list` and `show`.
- `2`: a usage error, such as an unknown `--level`.

## Job metrics (`pinexq job metrics <ref>`)

`pinexq job metrics <ref>` fetches the Metrics of the Worker that ran a Job: CPU
usage in cores, memory usage in bytes, and both as a percentage of the Worker's
requested resources. The window is the Job's own lifetime as the JMA built it (a
running Job is read up to now); there are no time flags.

- Default: a header with the Worker name, status, termination reason, duration,
  queried window, and the peak CPU and memory with their utilization percentages,
  followed by two terminal plots (drawn with plotille, braille glyphs): CPU usage
  in cores over time, then memory usage in MiB over time. The x axis spans the
  queried window. Plots take the terminal width on a TTY and 100 columns when
  piped; colour follows the `job logs` rule (`--no-color` / `NO_COLOR` turn it
  off). Use `-o json` or `-o csv` on a terminal that cannot show braille.
- `-o json`: one flat object with `worker`, `status`, `terminated`,
  `termination_reason`, `duration_seconds`, `queried_start`, `queried_end`, and the
  four series `cpu_usage`, `memory_usage`, `cpu_utilization`, `memory_utilization`
  as lists of `{timestamp, value}`.
- `-o csv`: the columns `timestamp,cpu_usage,memory_usage,cpu_utilization,memory_utilization`,
  one row per distinct timestamp across the four series, blank where a series has
  no sample, ascending.

Exit codes follow `job logs`: `0` with a stderr notice when the series are empty
(`-o csv` then prints the header row only), `1` when the Job has no Metrics link
(`job <ref> has no metrics`), was not found, or the connection or authentication
failed.

## Deploying on Docker Desktop (containerd image store)

Fresh Docker Desktop installs since version 4.34.0 default to the **containerd
image store**. This changes what a local image looks like, and both differences
are expected and supported by `pinexq deploy`:

- An image ID is the manifest (or index) digest rather than the config digest.
- A `docker build` produces an OCI image index that also carries provenance and
  SBOM attestation manifests. This is normal; the deploy pushes it and registers
  the digest the registry reports.

Check which store is active:

```bash
docker info --format '{{ .DriverStatus }}'   # containerd store shows "io.containerd..."
```

Note that `docker system prune` **without `-a`** does not remove tagged images —
so if a bad deploy is fixed by "prune and retry", it is the retry's fresh push
that fixes it, not the prune.

## Troubleshooting: `ImagePullBackOff` right after a deploy

If a function pod fails to pull the image you just deployed, work through this
checklist. The full analysis is in
[`docs/research/2026-08-28-imagepullbackoff-after-successful-push.md`](docs/research/2026-08-28-imagepullbackoff-after-successful-push.md).

1. **Re-run the deploy with `--verbose`.** The push stream is printed chunk by
   chunk, so a swallowed registry error (`denied`, `unauthorized`, `blob upload
   unknown`) becomes visible. Since the fix in this repo, a failed push aborts the
   deploy instead of registering an unpullable digest.

2. **Read the pod events and map the error text to a cause:**

   ```bash
   kubectl describe pod <pod>
   ```

   - `failed to resolve reference ... not found` → the registered digest is not a
     manifest the registry holds for this repository (the classic bug).
   - `failed to authorize ... 401 Unauthorized` → registry token / pull-secret
     problem, not a missing image.
   - `could not fetch content descriptor ... not found` / `httpReadSeeker: failed
     open` → the manifest exists but a referenced blob is missing.
   - A `localhost:6443` / mirror address in the message is the k3s Spegel mirror
     being tried first; the upstream registry error follows it.

3. **Inspect the local image digests** (compare against what was registered):

   ```bash
   docker image inspect --format '{{json .RepoDigests}}' <name:tag>
   docker image inspect --format '{{ .Id }}' <name:tag>
   docker buildx imagetools inspect <registry>/<context>/<function>:<version>
   ```

4. **Ask the registry directly** whether the digest is present:

   ```bash
   # with a token from the registry realm
   curl -sI -H "Authorization: Bearer $TOKEN" \
     https://<registry>/v2/<context>/<function>/manifests/<digest>
   # or, if available:
   skopeo inspect --raw docker://<registry>/<context>/<function>@<digest>
   crane manifest <registry>/<context>/<function>@<digest>
   ```

5. **Check the registry pod** for restarts or `manifest unknown` log lines around
   the time of the deploy:

   ```bash
   kubectl -n <ns> logs deploy/registry --since=1h | grep -iE 'manifest|blob|unknown'
   kubectl -n <ns> get pod -l app.kubernetes.io/name=registry
   ```