# Workflow Queues

<!-- BEGIN GENERATED: guide-map -->
## Guide map

Read only the section needed for the next decision. Line ranges are exact
for this generated runtime target.

| Section | What it covers | Lines |
| --- | --- | --- |
| [Purpose](#purpose) | Scope, ownership, and the central operating question. | 27-37 |
| [Use This When](#use-this-when) | Tasks and conditions that belong in this guide. | 38-48 |
| [Do Not Use This For](#do-not-use-this-for) | Boundaries and the correct routes for excluded work. | 49-61 |
| [Mental Model](#mental-model) | Concepts and state boundaries needed for correct operation. | 62-76 |
| [Preflight](#preflight) | State, authority, inputs, and readiness to verify before execution. | 77-98 |
| [Risk And Side Effects](#risk-and-side-effects) | Effects, approvals, cost, privacy, and retry boundaries. | 99-112 |
| [Canonical Operating Pattern](#canonical-operating-pattern) | The normal end-to-end execution sequence. | 113-128 |
| [Queue States](#queue-states) | Operating guidance and contracts for this section. | 129-138 |
| [Core SDK Surfaces](#core-sdk-surfaces) | Exact source-backed callable contracts owned by this guide. | 139-152 |
| [Data Contracts](#data-contracts) | Required identities, fields, evidence, handles, and durable shapes. | 153-180 |
| [Examples](#examples) | Representative usage patterns and code. | 181-229 |
| [Verification](#verification) | Evidence required before reporting success. | 230-242 |
| [Troubleshooting](#troubleshooting) | Failure diagnosis and corrective routes. | 243-255 |
| [Related References](#related-references) | Adjacent boundaries and when to cross them. | 256-266 |
<!-- END GENERATED: guide-map -->

## Purpose

Use this guide when several local workflow roots need to be listed, grouped,
run, retried, inspected, or exported together. A workflow queue is a local
coordination/status index for multiple generated workflow roots. It is not a
hosted scheduler, backend `/plays` API, report renderer, or approval engine.

Queue state lives in the global workflow catalog database under `~/.verbiflow`
or `VERBIFLOW_WORKFLOW_CATALOG_PATH`; each workflow root keeps its own SQLite
rows, runs, outputs, reports, and approval state.

## Use This When

- The user wants to inspect or operate several existing workflow roots as one
  local batch.
- You need a local status index across multiple generated Workflow Plays.
- Failed items should be retried while completed items remain untouched.
- A blocked approval/confirmation item should stay blocked while other queue
  items continue.
- You need a static queue index for an operator review, separate from each
  workflow's final Play UI/report.

## Do Not Use This For

- One workflow's internal step ordering, source checkpoints, repair/replay, or
  report build; use `workflow_operations.md` and `workflow_sources.md`.
- Designing a new multi-step workflow before roots exist; use
  `workflow_composition.md`.
- Backend `/plays` list behavior, workspace Play sync, or final Play
  UI/report design; use `workflow_operations.md` and `workflow_reports.md`.
- Hosted schedules, cron, always-on daemons, or automatic provider execution.
  Queues run only when `vf.workflows.queue.run(...)` is called.
- Auto-approving sequence publish, sends, purchases, or other
  customer-visible mutations. Queues preserve blockers; they do not decide.

## Mental Model

The queue coordinates roots; it does not merge their state. Each workflow root
keeps its own SQLite rows, runs, outputs, and approvals. Queue items record
workflow id, root, run id, status, errors, timestamps, and latest run summary.

`queue.run(...)` claims pending items atomically, then invokes each generated
workflow root's `run.py --json` in a local thread pool. `concurrency` controls
how many roots run at once, not provider batching inside a root. `pipelined=True`
is passed through to each workflow run; it does not parallelize arbitrary Python
loops inside a step.

Queue records are scoped to the active SDK profile/workspace. A queue created
in one workspace cannot be opened, run, or amended from another workspace.

## Preflight

1. Resolve the active workspace/profile before listing or running queues.
2. Use `vf.workflows.queue.list(...)` or `vf.workflows.queue.open(...)` to
   confirm the queue id, name, tags, root path, and item count.
3. Inspect `vf.workflows.queue.status(queue_id)` before running. Print counts,
   item workflow ids, roots, requested run ids, statuses, and latest errors.
4. Confirm every queued workflow root is the intended root. If the same org has
   several similar workflows, inspect names/root paths before adding or running.
5. Check whether queue items are pending, failed, running, blocked, or completed.
   `run(...)` normally claims pending items only; failed items require
   `retry_errors=True`.
   A stale `running` claim older than 24 hours is recovered to `failed` before
   new claims are selected.
6. Confirm mode and side-effect policy before running batch or production. Queue
   execution can trigger whatever side effects the workflow's own policy and
   steps allow.
7. Use an absolute export path when exporting a queue status index.

Stop if any item targets the wrong workspace, root, sequence/source, or mode.
Fix the queue contents or run the individual workflow first.

## Risk And Side Effects

| Operation class | Examples | Queue rule |
| --- | --- | --- |
| Local queue read | `list`, `open`, `status` | Safe after workspace focus is verified. Print ids/counts. |
| Local queue write | `create`, `add`, `export` | Writes the global workflow catalog DB or local files. Preserve `queue_id`, `item_id`, and export paths. |
| Local workflow execution | `run` in preview with no customer-visible effects | Runs local generated `run.py` files and writes per-root workflow state. Inspect item summaries. |
| Provider/credit work | queued roots whose steps call provider, enrichment, web, CRM, or sequence reads | Queue does not add safety. The workflow's own refs and policy decide credit usage. |
| Sequence/source staging | queued roots in batch/production with `sequence_stage` effects | Allowed only when each workflow declares/guards the effect. Verify source run/import ids after run. |
| Customer-visible mutation | publish, send, sender/channel changes, purchases | Do not hide these in queue execution. Require explicit confirmation in the owning workflow/sequence/sender/billing path. |

Queues should never be used as a shortcut around approval gates. A blocked item
remaining blocked is correct behavior.

## Canonical Operating Pattern

1. List or create the queue.
2. Add only cataloged workflow roots or explicit workflow root/state paths.
3. Read queue status before execution and print the queue id, counts, item
   workflow ids, roots, and current statuses.
4. Choose run options deliberately: `mode`, `max_source_rows`, `max_items`,
   `limit`, `pipelined`, `retry_errors`, and `concurrency`.
5. Run the queue only after confirming side-effect risk for every item.
6. Inspect the returned status: counts, item status, run ids, summaries,
   return codes, stdout/stderr snippets, and errors.
7. Export a queue status index when the user needs a durable multi-root review
   file.
8. For any failed or blocked item, switch to `workflow_operations.md` for the
   specific root before retrying broadly.

## Queue States

| Status | Meaning | Normal next action |
| --- | --- | --- |
| `pending` | Item has been added but not claimed by a queue run. | Safe to run after preflight. |
| `running` | Item has been claimed and its generated `run.py` is executing. | Wait for completion; do not start ad hoc duplicate runs. |
| `completed` | The generated run returned success or summary status completed. | Inspect run id, outputs, report/sync state if needed. |
| `failed` | `run.py` returned nonzero, was missing, or summary indicated failure. | Inspect item error/summary, then retry with `retry_errors=True` only after fixing the root. |
| `blocked` | Workflow summary reported an approval/interruption blocker. | Keep blocked; resolve the specific workflow approval path. |

## Core SDK Surfaces

<!-- BEGIN GENERATED: core-sdk-surfaces -->
| Function | Use When | Inputs | Output And Handles To Preserve | Side Effect | Source |
| --- | --- | --- | --- | --- | --- |
| `vf.workflows.queue.create(name, tags=(), root=None)` | Create or upsert a named local queue. | Required `name` (`str`). Optional `tags` sequence for lookup and optional `root` (`str \| Path`) for the queue directory. Without `root`, the SDK allocates `~/.verbiflow/workflows/<workspace-scope>/queues/<slug>/`. | Queue dict with `queue_id`, `name`, `root_path`, `tags`, timestamps, profile/org/base URL. Preserve `queue_id`. | Local catalog DB write and queue directory creation. | installed SDK: `verbiflow/workflows.py:472` |
| `vf.workflows.queue.list(limit=50, tag=None)` | Find queues in the active workspace/profile. | Optional positive `limit`; optional exact `tag` filter. | List of queue records. Preserve candidate `queue_id` values before open/status/run. | Local catalog DB read. | installed SDK: `verbiflow/workflows.py:640` |
| `vf.workflows.queue.open(queue_id)` | Inspect one queue record before adding/running items. | Required `queue_id` (`str`). | Queue dict without item summaries. Use `status(...)` when item counts/details matter. | Local catalog DB read; validates active workspace scope. | installed SDK: `verbiflow/workflows.py:660` |
| `vf.workflows.queue.add(queue_id, workflow_id_or_path, run_id=None, params=None)` | Add an existing cataloged workflow root, root path, or state SQLite path. | Required `queue_id`; required `workflow_id_or_path`; optional existing `run_id`; optional JSON-like `params` stored on the item. | Queue item dict with `item_id`, `workflow_id`, `workflow_root`, `workflow_state_path`, requested run id, status, params, timestamps. Preserve `item_id` and workflow root. | Local catalog DB write; validates queue/workflow workspace scope. | installed SDK: `verbiflow/workflows.py:512` |
| `vf.workflows.queue.status(queue_id)` | Inspect queue state without running items. | Required `queue_id`. | Status dict with queue metadata, `counts`, and `items`. Each item includes status, run ids, error, summary, roots, and timestamps. | Local catalog DB read; validates active workspace scope. | installed SDK: `verbiflow/workflows.py:618` |
| `vf.workflows.queue.run(queue_id, concurrency=3, mode=None, max_source_rows=None, max_items=None, retry_errors=False, pipelined=False, limit=None)` | Execute pending queue items with bounded local root-level concurrency. | Required `queue_id`; positive `concurrency`; optional `mode` (`preview`, `batch`, `production`); optional positive limits; `retry_errors=True` also claims failed items; `pipelined=True` passes `--pipelined` to generated `run.py`. | Queue status dict after execution. Preserve per-item `run_id`, status, error, summary, return code, stdout/stderr snippets. | Local workflow execution; may trigger provider/platform side effects declared by each workflow. | installed SDK: `verbiflow/workflows.py:578` |
| `vf.workflows.queue.export(queue_id, path)` | Write a static queue status index for operator review. | Required `queue_id`; required absolute `path` as output directory or `.html` file path. | Dict with `queue_id`, `html_path`, `json_path`, and `item_count`. Preserve both paths. | Local file write. | installed SDK: `verbiflow/workflows.py:685` |
<!-- END GENERATED: core-sdk-surfaces -->

## Data Contracts

Queue records include `queue_id`, `name`, `root_path`, `tags`, timestamps,
profile, org id, and base URL. Queue item records include `item_id`,
`workflow_id`, `workflow_root`, `workflow_state_path`, `requested_run_id`,
`run_id`, `status`, `params`, `error`, `summary`, and timestamps.

The queue run summary is parsed from generated workflow `run.py --json` output.
It may include `run_id`, `status`, `steps`, row counts, outputs, and errors
from that workflow. The queue stores bounded stdout/stderr snippets for
debugging, but the canonical rows, outputs, and reports remain inside the
workflow root.

If a process dies after claiming an item, a later queue run moves `running`
items older than 24 hours to `failed`. The queue does not immediately retry that
item unless `retry_errors=True` is passed, so an operator/agent can inspect the
root before replaying side effects.

Do not treat queue export JSON/HTML as workflow state. It is a queue status file
that can go stale after the next workflow run.

The HTML export is the operator-facing queue view. It must keep all five states
visually and verbally distinct: queued, running, blocked, failed, and complete.
Lead with the queue's current condition and safe next action, then counts, then
item detail. Preserve workflow ids, root paths, run ids, exact errors, and update
times; visual polish must never hide diagnostic evidence. Treat the export as a
read-only snapshot and say so in the UI.

## Examples

Create a local workflow, add it to a queue, run pending items, and export an
absolute queue index. This example creates local files only unless the generated
workflow itself calls provider/platform SDK functions.

```python
from pathlib import Path
import verbiflow as vf

workflow = vf.play_scaffold.init(
    Path.cwd() / "queued-review-play",
    name="Queued Review Play",
    git=False,
)
queue = vf.workflows.queue.create("weekly-research", tags=("weekly",))
item = vf.workflows.queue.add(queue["queue_id"], workflow["workflow_id"])

before = vf.workflows.queue.status(queue["queue_id"])
print(before["queue_id"], before["counts"], item["item_id"])

status = vf.workflows.queue.run(
    queue["queue_id"],
    concurrency=2,
    mode="preview",
    max_source_rows=100,
    pipelined=True,
)
print(status["counts"])

exported = vf.workflows.queue.export(
    queue["queue_id"],
    Path.cwd().resolve() / "queue-index",
)
print(exported["html_path"], exported["json_path"])
```

Use `retry_errors=True` only after inspecting and fixing failed items:

```python
status = vf.workflows.queue.status(queue_id)
failed = [item for item in status["items"] if item["status"] == "failed"]
for item in failed:
    print(item["workflow_id"], item["workflow_root"], item.get("error"))

rerun = vf.workflows.queue.run(queue_id, retry_errors=True, concurrency=1)
print(rerun["counts"])
```

## Verification

- Preserve the `queue_id`.
- Preserve `item_id`, `workflow_id`, root path, requested run id, and final
  `run_id` for each item.
- Inspect counts, per-item statuses, errors, and summaries after `run(...)`.
- Confirm blocked items stayed blocked rather than being auto-approved.
- Confirm failed items were not retried unless `retry_errors=True` was
  intentionally used.
- Confirm export wrote both `html_path` and `json_path` when a durable status
  view is needed.
- Export the queue index when the user needs a durable multi-root status view.

## Troubleshooting

| Symptom | First check | Fix |
| --- | --- | --- |
| Queue cannot be opened or run | Active workspace/profile and queue `profile`/`org_id`. | Focus the correct workspace/profile, then open/status again. |
| Adding a workflow fails as wrong workspace | Workflow root `.verbiflow/context.json` and catalog record scope. | Use the workflow root from the same workspace or refocus before adding. |
| `run(...)` returns no changes | `status(...)` counts; only `pending` items are claimed by default. | Add items or pass `retry_errors=True` after fixing failed items. |
| Item is stuck `running` after a crashed process | Item `updated_at`; stale claims older than 24 hours are recovered to `failed` on the next `run(...)`. | Inspect the workflow root, then run with `retry_errors=True` only if replay is safe. |
| Item fails with missing `run.py` | Item `workflow_root` and error text. | Use a generated workflow root or recreate/register the workflow correctly. |
| Item remains blocked | Item summary and specific workflow approval state. | Resolve the approval in the workflow/sequence/sender/billing path; do not auto-approve through the queue. |
| Queue export fails | Export path absolute and parent permissions. The SDK error is `queue export path must be absolute.` | Pass an absolute directory or `.html` path. |
| Queue run seems serial inside one workflow | Step code and provider-job wait loop. | Queue `concurrency` is root-level; use workflow step concurrency or start jobs first inside the workflow. |

## Related References

| If you are doing this | Read next | Why |
| --- | --- | --- |
| Operating or repairing one workflow root | `workflow_operations.md` | Owns root-specific run/resume/retry/reset/report behavior. |
| Syncing a Play after operating its root | `workflow_schedules.md` | Owns canonical revision/output sync. |
| Designing source windows/checkpointing | `workflow_sources.md` | Owns freshness, object memory, cursor, and `rows_available` semantics. |
| Designing the play before roots exist | `workflow_composition.md` | Owns multi-step play design and side-effect boundaries. |
| Building final Play UI/report output | `workflow_reports.md` | Queue export is only a queue index; final Play UI belongs to the report guide. |
| Preserving provider job/artifact handles | `artifacts_jobs.md` | Queue summaries are not a replacement for job/artifact inspection. |
| Sequence staging/publish/send or sender/domain purchase risk | `sequences.md`, `mailboxes.md` | Queue run can trigger workflow side effects; these refs own approval rules. |
