# Workflow Sources

<!-- BEGIN GENERATED: guide-map -->
## Guide map

Read only the section needed for the next decision. Line ranges are exact
for this generated runtime target.

| Section | What it covers | Lines |
| --- | --- | --- |
| [Purpose](#purpose) | Scope, ownership, and the central operating question. | 27-36 |
| [Use This When](#use-this-when) | Tasks and conditions that belong in this guide. | 37-46 |
| [Do Not Use This For](#do-not-use-this-for) | Boundaries and the correct routes for excluded work. | 47-58 |
| [Mental Model](#mental-model) | Concepts and state boundaries needed for correct operation. | 59-79 |
| [Preflight](#preflight) | State, authority, inputs, and readiness to verify before execution. | 80-101 |
| [Risk And Side Effects](#risk-and-side-effects) | Effects, approvals, cost, privacy, and retry boundaries. | 102-111 |
| [Canonical Operating Pattern](#canonical-operating-pattern) | The normal end-to-end execution sequence. | 112-127 |
| [Checkpointing And Freshness](#checkpointing-and-freshness) | Checkpointing is not proof that the provider fetched the correct freshness window, and it is not proof that downstream steps processed every row. | 128-146 |
| [Workflow Runtime Surfaces](#workflow-runtime-surfaces) | These workflow-context and source-adapter contracts are not top-level `@sdk_function` calls. | 147-169 |
| [Data Contracts](#data-contracts) | Required identities, fields, evidence, handles, and durable shapes. | 170-186 |
| [Examples](#examples) | Representative usage patterns and code. | 187-248 |
| [Verification](#verification) | Evidence required before reporting success. | 249-268 |
| [Troubleshooting](#troubleshooting) | Failure diagnosis and corrective routes. | 269-281 |
| [Related References](#related-references) | Adjacent boundaries and when to cross them. | 282-295 |
<!-- END GENERATED: guide-map -->

## Purpose

This reference owns checkpointed source acquisition inside generated workflow
plays: source windows, cursor/checkpoint semantics, object memory,
observations, restart safety, and `rows_available` events.

It answers: “How do rows from a provider job, host capture, static list,
custom adapter, or existing sequence audience become durable local workflow
rows without losing provenance or confusing source freshness?”

## Use This When

- A workflow source step needs restart-safe acquisition across interrupts.
- A recurring play needs an explicit source window or cursor contract.
- A provider job or artifact must be consumed into local workflow rows.
- Host/browser rows or static rows need object memory and cross-run dedupe.
- A workflow needs to sync the current sequence audience as suppression input.
- You are debugging stale source rows, invalid checkpoint assumptions, or
  `rows_available` behavior.

## Do Not Use This For

| Task | Read instead | Why |
| --- | --- | --- |
| Choosing the audience universe, search terms, provider, or discovery strategy | `audience_discovery.md` | Source docs do not decide what market or people to search. |
| CRM schemas, HubSpot/Salesforce filters, properties, associations, active-customer/deal suppression | `crm.md` | CRM owns provider-specific fields, filters, and associations. |
| Email, domain, profile, company, phone, or identity enrichment | `enrichment.md` | Enrichment owns resolver fields and provider semantics. |
| Local row grouping, joins, mapping, semantic qualification, or CSV shaping after acquisition | `row_operations.md` | Source acquisition should not hide business transforms. |
| Job polling, artifact export, raw streaming, or one-off inspection outside a workflow root | `artifacts_jobs.md` | Source docs only cover workflow checkpointed consumption. |
| Sequence source staging, source-run promotion, publish, send, or sender/domain purchases | `sequence_authoring.md`, `sequences.md`, `mailboxes.md` | Sequence writes and customer-visible or paid actions have separate approval rules. |
| Final report or Play UI rendering | `workflow_reports.md` | Reports are outputs, not source acquisition. |

## Mental Model

There are four separate layers. Keep them separate in code and in explanations.

| Layer | Owns | Use |
| --- | --- | --- |
| Growth capability | Provider execution, Bridge policy, credits, batching, artifacts, and platform audit. | `vf.sources.*`, `vf.crm.*`, `vf.linkedin.*`, `vf.web.*`, `vf.enrichment.*`, `vf.clay.*`, `vf.host.crunchbase.*`. |
| Checkpointed workflow acquisition | Local source identity, consumed-row checkpoint, object memory, observations, and cross-run dedupe. | `ctx.collect_artifact_job(...)` for artifact jobs; `ctx.collect_host_rows(...)` for host/browser rows; `ctx.collect_source(...)` for custom adapters. |
| Workflow transforms | Play-specific grouping, filtering, joins, suppression, thresholds, scoring, and table shaping. | Normal Python plus `ctx.rows(...)`, `ctx.save_rows(...)`, and `row_operations.md`. |
| Side-effect claims | Idempotent sequence staging, exports, provider writes, sends, purchases, and customer-visible mutations. | `workflow_operations.md`, `sequences.md`, `ctx.claim_once(...)`, and explicit typed SDK calls. |

Do not reimplement HubSpot, LinkedIn, Clay, Crunchbase, targeted search, web,
or places provider logic in workflow adapters. Those are already typed
SDK/Growth capability surfaces. A `SourceAdapter` means checkpointed workflow
acquisition for a workflow run; it is not a provider execution system.

Checkpointing is not scheduling or provider incremental sync. The workflow
only checkpoints the workflow consumption of rows for a stable `scope_key`,
`source_kind`, `source_ref`, and `cursor_key`. Provider freshness must be
expressed in the provider request itself.

## Preflight

Before writing or changing a source step:

1. Resolve the workflow root and workspace identity from inside that root.
2. Read the workflow config and print the intended source window, mode, output
   table, target sequence if any, and provider readiness.
3. Pick the owning provider reference first. For example, CRM filters come from
   `crm.md`; people/account discovery comes from `audience_discovery.md`;
   enrichment comes from `enrichment.md`.
4. Define `table`, `key`, `object_type`, and `identities` before collecting
   rows. `key` materializes per-run workflow items; `identities` build
   cross-run workflow object memory.
5. Decide whether downstream consumers can start on partial rows. Use
   `downstream_ready=True` only when row-level downstream steps are idempotent.
6. Preserve `job_id`, `artifact_id`, `source_ref`, `cursor_key`, source window,
   row counts, and sequence/source ids in outputs.

Stop if the source contract changed and old downstream rows lack required
fields. A completed source step will not rerun just because code changed; use
workflow replay/reset from `workflow_operations.md`.

## Risk And Side Effects

| Operation | Risk | Rule |
| --- | --- | --- |
| `RowsSourceAdapter` or `ctx.collect_host_rows(...)` over already-available rows | Workflow-state writes only. | Safe after row shape and identities are verified. |
| `ctx.collect_source(...)` with a custom non-row adapter | Workflow-state writes plus whatever the adapter does. | Do not bypass typed SDK provider surfaces; preview mode rejects provider-like adapters. |
| `ctx.collect_artifact_job(...)` | Workflow-state writes and artifact/job reads; the job itself may have already spent credits. | Start provider jobs in the owning provider reference, then collect their artifact rows here. |
| `ctx.sync_sequence_audience(...)` | Reads sequence/source audience and writes workflow memory. | Use as suppression/input state, not as sequence staging or promotion. |
| `ctx.stage_rows_to_sequence_source(...)`, `vf.sequences.*` staging/publish/send | Platform mutation or customer-visible action. | Route to sequence docs and approval gates. |

## Canonical Operating Pattern

1. Build the provider request in the owning reference, including source window,
   filters, limits, and credit/provider bounds.
2. Start the provider job through the typed `vf.*` surface when provider work is
   needed.
3. Collect the returned `PendingJob`/`job_id` with
   `ctx.collect_artifact_job(...)`, or collect existing/host rows with
   `ctx.collect_host_rows(...)` / `ctx.collect_source(...)`.
4. Save source outputs with row counts, ids, checkpoint fields, and freshness
   provenance.
5. Use `start_when="rows_available"` only for idempotent row-level consumers.
   Aggregate steps should wait for upstream completion.
6. Run suppression, grouping, enrichment, and staging as explicit later steps
   using their owning references.

## Checkpointing And Freshness

Checkpointing is not proof that the provider fetched the correct freshness
window, and it is not proof that downstream steps processed every row.

| Question | Answer |
| --- | --- |
| “Will it keep fetching from where the provider last left off?” | Only if the source layer explicitly defines that provider window/cursor. `ctx.collect_artifact_job(...)` resumes consuming committed batches for a specific job; it does not choose the next HubSpot `createdate`, LinkedIn page cursor, search query, or “last pulled to now” window. |
| “How do I do recurring CRM pulls?” | Use `WorkflowPolicy.batch(..., source_window=...)` or generated run arguments, then map `ctx.source_window.start/end` into CRM filters before `vf.crm.fetch_objects(...)`. The generated scaffold has `crm_filters_for_policy(...)`; use `crm.md` for provider-specific fields. |
| “How do I handle sources with no stable cursor?” | Use bounded rerunnable queries plus `object_type`/`identities` object memory and `ctx.claim_once(...)` before expensive downstream side effects, or mark the source not resumable. Do not invent hidden heuristics from response shape. |
| “Does a source checkpoint mean downstream work is complete?” | No. Source checkpoints prove acquired rows were committed. Downstream completion is tracked by workflow step status, `workflow_items` statuses for row consumers, and downstream claims for side effects. |
| “Why did a new provider job not resume the old artifact checkpoint?” | By default `ctx.collect_artifact_job(...)` makes `source_ref` and `cursor_key` job-scoped. Use stable `source_ref_value` / `cursor_key` only when the new job is intentionally the same logical source contract. |

Use `start_when="rows_available"` only for downstream steps that process
individual rows or idempotent chunks. Aggregate steps that need the complete
source table, such as grouping domains or computing account-level thresholds,
should wait for `upstream_completed` unless the step is explicitly written as an
incremental reducer.

## Workflow Runtime Surfaces

These workflow-context and source-adapter contracts are not top-level
`@sdk_function` calls. They are stable, authored client surfaces owned by this
guide, so they remain visible in both Local and Hosted guidance rather than
being reconstructed from the top-level callable index.

| Function | Use When | Inputs | Output And Handles To Preserve | Side Effect | Source |
| --- | --- | --- | --- | --- | --- |
| `WorkflowContext.collect_artifact_job(...)` / `ctx.collect_artifact_job(...)` | A typed provider/Growth call returned a `PendingJob`, job mapping, or `job_id` and the workflow must consume committed artifact rows restart-safely. | Required `job`, `table`, `key`, `object_type`, `identities`. Important optional args: `name` labels outputs; `source_kind` labels memory; `source_ref_value` and `cursor_key` override default job-scoped checkpoint identity; `require_success` controls fail-fast on failed jobs; `poll_interval` / `timeout` bound waiting; `page_size` pages artifact rows up to SDK max; `max_rows` caps acquisition; `downstream_ready` emits `workflow_rows_available`; `allow_empty` permits terminal empty jobs. | Returns dict with `name`, `scope_key`, `source_kind`, `source_ref`, `cursor_key`, `table`, `key`, `job_id`, `artifact_id`, status fields, `fetched`, `upserted`, `pages`, `source_finished`, `client_limit_reached`, `checkpoint`, `objects`, and `observations`. Preserve `job_id`, `artifact_id`, `source_ref`, `cursor_key`, row counts, and checkpoint. | Workflow state writes plus job/artifact reads; the provider job may have already spent credits. Requires workflow `artifact_job` effect allowance. | Use `vf.sdk.describe("ctx.collect_artifact_job")`. |
| `WorkflowContext.collect_source(...)` / `ctx.collect_source(...)` | A `SourceAdapter` or `SourceSpec` owns local/custom cursor acquisition and rows must be materialized into workflow item/object memory. | Required `source`, `object_type`, `identities`; optional `config`, `table`, `key`, `page_size`, `max_rows`, `source_ref_value`. With a bare adapter, `key` defaults to `_vf_source_row_key`; pass `key=` when a real row column should be the item key. | Returns dict with `scope_key`, `source_kind`, `source_ref`, `cursor_key`, `table`, `upserted`, `fetched`, `pages`, and `source_finished`. Preserve `source_ref`, `cursor_key`, and row counts. | Workflow state writes. Non-`RowsSourceAdapter` adapters are treated as provider-like and are rejected in preview mode. | Use `vf.sdk.describe("ctx.collect_source")`. |
| `WorkflowContext.collect_host_rows(...)` / `ctx.collect_host_rows(...)` | Host/browser or existing rows are already materialized and need checkpointed workflow memory. | Required `rows`, `table`, `key`, `object_type`, `identities`; optional `source_kind`, `source_ref_value`, `page_size`, `max_rows`. `rows` may be a `RowSet` or sequence of row mappings. | Same return contract as `collect_source(...)`. Preserve `source_ref`, row counts, and table/key identity. | Workflow state writes only. Host work itself must be done through the owning host reference first. | Use `vf.sdk.describe("ctx.collect_host_rows")`. |
| `WorkflowContext.sync_sequence_audience(...)` / `ctx.sync_sequence_audience(...)` | A workflow needs current target sequence/source audience rows for suppression, dedupe, or audit before staging new candidates. | Required `target` mapping. Optional `table`, `key`, `suppress_scope` (`source_ingress` or `sequence`), `object_type`, `identities`, `page_size` 1..500, `max_rows`, `source_ref_value`, `downstream_ready`. | Returns dict with sequence/source/ingress ids, `suppress_scope`, table/key, `fetched`, `upserted`, `pages`, `source_finished`, imports, lead list ids, `objects`, and `observations`. Preserve sequence ids, imports, lead list ids, and counts. | Platform reads plus workflow state writes. It is not sequence staging, promotion, publish, or send. | Use `vf.sdk.describe("ctx.sync_sequence_audience")`. |
| `WorkflowContext.assign_stable_groups(...)` / `ctx.assign_stable_groups(...)` | Accounts or another durable subject must be split across a fixed roster without reshuffling on rerun. | Required rows, `subject_field`, `assignee_keys`, and stable `namespace`; optional `assignment_field`. The subject must already be canonical, such as a normalized company domain/account key. | Returns a `RowSet` with the assignment field. Existing subjects retain owners; new subjects go to the least-loaded active owner with deterministic tie-breaking. Repeated contact rows for one account inherit one assignment. | Workflow memory write only. A missing historical assignee fails closed; ownership migration must be explicit. | installed SDK: `verbiflow/workflows.py` |

| Client SDK Surface | Use When | Inputs | Output And Handles To Preserve | Side Effect | Source |
| --- | --- | --- | --- | --- | --- |
| `SourcePage` | Implementing a custom `SourceAdapter.fetch_page(...)`. | `rows`, optional `checkpoint`, `source_finished`, `page_id`, `stats`. `checkpoint` must advance on non-terminal pages. | Source page consumed by `collect_source(...)`; preserve meaningful `page_id` and stats for debugging. | None by itself. | installed SDK: `verbiflow/source_adapters.py:23` |
| `SourceAdapter` / `SourceAdapter.fetch_page(...)` | A source has real cursor or resumability semantics that cannot be represented as static rows. | Adapter attributes: `source_kind`, `table`, `cursor_kind`, `resume_policy`; methods `checkpoint_key(config)`, `row_key(row)`, and `fetch_page(...)`. | `collect_source(...)` stores rows, checkpoints, objects, and observations from the adapter. | Depends on adapter; do not use it to bypass typed provider surfaces. | installed SDK: `verbiflow/source_adapters.py:32` |
| `SourceSpec` | Passing adapter defaults as one object. | `adapter`, optional `config`, `table`, `key`, `object_type`, `identities`, `source_ref`, `page_size`, `max_rows`. | Lets `collect_source(...)` use explicit defaults without repeating every arg. | None by itself. | installed SDK: `verbiflow/source_adapters.py:52` |
| `RowsSourceAdapter` | Already-available rows need paging, row keys, and object memory. | `rows` or callable rows, required `key`, optional `source_kind`, `table`. | Adapter emits `_vf_source_row_key` unless `collect_source(..., key=...)` overrides the stored item key. | None by itself; `collect_source(...)` writes workflow state. | installed SDK: `verbiflow/source_adapters.py:65` |
| `source_ref(source_kind, config)` | You need a stable source identity from source kind plus config. | `source_kind` string and JSON-like config mapping. | Returns stable `source_kind_<hash>` string; preserve it when source contract identity matters. | None. | installed SDK: `verbiflow/source_adapters.py:129` |

## Data Contracts

| Field | Meaning | Common Bug |
| --- | --- | --- |
| `table` | Workflow table where rows are materialized. | Reusing a table after changing schema makes old rows look valid. |
| `key` | Per-run `workflow_items.item_key`; must exist on every row. | Confusing adapter `row_key` with stored item key; pass `key=` to `collect_source(...)`. |
| `object_type` | Cross-run memory object class such as `contact`, `account`, or `company`. | Using customer-specific types in generic workflows. |
| `identities` | Ordered canonical aliases used to resolve one cross-run object. Every populated configured alias is remembered, so a rich row can link later partial observations. | Omitting canonical fields such as email/domain/LinkedIn URL, using names as authoritative identity, or expecting two already-conflicting objects to merge silently. |
| `source_kind` | Short class of source, for example `artifact_job`, `host_company_rows`, or `sequence_audience`. | Making it too specific and losing grouping across equivalent sources. |
| `source_ref` | Stable logical source identity. | Letting defaults be job-scoped when the workflow expects logical source-scoped resume. |
| `cursor_key` | Cursor/checkpoint identity under the source ref. | Reusing a cursor across different source contracts. |
| `job_id` / `artifact_id` | Backend job/artifact handles. | Dropping them, which makes later audit, report, or sequence staging impossible. |
| `checkpoint` | Serialized cursor state, row offset, source-finished flag, and rows materialized. | Treating it as provider freshness instead of local consumption state. |
| `objects` | New cross-run workflow objects inserted by this collection call. | Treating it as total observed rows; use `observations` for sightings. |
| `observations` | Source observations written for acquired rows. | Treating observations as deduped objects. |
| `source_window` | Provider request window selected by policy/config. | Forgetting to map it into the provider request before starting the job. |

## Examples

### Collect A Provider Artifact Job

Start the provider job in the owning provider reference, with source-window
filters and limits applied there. Then collect the returned `PendingJob`,
job mapping, or `job_id` with `ctx.collect_artifact_job(...)` using the exact
contract in `Workflow Runtime Surfaces`. This reference intentionally does not show a
generic provider-job starter because CRM, audience discovery, enrichment, web,
places, Clay, Crunchbase, and LinkedIn have different required inputs.

### Collect Existing Rows With Object Memory

```python
@workflow.step(writes="seed_accounts", key="company_domain")
def collect_seed_accounts(ctx: vf.workflows.WorkflowContext) -> None:
    source = vf.source_adapters.RowsSourceAdapter(
        [
            {"company_domain": "acme.com", "source": "manual"},
            {"company_domain": "example.com", "source": "manual"},
        ],
        key="company_domain",
        source_kind="manual_seed_accounts",
        table="seed_accounts",
    )
    ctx.collect_source(
        source,
        table="seed_accounts",
        key="company_domain",
        object_type="account",
        identities=("company_domain",),
        page_size=100,
    )
```

### Collect Host Rows

```python
@workflow.step(writes="companies", key="company_domain")
def collect_host_company_rows(ctx: vf.workflows.WorkflowContext) -> None:
    rows = [
        {"company_domain": "acme.com", "source": "host_capture"},
    ]
    ctx.collect_host_rows(
        rows,
        table="companies",
        key="company_domain",
        object_type="account",
        identities=("company_domain",),
        source_kind="host_company_rows",
        source_ref_value="host_capture_2026_06_07",
    )
```

### Sync Existing Sequence Audience For Suppression

Use `ctx.sync_sequence_audience(...)` when a workflow already has a real target
sequence/source/ingress mapping from config or sequence authoring. The target
must include real sequence ids; do not paste placeholder ids into a workflow.
This only reads existing audience state and writes workflow memory. Staging new
rows, promoting source runs, publishing, or sending belongs in `sequences.md`.

## Verification

After a source step runs, verify:

- The output includes expected `job_id`, `artifact_id`, `source_ref`,
  `cursor_key`, table, key, and row counts.
- `source_finished` matches the intended stop condition. If
  `client_limit_reached=True`, the source was intentionally capped and may need
  another run or a higher limit.
- Workflow rows exist in the expected table with the expected key field.
- New-object count and observation count make sense for dedupe expectations.
- Downstream row-level steps only started early when `downstream_ready=True`
  and `start_when="rows_available"` were intentional.
- Sequence-audience sync outputs include the expected `sequence_id`,
  `source_node_id`, `ingress_id`, imports, and lead list ids when used.

Prefer generated `status.py`, `export.py --all-tables --sql`,
`workflow.status()`, and `workflow.rows(...)` before raw SQLite. Query raw
SQLite only when the SDK inspection surface is insufficient.

## Troubleshooting

| Symptom | First check | Fix |
| --- | --- | --- |
| Rows are stale after code changed | Did the source step already complete? | Use workflow replay/reset from `workflow_operations.md`; code changes do not invalidate completed steps. |
| Recurring CRM/source run fetched the wrong window | Was `ctx.source_window` mapped into the provider request before job start? | Fix the provider call in the owning reference; source checkpoints cannot repair provider freshness. |
| `rows_available` did not fire | Did the collection save new rows and set `downstream_ready=True`? | Use `upstream_completed` for aggregate steps; only row-level idempotent consumers should rely on partial rows. |
| Preview mode rejects a source | Is it a non-`RowsSourceAdapter` custom adapter? | Use already-available rows in preview or run batch mode for provider-backed acquisition. |
| Artifact job finished empty | Is empty expected? | Pass `allow_empty=True` only for expected empty results; otherwise inspect the job/artifact in `artifacts_jobs.md`. |
| Partial artifact acquisition repeats rows | Inspect checkpoint `kind`, `row_offset`, and `source_finished`. | Artifact rows must checkpoint as `artifact_rows`; batch rows checkpoint as `artifact_batches`. |
| Object counts look lower than fetched rows | Are identities deduping existing objects? | `objects` counts new cross-run objects; use `observations` for source sightings. |
| Sequence audience suppression misses people | Was `sync_sequence_audience(...)` run with the intended `suppress_scope` and identities? | Re-sync existing audience before staging and route staging diagnostics to `sequences.md`. |

## Related References

| If you are doing this | Read next |
| --- | --- |
| Operating, replaying, repairing, or reporting from an existing root | `workflow_operations.md` |
| Syncing a Play revision/output or scheduling it | `workflow_schedules.md` |
| Designing the whole play across discovery, enrichment, reports, and activation | `workflow_composition.md` |
| Choosing people/account/company source strategy or provider-backed discovery | `audience_discovery.md` |
| CRM fields, filters, associations, active-customer/deal suppression | `crm.md` |
| Email/domain/profile/company/phone enrichment or resolver field mappings | `enrichment.md` |
| Row transforms, joins, mappings, CSV shaping, or semantic qualification | `row_operations.md` |
| Job handles, artifact paging/export, raw streams, and file uploads | `artifacts_jobs.md` |
| Sequence source staging, promotion, publish, send, or sequence audience diagnostics | `sequence_authoring.md`, `sequences.md` |
| Final report and Play UI bundle output | `workflow_reports.md` |
