# Testing

<!-- BEGIN GENERATED: guide-map -->
## Guide map

Read only the section needed for the next decision. Line ranges are exact
for this generated runtime target.

| Section | What it covers | Lines |
| --- | --- | --- |
| [Purpose](#purpose) | Scope, ownership, and the central operating question. | 25-35 |
| [Use This When](#use-this-when) | Tasks and conditions that belong in this guide. | 36-48 |
| [Do Not Use This For](#do-not-use-this-for) | Boundaries and the correct routes for excluded work. | 49-64 |
| [Mental Model](#mental-model) | Concepts and state boundaries needed for correct operation. | 65-90 |
| [Preflight](#preflight) | State, authority, inputs, and readiness to verify before execution. | 91-105 |
| [Risk And Side Effects](#risk-and-side-effects) | Effects, approvals, cost, privacy, and retry boundaries. | 106-115 |
| [Canonical Operating Pattern](#canonical-operating-pattern) | The normal end-to-end execution sequence. | 116-129 |
| [Data Contracts](#data-contracts) | Required identities, fields, evidence, handles, and durable shapes. | 130-198 |
| [Verification](#verification) | Evidence required before reporting success. | 199-276 |
| [Examples](#examples) | Representative usage patterns and code. | 277-351 |
| [Troubleshooting](#troubleshooting) | Failure diagnosis and corrective routes. | 352-363 |
| [Related References](#related-references) | Adjacent boundaries and when to cross them. | 364-375 |
<!-- END GENERATED: guide-map -->

## Purpose

Use this reference to prove a GTM rowset, artifact, workflow step, report, or
sequence preview is actually correct before moving to the next operation. The
goal is not "write some pytest"; the goal is to catch wrong source shape,
silent caps, bad joins, stale checkpoints, missing variables, and unsafe
activation rows before credits are spent or customer-visible state changes.

Testing here means data-quality verification plus focused regression checks
around the SDK calls that produced the data.

## Use This When

- A workflow step, source job, enrichment, qualification, report, or sequence
  preview produced rows and the next step depends on those rows being correct.
- You are about to stage, publish, send, sync a final Play snapshot, or show a
  business-facing report from generated data.
- Row counts changed unexpectedly, a table preview is empty, a provider job
  returned fewer rows than requested, or output labels/counts do not match what
  the table actually contains.
- You are changing a Skill reference, generated workflow template, source
  adapter, row operation, artifact handoff, or report view and need a regression
  test that proves the real contract.

## Do Not Use This For

- Generic Python unit-test style guidance. Use the nearest repository tests and
  normal engineering workflow.
- Exact callable contracts. Use `vf.sdk.describe(...)` and bounded
  `vf.sdk.read_source(...)`; inspect the nearest repository tests when running
  from a source checkout.
- Workflow run/repair/replay mechanics. Use `workflow_operations.md`.
- Watching whether an instrumented running job is stuck or wiring same-machine
  progress telemetry. Use `logging.md`; use `artifacts_jobs.md` for backend job
  handles, event streams, and durable artifact state.
- Source cursor/checkpoint theory. Use `workflow_sources.md`.
- Row transformation APIs and LLM row jobs. Use `row_operations.md`.
- Sequence authoring and publish/send approval. Use `sequence_authoring.md`,
  `sequences.md`, and `safety.md`.

## Mental Model

Every non-trivial GTM step should prove four things:

| Question | What To Check | Common Bug |
| --- | --- | --- |
| Did it produce the right amount? | row count, source-to-output coverage, caps, drop rate | provider cap, stale checkpoint, filter too tight |
| Does the shape match the next step? | required columns, null rates, variable coverage, mapping | wrong field name, missing email/domain, bad source schema |
| Does the content match the intent? | distributions, top values, geo containment, sampled rows | LinkedIn/company scrape artifact, wrong location, bad dedupe |
| Is it safe to advance? | preview status, copied ids, activation gates, side-effect counts | staging wrong rows, treating preview as final, hidden provider spend |

Samples catch individual bad rows. Aggregates catch systematic failure. Use
both. For high-risk rows, deterministic checks should run before LLM/AI checks:
validate domains, emails, IDs, counts, and mappings first; use AI only for
semantic qualification or fuzzy categorization.

Progress is not completion evidence. `ctx.progress(...)` proves only that a
numeric unit moved. `ctx.milestone(...)` records a semantic stage, and a
completed milestone must cite `run`, `output`, `job`, or `artifact` ids that
already resolve from the current run's durable workflow store. This prevents a
made-up identifier from becoming a completion signal; it does not judge whether
the cited evidence semantically proves the business claim. Before claiming the
whole task is complete, inspect the exact final bytes and receipts. For a
durable Play, verify the synced code revision, run, and declared outputs. If any
are absent, report the saved partial state.

## Preflight

1. Identify the output owner: local workflow table, artifact, source preview,
   AI-filter preview, report sidecar, or sequence audience.
2. Preserve handles before testing: `run_id`, `job_id`, `artifact_id`,
   sequence/source/ingress ids, report path, source run id, table name, and
   local host-capture handles such as `action_id`, `manifest_path`,
   `markdown_path`, `html_path`, `screenshot_path`, and `network_log_path`.
3. Inspect status before rows. A failed provider job and a legitimate empty
   result need different actions.
4. Confirm row identity keys: company domain, CRM object id, contact email,
   LinkedIn URL, source row id, or workflow item key.
5. Print small previews only after aggregate counts and null/coverage checks;
   otherwise a plausible sample can hide a broken population.

## Risk And Side Effects

| Operation | Risk |
| --- | --- |
| Local row counts, histograms, coverage, CSV checks | Local/read-only. |
| `vf.artifacts.inspect(...)`, artifact paging, job result inspection | Platform read. Preserve `artifact_id` and `job_id`. |
| `vf.rows.geo_classify(...)`, `vf.rows.ai_analyze(mode="qualify", ...)`, `vf.rows.ai_analyze(mode="enrich", ...)`, source/AI previews | Provider or Growth job; may consume credits. Run on a bounded sample first when shape is uncertain. |
| `vf.sequences.verify_copy(...)` | Read/validation call; use before draft creation or step updates. |
| Staging, publish, send, purchase | Not owned here. Read sequence/safety docs and require explicit approval. |

## Canonical Operating Pattern

1. Status: check job/workflow/preview status and terminal state.
2. Count: record total rows and compare to source rows, expected audience size,
   and previous run.
3. Shape: verify required columns, type expectations, null/empty rates, and
   downstream variables.
4. Distribution: compute top values and unexpected skew for categorical fields.
5. Coverage: measure input-to-output hits by stable key.
6. Delta: inspect rows dropped between steps and rows newly added since the
   prior run.
7. Sample: inspect 5-10 representative rows only after aggregate checks.
8. Gate: decide whether to advance, rerun, repair, or stop for user judgment.

## Data Contracts

### Workflow Tables

Workflow-local rows live in the workflow state store and are exposed through
workflow status/export/report helpers. Do not assume a first-class SQL table is
the canonical source unless the workflow materialized one. Use the generated
workflow helpers or `vf.workflows.open(...).status()` first; query raw SQLite
only when the SDK inspection path is insufficient.

Raw SQL is acceptable only after the rows have been intentionally materialized
or exported from workflow state, for example through generated export helpers
or local-store materialization. The canonical persisted workflow rows are not
"whatever table name looks plausible"; they are workflow item/output rows with
run identity, table identity, status, payload, and keys.

Required checks:

- table name and row kind match the label shown in the Play/report;
- stable key is present for dedupe and cross-step comparisons;
- output count labels match what the rows represent, for example contacts vs
  companies;
- step output was produced by the current source contract, not stale cached
  rows from an older workflow definition.

### Artifacts And Jobs

Provider/Growth jobs should be tested through job/artifact handles. Preserve
`job_id` and `artifact_id`; do not rerun the provider search just to inspect
rows. Use `vf.jobs.result(...)`, `vf.artifacts.inspect(...)`, and artifact
row paging/export surfaces to read committed output.

Required checks:

- job terminal status is success before trusting row counts;
- `artifact_id` exists when you need durable paging/export;
- response status distinguishes empty result from failed provider call;
- `total_matching`, returned rows, and requested limits are compared when the
  provider exposes them.

### Sequence Preview Rows

Sequence source preview and AI-filter preview are test surfaces, not activation.
They should prove source shape, variable availability, AI-filter pass/reject
reasons, and copy inputs before staging or publishing.

Required checks:

- source preview row count and sample rows exist before AI-filter preview;
- `passed_rows`, `rejected_rows`, `needs_data_rows`, and reasons are reviewed;
- copy variables used in sequence steps have coverage;
- cached preview results are not mistaken for a fresh production audience.

### Local Host Capture Outputs

`vf.host.web.capture_pages(...)` returns local evidence files, not a backend
artifact by default. Before using captured pages in a report, enrichment, or
workflow decision, verify:

- result `status`, `completed_count`, `failed_count`, and per-row `status`;
- `manifest_path` exists and matches the returned rows/counts;
- requested artifact paths exist and are non-empty: markdown/html/screenshots;
- `network_log_path` exists and has entries when `network_log=True`;
- failed rows preserve `seed_url`/`url` and error text so retries target only
  the failed pages;
- output files live under the workflow/report root when they are evidence for
  a Workflow Play.


## Verification

### Terminal Truth

Test low-level progress and semantic milestones as separate contracts. A
workflow may emit many numeric progress updates while exposing only a few
stable milestone stage ids. Verify that repeated milestone events for one stage
retain the same `stage_id`, and that every completed stage carries evidence
references that resolve to the exact produced state.

Do not turn open-ended agent quality into a scenario-specific deterministic
test. Mechanical assertions should prove IDs, receipts, bytes, row shape, and
state transitions. A reviewing agent should judge whether the evidence actually
answers the user's business request.

### Row Count

Compare every step output to:

- previous step row count;
- expected audience size;
- requested provider limit;
- prior run count for the same workflow/source window.

Large unexpected drops usually mean a bad filter, stale checkpoint, failed
join, or provider cap. Large unexpected growth usually means fanout, duplicate
identity keys, or a join without the intended key.

### Distribution And Top Values

For every categorical field that affects routing, qualification, or report
labels, inspect top values and skew.

Look for:

- top values that are clearly not in the field's semantic domain;
- `null`, empty string, `unknown`, or provider placeholders at the top;
- one value dominating a field that should vary;
- company/contact/table labels that do not match the actual row kind.

### Null And Variable Coverage

Measure null/empty rates for columns required by downstream operations. Before
sequence preview or staging, use variable coverage checks rather than eyeballing
copy templates.

Required downstream fields commonly include:

- `company_domain` for company-level dedupe and suppression;
- `contact_email` for email staging;
- `contact_linkedin_url` or public identifier for LinkedIn actions;
- sequence variables referenced in copy;
- CRM object ids when suppression or ownership depends on CRM associations.

### Per-Input Coverage

For each source input, count output hits. Zero-hit inputs need explanation, and
outlier high-hit inputs usually indicate unintended fanout.

Examples:

- company input -> discovered people;
- source contact -> enriched email;
- CRM company -> associated deals/customers;
- sequence source row -> staged/audience row.

### Cross-Step Deltas

Inspect dropped rows between step N and step N+1. If dropped rows look valid,
the filter or mapping is too tight. Also inspect newly added rows after a rerun
so stale local state is not mixed with the new source contract.

### Sampled Rows

After aggregate checks, inspect 5-10 rows that cover common, edge, and failed
cases. Do not sample only the first rows; sort/order effects can hide bad
tails.

## Examples

### Local Row Smell Test

```python
from collections import Counter

rows = vf.from_records(records, label="candidate_contacts")
items = rows.to_list()

assert items, "candidate_contacts produced 0 rows"
assert all(row.get("company_domain") for row in items), "missing company_domain"

top_titles = Counter(str(row.get("contact_title") or "") for row in items).most_common(10)
print("rows", len(items))
print("top_titles", top_titles)
print("sample", rows.sample(5, seed=7).to_list())
```

### Artifact Inspection

```python
result = vf.jobs.result(job_id).wait(timeout=600)
artifact_rows = result.rows(offset=0, limit=100)
print(result.job_id, result.job.status, result.artifact_id)
print("rows", len(artifact_rows), artifact_rows[:3])

if result.artifact_id:
    print(vf.artifacts.inspect(result.artifact_id, sample_limit=10))
```

### Sequence Preview Gate

```python
preview = vf.sequences.run_source_preview(sequence_id=sequence_id, source_node_id=source_node_id)
print(preview.get("status"), len(preview.get("rows") or []))

filter_preview = vf.sequences.run_ai_filter_preview(
    sequence_id=sequence_id,
    filter_node_id=filter_node_id,
    prompt=ai_filter_prompt,
)
filter_run_id = str(filter_preview["filter_run_id"])
print(filter_run_id, filter_preview.get("passed_count"), filter_preview.get("failed_count"))
print(
    vf.sequences.ai_filter_preview_status(
        sequence_id=sequence_id,
        filter_node_id=filter_node_id,
        filter_run_id=filter_run_id,
    )
)
```

### Report Source Verification

```python
source = vf.reports.source.workflow_table(
    "/absolute/path/to/state/workflow.sqlite",
    "final_contacts",
    run_id=run_id,
)
view = vf.reports.views.table(
    view_id="final_contacts",
    columns=("company_name", "contact_name", "contact_email"),
)
report = vf.reports.define(
    name="review_output",
    source=source,
    views=[view],
    title="Review output",
)
rendered = vf.reports.render(report, output_dir="/absolute/path/to/reports/review")
print(rendered["manifest_path"], rendered["serve_command"])
```

## Troubleshooting

| Symptom | Likely Cause | What To Do |
| --- | --- | --- |
| 0 rows but job succeeded | true empty result, wrong source input, or too-tight filter | Inspect source input, requested limits, and provider status; do not retry blindly. |
| 0 rows and job failed/errored | provider/API failure | Preserve `job_id`, inspect status/error, retry only if failure is retryable. |
| Table preview says count exists but UI shows no rows | preview rows were omitted, report artifact missing, or table paging path broken | Inspect report sidecars/artifact ids and the workflow output metadata. |
| One input creates most output rows | fanout or duplicate key | Group by source key and inspect the high-count input. |
| Geo output is wrong | hand-written location filter or ambiguous location | Use `geo.md` and `vf.rows.geo_classify(...)`; inspect supported targets first. |
| Sequence rows missing copy variables | source/config/copy mismatch | Run variable coverage and source/AI preview before staging. |
| Rerun did not reflect code changes | completed workflow step reused cached state | Use `workflow_operations.md` replay/reset guidance; do not trust old downstream rows after source contract changes. |

## Related References

| If you are doing this | Read next |
| --- | --- |
| Transforming or qualifying rows | `row_operations.md` |
| Waiting on jobs or paging artifacts | `artifacts_jobs.md` |
| Watching live same-machine job progress | `logging.md` |
| Operating, replaying, or repairing a workflow root | `workflow_operations.md` |
| Source windows, checkpoints, and object memory | `workflow_sources.md` |
| Report/Play UI output verification | `workflow_reports.md` |
| Sequence source/copy/filter preview and activation | `sequence_authoring.md`, `sequences.md`, `safety.md` |
| Geo containment | `geo.md` |
