---
name: registry-generation-e2e-test
description: Run an end-to-end registry artifact generation test. Generates artifacts for a set of connectors, optionally publishes to GCS, compiles, and compares against the prod registry. Use when validating that the new generation pipeline produces artifacts matching the legacy pipeline.
devin-enabled: true
icon: flask-conical
---

# Registry Generation E2E Test

Run a full end-to-end test of the registry artifact generation pipeline. Validates that locally generated artifacts match (or acceptably differ from) the production registry.

## Prerequisites

- **Repos:** `~/repos/airbyte-ops-mcp` (this repo) and `~/repos/airbyte` (connector metadata source)
- **Docker:** Running and accessible (required for `artifacts generate` to run connector specs)
- **Credentials:** `GCS_CREDENTIALS` or `GCP_GSM_CREDENTIALS` environment variable with a service account JSON key that has access to:
  - Read: `prod-airbyte-cloud-connector-metadata-service` (in GCP project `prod-ab-cloud-proj`)
  - Read/Write: `dev-airbyte-cloud-connector-metadata-service-2` (in GCP project `dev-ab-cloud-proj`)
- If `GCS_CREDENTIALS` is not set but `GCP_GSM_CREDENTIALS` is, export it: `export GCS_CREDENTIALS="$GCP_GSM_CREDENTIALS"`

## Modes

This skill supports two modes:

### Local Mode (no GCS writes)

Generates artifacts locally and compares them against prod by downloading reference artifacts read-only. No files are written to GCS. Use this for quick validation without side effects.

### GCS Mode (full pipeline)

Generates artifacts, publishes them to a datestamped directory in the dev GCS bucket, compiles the store, and compares against prod. Creates a permanent record of the test run. Use this for thorough validation.

## Step 1: Select Test Connectors

Choose at least 5 connectors covering multiple types. Use the latest GA version from each connector's `metadata.yaml` in the airbyte monorepo.

**Recommended default set** (covers manifest-only, Python, and Java):

| Connector | Type |
|-----------|------|
| source-faker | Python source |
| source-github | Python source |
| source-google-sheets | Manifest-only source |
| source-notion | Manifest-only source |
| destination-s3 | Java destination |

To resolve the current version for each connector:

```bash
# From ~/repos/airbyte, for each connector:
grep 'dockerImageTag:' airbyte-integrations/connectors/<connector-name>/metadata.yaml | head -1
```

Record each connector name and version. These are your test targets.

## Step 2: Generate Artifacts Locally

For each connector, generate artifacts using the CLI. All commands run from `~/repos/airbyte-ops-mcp`.

```bash
uv run airbyte-ops registry connector-version artifacts generate \
  --metadata-file ~/repos/airbyte/airbyte-integrations/connectors/<connector-name>/metadata.yaml \
  --docker-image airbyte/<connector-name>:<version> \
  --output-dir /tmp/e2e-artifacts/<connector-name> \
  --repo-root ~/repos/airbyte
```

**Failure criteria:** If generation fails for ANY connector, the test fails. Record the error and stop.

**Validation:** After generation, verify the output directory contains:
- `metadata.yaml`
- `cloud.json`
- `oss.json`
- `spec.json`

## Step 3: Choose Mode and Continue

### Local Mode: Mirror Prod and Compare Locally

Use the `store mirror` CLI with `--local` to download prod artifacts for the selected connectors:

```bash
uv run airbyte-ops registry store mirror \
  --source-store coral:prod \
  --local \
  --output-path-root /tmp/e2e-reference \
  --connector-name <connector-name-1> \
  --connector-name <connector-name-2> \
  --connector-name <connector-name-3> \
  --connector-name <connector-name-4> \
  --connector-name <connector-name-5>
```

**Failure criteria:** If the mirror fails, the test fails. Record the error and stop.

Then diff each connector's generated artifacts against the mirrored prod artifacts:

```bash
# For each connector and artifact (cloud.json, oss.json, metadata.yaml, spec.json):
diff /tmp/e2e-artifacts/<connector-name>/cloud.json \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/cloud.json
diff /tmp/e2e-artifacts/<connector-name>/oss.json \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/oss.json
diff /tmp/e2e-artifacts/<connector-name>/metadata.yaml \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/metadata.yaml
diff /tmp/e2e-artifacts/<connector-name>/spec.json \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/spec.json
```

Skip to **Step 6** after collecting diffs.

### GCS Mode: Publish, Compile, Compare

Continue with Steps 4 and 5 below.

## Step 4: Publish to Dev Store (GCS Mode Only)

Create a datestamped store ID:

```bash
STORE_ID="$(date +%Y%m%d)-e2e-$(openssl rand -hex 3)"
```

For each connector, publish the generated artifacts:

```bash
uv run airbyte-ops registry connector-version artifacts publish \
  --name <connector-name> \
  --version <version> \
  --artifacts-dir /tmp/e2e-artifacts/<connector-name> \
  --store coral:dev/${STORE_ID}
```

**Failure criteria:** If publish fails for ANY connector, the test fails. Do not attempt workarounds — hard fail and report the CLI error.

## Step 5: Compile and Compare (GCS Mode Only)

### Compile the store

```bash
uv run airbyte-ops registry store compile --store coral:dev/${STORE_ID}
```

**Failure criteria:** If compile fails, the test fails.

### Compare against prod

```bash
uv run airbyte-ops registry store compare \
  --store coral:dev/${STORE_ID} \
  --reference-store coral:prod
```

The compare command will report:
- **Tolerated diffs:** Timestamp fields (`registry_entry_generated_at`, `metadata_etag`, `metadata_last_modified`) — these are expected and suppressed.
- **Toleration violations:** Fields where presence or type differs between store and reference — these are warnings.
- **Real diffs:** Any remaining differences in artifact content.
- **Index diffs:** Global registry index differences (expected since the dev store only has the test connectors).

## Step 6: Write Structured Results

Create a results directory and write structured output:

```
/tmp/e2e-results/<STORE_ID or "local">/
  results.json        # Structured pass/fail per connector
  summary.md          # Human-readable summary
  diffs/              # Per-connector diff files
    <connector-name>.cloud.json.diff
    <connector-name>.oss.json.diff
    <connector-name>.metadata.yaml.diff
```

### results.json schema

```json
{
  "mode": "local|gcs",
  "store_id": "<STORE_ID or null>",
  "timestamp": "<ISO-8601>",
  "overall_result": "pass|fail",
  "connectors": [
    {
      "name": "<connector-name>",
      "version": "<version>",
      "type": "<manifest-only|python|java>",
      "steps": {
        "generate": {"status": "pass|fail", "error": null},
        "publish": {"status": "pass|fail|skipped", "error": null},
        "compile": {"status": "pass|fail|skipped", "error": null},
        "compare": {
          "status": "pass|fail|skipped",
          "tolerated_diffs": ["<field paths>"],
          "toleration_violations": ["<field paths>"],
          "real_diffs": {
            "cloud.json": {"keys_only_in_store": [], "keys_only_in_reference": [], "values_differ": []},
            "oss.json": {"keys_only_in_store": [], "keys_only_in_reference": [], "values_differ": []},
            "metadata.yaml": {"differs": true}
          }
        }
      }
    }
  ],
  "known_gaps": [
    "Missing fields: supportsDataActivation, supportsFileTransfer",
    "Missing ab_internal sub-fields: isEnterprise, requireVersionIncrementsInPullRequests",
    "Missing releases.rolloutConfiguration sub-fields: advanceDelayMinutes, initialPercentage, maxPercentage",
    "Empty packageInfo (missing cdk_version)",
    "Different generated.source_file_info schema (metadata_etag vs metadata_last_modified)",
    "YAML key ordering differences in metadata.yaml"
  ]
}
```

### summary.md

Write a human-readable markdown table summarizing pass/fail per connector per step, plus a section listing all unique diffs found with concrete examples.

## Step 7: Evaluate Results

**Overall PASS** requires ALL of:
- All connectors generated successfully
- All connectors published successfully (GCS mode)
- Compile succeeded (GCS mode)
- Compare completed without errors (GCS mode)
- All diffs are either tolerated or documented in `known_gaps`

**Overall FAIL** if ANY of:
- Any connector fails to generate
- Any connector fails to publish (GCS mode)
- Compile fails (GCS mode)
- Compare finds unexpected diffs not in `known_gaps`

## Step 8: Report Results

Send the user a summary including:

1. **Overall result**: PASS or FAIL
2. **Per-connector table**: showing status of each step
3. **Diffs found**: concrete examples of any differences
4. **Store location** (GCS mode): `gs://dev-airbyte-cloud-connector-metadata-service-2/<STORE_ID>/`
5. **Attach** `results.json` and `summary.md` files

If any new (previously unknown) diffs are discovered, file a GitHub issue in `airbytehq/airbyte-ops-mcp` documenting them — use the `create-github-issue` skill.

## Reference

- **Dev bucket:** `dev-airbyte-cloud-connector-metadata-service-2` (project: `dev-ab-cloud-proj`)
- **Prod bucket:** `prod-airbyte-cloud-connector-metadata-service` (project: `prod-ab-cloud-proj`)
- **Known gaps issue:** https://github.com/airbytehq/airbyte-ops-mcp/issues/452
- **Compare improvements PR:** https://github.com/airbytehq/airbyte-ops-mcp/pull/446
