Reporting protocols: reports, rendering, scheduled actions, alerts, notifications
Status: draft for review · revised 2026-10-06 (second review: simplified)
Seven optional, separately versioned vgi-rpc protocols that any VGI worker or standalone service may host:
| Protocol | Job | Methods | Spec |
|---|---|---|---|
vgi.reports.v1 |
Report store: revisions, publishing, redaction | 9 | reports.md |
vgi.report_render.v1 |
Render a report body into PDF, HTML or PNG | 4 | render.md |
vgi.schedules.v1 |
Run an action (render a report, run a query, a vendor kind) on a trigger, as its owner, with an optional condition and deliveries | 12 | schedules.md |
vgi.alerts.v1 |
Rules whose query returns the rows meeting a condition; one stateful instance per key; acknowledge, snooze, subscribe | 16 | alerts.md |
vgi.notify.v1 |
Deliver a message to email, Slack, Teams, Discord or a webhook | 4 | notify.md |
vgi.sql_tasks.v1 |
Run SQL that changes data on a trigger: load a query into a table (replace, append, merge, snapshot), incrementally by watermark, or a script of statements and CALLs in one transaction |
13 | sql_tasks.md |
vgi.delegations.v1 |
Hold an owner's delegations (grant + attach ticket per catalog) for unattended work | 3 | credentials.md |
Cupola's localStorage reports become the first client. The groundwork shipped
in every vgi-rpc port and VGI SDK (prerequisites.md):
multi-protocol hosting, the gRPC error model, Identity.v1 everywhere, and
grants accepted as bearer credentials.
Design rules
- The protocol is the unit of optionality. A host serves a protocol whole
or not at all; every method is required. What varies between hosts is data
in
get_*_info(e.g.writable), never the method set. - Composable across hosts. Nothing assumes co-location. A report can read
catalogs on several workers, live on a third, be scheduled on a fourth and
rendered on a fifth. What makes that work is two shared types in
credentials.md: a portable session description
(
DataSourcelist) and delegations: a grant plus a worker-sealed attach ticket per catalog, fromvgi_export_session(). - Wire contract only. Each spec says what crosses the wire and what a client can rely on. How the reference implementation builds its engine (leases, ticks, warm sessions, retries, retention) is in clearly marked, non-normative notes.
- Delegated grants are the only unattended data credential. No service
account ever reads data. Service principals exist only to call
infrastructure (a scheduler calling notify's
send).
Decisions
| Topic | Decision |
|---|---|
| Where reports live | Any host may implement vgi.reports.v1; a dedicated report service is the expected enterprise shape. A report names any number of catalogs in data_sources. |
| Report body | Standard envelope (title, path, tags, data sources, parameters); the body is opaque bytes tagged with a format such as cupola.evidence/1. |
| History | Append-only numbered revisions, one published pointer, tombstone redaction. Drafts and restore are client-side. |
| Access control | Each implementation's. The protocols carry advisory allowed_actions and standard refusals, not ACL management. |
| Data access | Any host may run a session: the scheduler, the renderer, an alert evaluator, any worker. A session is rebuilt anywhere from its DataSource list plus the owner's grants, with one ATTACH … bearer_token per source. |
| Unattended sessions | The client serializes the user's attached session with vgi_export_session(): per catalog, a grant (who) and an attach ticket (what, sealed by the worker, secrets included). Held as delegations, per owner per catalog (vgi.delegations.v1) by each host that runs unattended work; any runner reattaches with ATTACH … (bearer_token, attach_ticket). Schedules and rules carry no credentials. Lifetime is each data worker's policy; the framework default becomes unlimited unless a worker sets a maximum. |
| Scheduling | Generic actions: render_report, run_query, and <vendor>.<name> kinds. Optional condition_sql returns one boolean. Triggers are cron or once on the wire; presets are UI. |
| Report versions in schedules | The owner pins a revision, or follows the published revision or head and accepts that later edits run with their grants. |
| SQL tasks | Their own protocol on the shared engine, for SQL that changes data; schedules produce and deliver outputs. Bodies are a declarative load or a script; a CALL is a statement. Targets are any writable catalog (under the owner's delegation) or the host's own store. Watermarks are derived from the target (exactly once) or stored (at least once). One written database per transaction. |
| Alerts | Their own protocol. Rows are instances; notify on transitions; subscriptions with link-only details unless the owner shares them; acknowledge per instance, snooze per instance or rule, always with an end. |
| Delivery | vgi.notify.v1, channel-agnostic. Each service owns its destination policy and unsubscribe list. |
| Serverless | The reference scheduler is driven by an operator tick that a platform cron can call over HTTP; this is reference implementation, not protocol (schedules.md). |
| SQL access | Normative. Each protocol is also a schema in the host's VGI catalog: every method is a table function, annotated lists are tables, annotated create/update/delete back INSERT/UPDATE/DELETE, other verbs are procedures. Derived from the RPC definition by a generic adapter, never hand-written, so the two surfaces can't drift. |
| Naming | Report-only protocols are scoped (vgi.report_render.v1, later vgi.report_sharing.v1); the rest are general. |
| Discovery | Reflection for what an endpoint hosts; one catalog tag per protocol for where services live. |
Context
Cupola reports today (vgi-web-frontend/src/lib/evidence/): one
zod-validated JSON document per report (Evidence Markdown source, setupSql,
typed parameters, appearance and a single serviceUrl) in localStorage, keyed
per worker and per browser. History is a list of revisions with no author;
sharing is exporting a file; PDF export is built from the live DOM. Other
catalogs a report queries are referenced by alias in SQL and never declared.
What vgi-rpc and the SDKs give us (shipped 2026-10-06):
Worker.hosted_protocols()hosts extra protocols on every transport;vgi_rpc.Reflection.v1list_protocolssays which.- Errors carry
error_code,error_kindand typederror_details; clients exposeis_retryable()and never retry automatically. vgi_rpc.Identity.v1issue_grantmints a delegated credential for the calling principal (login no older than 900 s), and a worker with grant keys accepts its own sealedvgig1.grants asAuthorization: Bearer.- VGI's
ATTACHtakes abearer_tokenoption, treated as a secret.
Goals
- A report has one durable identity that any authorized person can open from any browser or client.
- Every saved change is an immutable, numbered revision with an author; publishing gives readers a stable version while editors keep working.
- A report can query any number of VGI workers and declares which.
- Reports, queries and other actions run on a schedule without a browser, as their owner, optionally only when a condition holds, and deliver to email, chat or webhooks.
- People get alerted when data meets a condition, once per affected entity, with the values that triggered it.
- Each protocol is optional and easy to implement; a worker can ship read-only reports from files in one line.
Non-goals for v1
- ACL management (a later
vgi.report_sharing.v1), a standard report body, users/groups/tenants (they come from the identity provider), bulk messaging, named publish channels, server-side drafts, and server implementations in every port (servers ship in Python and TypeScript; other ports get generated client types).
Architecture
flowchart TB
cupola["Cupola"]
reports["vgi.reports.v1
report service"]
sched["vgi.schedules.v1 + vgi.delegations.v1
scheduler"]
alerts["vgi.alerts.v1 + vgi.delegations.v1
alert evaluator"]
render["vgi.report_render.v1
renderer"]
notify["vgi.notify.v1"]
w1["sales worker
vgi.v2 + Identity.v1"]
w2["hr worker
vgi.v2 + Identity.v1"]
cupola -- "save, publish" --> reports
cupola -- "issue_grant, as the user" --> w1 & w2 & reports
cupola -- "vgi_export_session, put_delegations,
schedules, rules" --> sched & alerts
sched -- "read revision (owner's grant)" --> reports
sched -- "body + sources + grants" --> render
render & sched & alerts -- "ATTACH … bearer_token (owner's grant)" --> w1 & w2
sched & alerts -- send --> notify
The boxes are roles, not deployments: one process may host several of them
(the reference vgi-report-serve hosts reports, schedules, SQL tasks, alerts and
delegations), or each may be its own service. Every arrow carrying data access uses
the owner's grant for the location it goes to, and nothing else.
Shared conventions
Naming and versioning. vgi.<name>.v1, protocol_version = "1.0.0".
Minor versions only add methods or defaulted fields; breaking changes ship as
.v2, hosted side by side. No unions on the wire: every choice is an explicit
kind field plus plain values.
Errors
Every refusal is a vgi-rpc error with a canonical code, a reason
(error_kind) and catalog details. Kinds are shared across the protocols.
error_kind |
error_code |
Details | Client behaviour |
|---|---|---|---|
not_found |
NOT_FOUND |
ResourceInfo; optional access hint |
Show "Ask {contact} for access" when a hint is present |
action_denied |
PERMISSION_DENIED |
ErrorInfo.metadata {action} |
Explain; don't retry |
read_only_service |
PERMISSION_DENIED |
ResourceInfo |
Hide editing; writable should already have said so |
conflict |
ABORTED |
ResourceInfo, ErrorInfo.metadata with the current version or head (head_revision_id, updated_by, updated_at) |
Reload, then review, reapply or save as a copy |
invalid_request |
INVALID_ARGUMENT |
BadRequest |
Show field-level detail |
grant_required |
FAILED_PRECONDITION |
PreconditionFailure, one {type: "GRANT", subject: <location>} per location |
Connect or reauthenticate those sources |
quota_exceeded |
RESOURCE_EXHAUSTED |
QuotaFailure; RetryInfo for rate limits |
Show the limit |
service_unavailable |
UNAVAILABLE |
RetryInfo (required) |
Retry with backoff; never cache |
The access hint on not_found is LocalizedMessage, Help.links (request
access) and ErrorInfo.metadata.access_contact; no custom detail type.
Visibility. Return not_found for objects the caller can't read, so ids
can't be probed; list methods filter by read permission.
Permission hints. Returned objects carry allowed_actions for the caller:
read, edit, publish, redact, delete, schedule, run, manage, plus
alert actions. Advisory; the server enforces on every call.
Principals. Authorship is PrincipalRef {id, display_name, email}, id
being AuthContext.principal.
Concurrency and idempotency. Mutations of existing objects take
expected_revision_id or expected_version (conflict on mismatch). Creates,
sends and runs take a client-chosen request_id; a repeat within 24 hours
returns the original result.
Lists are producer streams with a fixed Arrow schema, paged by continuation
tokens. Times are timestamp[us, UTC] with IANA zone names. Parameter
values are tagged ParamValue records whose literal payload is JSON text.
body_sha256 is over the body bytes as sent.
Transports. Hosted on every transport through hosted_protocols(). On
HTTP callers have identities; on stdio and unix the caller is the operator,
which suits tests and single-user tools. Grants are HTTP only.
Unattended sessions across workers
The full model is in credentials.md. In short:
- Serialize: while logged in, the client runs
vgi_export_session(). Each attached worker returns a grant (who) and an attach ticket (what: the attachment's options, secrets included, sealed so only that worker can open them). Only catalogs the user attached themselves are exported. - Store: each host that runs unattended work keeps the owner's delegations,
per catalog, via
put_delegations. Never returned. - Reattach: any runner attaches each report source with
ATTACH … (bearer_token '<grant>', attach_ticket '<ticket>'), only at the delegation's own location, only for that job. It never sees an option or secret.
Known limits:
- Grants carry no identity-provider claims. A claim-based
AccessPolicy, on a data worker or the report store, sees unattended work with none and must decide from the principal or fail closed. - Scopes are advisory; by default a grant is as powerful as its owner on that worker.
- Non-VGI attachments (files,
:memory:, other database types) can't be serialized. - With no maximum lifetime set, grants and tickets never expire and are revoked by revoking the delegation or rotating the worker's grant key.
Discovery
- Reflection.
list_protocolson an endpoint says which protocols it hosts. Clients never probe by calling and catching. - Catalog tags. A worker that relies on services elsewhere returns them in
CatalogAttachResult.tags, one per protocol, followingvgi_secret_service_url:vgi_reports_url,vgi_report_render_url,vgi_schedules_url,vgi_alerts_url,vgi_notify_url. Values must be https (localhost excepted) and same-site or on an operator allowlist; the client confirms each with reflection.
When catalogs in one session point at different report services, each report lives in exactly one, and the library groups by service.
SQL binding
Every protocol here except the scheduler driver is also usable from SQL. A host
that serves a protocol over vgi-rpc and also serves vgi.v2 exposes the same
protocol as a schema in its VGI catalog, so after
ATTACH 'ops' (TYPE vgi, LOCATION 'https://reports.example.com');
ops.sql_tasks.tasks lists tasks, INSERT INTO ops.schedules.schedules …
creates a schedule, and CALL ops.sql_tasks.run_now(task_id := '…') runs one.
The binding is normative: SQL written against one implementation runs against
another.
It is derived, never hand-written. The SQL binding is a function of the RPC
definition plus a small per-protocol annotation (which list is a table, which
methods back its INSERT, UPDATE and DELETE). Implementations write the RPC
service only; a generic adapter in each SDK serves the catalog from it, so the
two surfaces cannot drift. Each spec's "SQL binding" section is that
annotation, and nothing else.
Derivation rules
- Schema.
vgi.<name>.v1is the schema<name>; a.v2hosted beside it is<name>_v2. A host that serves several protocols has one schema each. - Every method is a table function, named as the method, with the method's
parameters as named arguments and its result as rows: a unary method returns
one row (or one row per list element it returns), a stream returns its
batches.
CALLinvokes them. This rule is total: no method is left out, so nothing reachable over RPC is unreachable from SQL. get_*_info()is also the one-row tableinfo.- A list method may be annotated as a table, named for its resource. Columns are the list's row schema exactly. Filters on columns that match the method's arguments are pushed down to them; every other filter is applied by DuckDB after the scan. A list with a required argument is never a table; it stays a table function.
- Fetched columns. A table may name a get method. Columns only that method
returns (a report's
body) appear in the table but are fetched per row only when a query selects them. - DML is annotated, per table.
INSERTcalls the create method (one call per row, or one call per statement when the method takes a list);UPDATEcalls the update method with the row as changed, passing the row's oldversion(or revision id) as the expected value;DELETEcalls the delete method the same way. Arguments the method takes beyond the row (request_id, a commitmessage) are extra columns: writable,NULLon read.DELETEcan't set columns, so it uses those arguments' defaults; a delete that needs one (drop_target) is aCALL. A table without an annotation is read-only. A verb that isn't create, update or delete (publish,run_now,acknowledge) is only ever a procedure, soSELECTnever has side effects. - Types are the RPC's. The RPC already speaks Arrow: records are
STRUCTs, lists areLISTs,kindfields areVARCHAR. Field names are column names verbatim, so an RPC field name must be one DuckDB accepts unquoted (grantis fine; check new names against DuckDB's keyword list). - Write-only fields (grants, tickets) are accepted by
INSERTand read back asNULL. - Identity and access are the ATTACH's: the same caller, the same
AccessPolicy, the samenot_foundfor objects the caller can't read.
Semantics
- Concurrency. An
UPDATEorDELETEwhose row changed since the scan read it fails withconflict, as over RPC. - Idempotency.
request_idis an optionalINSERTcolumn; leftNULL, the adapter generates one, and the insert is then not retry-safe. - Autocommit. Each row's call takes effect when the statement executes and
ROLLBACKcan't undo it, so mutations are refused inside an explicit multi-statement transaction (invalid_request). Reads are allowed anywhere. A statement that fails part-way reports how many rows it applied. - Errors. The SQL error text is
[<error_kind>] <message>, followed by the catalog details as JSON, so a script can still tellconflictfromgrant_requiredand which locations need a grant. - One written database per transaction applies as everywhere in DuckDB: a statement that writes a protocol table writes nothing else.
Drift guards
- The adapter builds the catalog from the RPC protocol class at startup; there is no second definition to keep in step.
scripts/regen_generated.py --checkregenerates each spec's binding tables from the protocol classes and annotations, as it does for thevgi.v2registry.- Each protocol's conformance suite runs over both surfaces, and the extension
runs cross-SDK sqllogictests for every binding (
test/sql/integration/ <protocol>/).
Developer experience
from pathlib import Path
from vgi.reports import AccessPolicy, ReadOnlyReportStore, ReportsProtocol
from vgi.reports.storage import SharedStorageReportStore
from vgi.worker import Worker
class SalesWorker(Worker):
functions = [...]
@classmethod
def hosted_protocols(cls):
# Ship .cupola-reports.json exports as read-only reports.
return [(ReportsProtocol, ReadOnlyReportStore.from_files(Path(__file__).parent / "reports"))]
class TeamWorker(Worker):
functions = [...]
@classmethod
def hosted_protocols(cls):
return [(ReportsProtocol, SharedStorageReportStore.from_env(policy=OwnerPolicy()))]
class OwnerPolicy(AccessPolicy):
def allowed_actions(self, auth, report) -> set[str]:
if report.created_by.id == auth.principal:
return {"read", "edit", "publish", "redact", "delete", "schedule"}
return {"read", "schedule"} if report.published_revision_id else set()
AccessPolicy is the one function an implementor writes to own access, and it
decides from auth.principal, so it works for unattended (grant) callers too.
Reference stores sit on FunctionStorage, so sqlite, Azure SQL and Cloudflare
Durable Objects come free. Conformance is per protocol: every method, the
error mapping, concurrency, idempotency and authorship; pytest parameterized by
URL so the TypeScript implementation runs it too.
Implementation plan
The build plan, its status and the decisions made along the way are in IMPLEMENTATION.md. It is the single source for what to build next.
Review history
First review (principal engineer, BI UX lead, DX lead): grants accepted as
bearers (shipped), credentials bound to locations, run uniqueness, no unions,
the gRPC error model, test_run, plain-English trigger previews, structured
run errors, labels on data sources, writable services.
Second review (senior engineer, simplification and composability). Taken:
| Finding | Change |
|---|---|
| Three shapes for one credential, keyed by alias | One Delegation per owner per catalog; DataSource is the shareable description |
Grants per schedule: renewal toil, and purpose named a schedule id before it existed |
Per owner per catalog in vgi.delegations.v1, minted by vgi_export_session() |
| Minting at author-controlled locations leaks the user's session | Mint only at attached or confirmed locations |
| "Same grant" for the secret service can't verify there | Replaced by worker-sealed attach tickets, which carry secret options without exposing them |
| Report sources copied at schedule creation go stale | Taken from the revision rendered, every run |
| Claim-based policies on the report store fail for grant callers | Documented; reference AccessPolicy decides by principal |
| Notify assumed co-location (unsubscribe, attachments) | Notify owns the suppression list; attachments use signed URLs |
| Server drafts, publications log, undelete, copied_from, labels | Removed; reports 15 → 9 methods |
| Trigger presets and misfire policy on the wire | Cron or once; presets are UI |
| Engine rules (driver, leases, retries, retention, session hardening, grouping) as protocol | Moved to non-normative reference notes |
| Merges | publish unpublishes; set_enabled, reauthorize, retry_run folded in; one conflict kind; set_acknowledged; no get_notification or partial_ok |
Not taken, by decision: custom action kinds stay; alert subscriptions stay; schedules may follow a report's latest revision as an accepted risk; the unlimited default grant lifetime stands.
Resolved questions
| Question | Answer |
|---|---|
Features in protocol_hash? |
Moot: features is always empty |
| Publish channels? | One pointer |
| History removal? | Tombstone redaction |
| Grant lifetime? | The worker's; framework default unlimited unless set |
| DuckDB tables? | Yes, on the reference service, read-only |
| Artifact retention? | Operator policy (reference: 90 days) |
| Tags? | One flat tag per protocol |
| Notify host? | New vgi-notify repo |
| Scheduling others' reports? | Anyone with schedule; runs as its owner; pin or follow |
| Where data access runs? | Anywhere; sessions are portable |
| Grant storage? | Per owner per location |
Open questions
None outstanding.
Unattended sessions
Reports, schedules and alerts read data from catalogs on other workers, and the work may run on another host while nobody is logged in. This file defines how a user's attached DuckDB session is serialized so that any runner, anywhere, can reattach it later as that user, without ever seeing the user's secrets:
- Attach tickets: each attached worker seals the attachment's options, secret ones included, into a ticket only it can open, and agrees to honour it later.
- Delegations: one per owner per attached catalog, deposited with whichever
host runs unattended work (
vgi.delegations.v1).
Shared conventions are in README.md. The new worker method and extension pieces are planned in attach-tickets-plan.md.
Serializing a session
While the user is attached and logged in, the client runs:
SELECT * FROM vgi_export_session();
-- alias | location | catalog_name | grant | ticket | expires_at | status
-- sales | https://sales.example.com | main | vgig1.… | vgia1.… | … | ok
-- hr | https://hr.example.com | people | vgig1.… | vgia1.… | … | ok
-- tmp | :memory: | | | | | not_vgi
One row per attached catalog. For each VGI attachment the extension asks that worker for two things, using the session's own credentials:
- A grant (
vgi_rpc.Identity.v1issue_grant): who the runner acts as. - An attach ticket (
seal_attach): what to attach. The worker seals the options the catalog was attached with, secret ones included, together with the catalog name and the caller's principal. Only that worker can open it.
Rows the worker can't or won't serialize say why in status (not_vgi,
not_supported, stale_login, refused) instead of failing the whole export.
A runner on any host reattaches with both, and never sees an option or a secret:
ATTACH 'sales' (TYPE vgi, LOCATION 'https://sales.example.com',
bearer_token 'vgig1.…', attach_ticket 'vgia1.…');
The grant authenticates the runner as the user. The worker opens the ticket, checks its principal matches the grant's, and attaches with the sealed options exactly as the user did.
Two parts, deliberately
The grant and the ticket stay separate:
- Grants already work everywhere. Every port and SDK authenticates
vgig1.bearers today. Tickets add no new authentication path. - A ticket is not a credential. It carries no authority. Without a grant for the same principal it attaches nothing, so issuing one needs no fresh login, and one leaking on its own exposes nothing usable.
- Secrets stay out of identity. Secret options never enter
AuthContextclaims, where they could be logged or forwarded. - They change at different rates. A rotated upstream API key needs a new ticket; a new login only needs new grants.
To the runner they are one row, held and presented together.
The session description in a report
A report envelope carries only what's safe to share and needed to show a "connect these sources" checklist:
@dataclass
class DataSource:
alias: str # catalog name in the session, e.g. "sales"
location: str # worker URL
catalog_name: str = "" # remote catalog, if not the worker's default
label: str = "" # "Sales warehouse (prod)", for people
required: bool = True
Cupola fills it from the non-secret columns of vgi_export_session() when the
author saves, with a checklist so the author can untick catalogs the report
doesn't use. No grants or tickets ever go into a report: they belong to the
person whose credentials they carry.
Delegations
@dataclass
class Delegation:
location: str # normalized worker URL
catalog_name: str
grant: str # write-only
ticket: str # write-only
expires_at: float # the earlier of the two; +inf when neither expires
Normalized location: scheme, lowercased host, explicit port, and path without a trailing slash, so two workers behind one host on different paths stay distinct.
vgi.delegations.v1
Hosted by any service that runs work unattended (in practice beside
vgi.schedules.v1 and vgi.alerts.v1). The caller is always the owner.
| Method | Kind | Purpose |
|---|---|---|
put_delegations(delegations) |
U | Store or replace the caller's delegations, keyed by (location, catalog_name) |
list_delegations() |
S | The caller's delegations with expires_at and updated_at. Never grants or tickets |
revoke_delegation(location, catalog_name) |
U | Destroy one delegation |
To run a report, query or rule, the runner takes each DataSource, looks up the
owner's delegation for its (location, catalog_name), and attaches it under
the source's alias. A missing delegation is grant_required, naming the
source.
Schedules and rules carry no credentials of their own.
SQL binding
Schema delegations. Derived by the rules in README.md; this is the annotation.
| Table | List / get | INSERT |
UPDATE |
DELETE |
|---|---|---|---|---|
delegations |
list_delegations |
put_delegations, one call per statement (replaces by key) |
revoke_delegation |
grant and ticket are write-only. The columns match vgi_export_session(),
so delegating a whole session is one statement and no credential appears in
the SQL text:
INSERT INTO ops.delegations.delegations BY NAME
SELECT * EXCLUDE (alias, status, message) FROM vgi_export_session()
WHERE status = 'ok';
Who does what
| Step | Who | Rule |
|---|---|---|
| Issue | Each attached worker | A grant needs a login no older than 900 s. A ticket needs only an authenticated caller. Either can be refused |
| Export | The client, as the user | vgi_export_session(), only for catalogs the user attached themselves. A report author can't make a reader export at a location the reader never attached |
| Store | Each host that runs unattended work | Encrypted under a key outside its database; never returned; each host keeps its own copy |
| Reattach | Whatever host runs the session | Presents a delegation only to its own location, only for that job |
Nothing assumes co-location: report store, scheduler, renderer, alert evaluator and data workers can all be different hosts. When a job moves between them (a scheduler asking a renderer), the delegations for that job travel with it and are dropped when it ends.
Scheduling someone else's report. The schedule's owner reattaches with their own delegations, exported from their own session, never the author's. If the owner hasn't attached a source the report needs, they're asked to before saving the schedule.
Renewal. Whenever the owner logs in, Cupola re-exports the sessions behind
their expiring delegations and calls put_delegations. Delegations from workers with no
maximum lifetime never need it. A rotated secret needs a fresh export from a
session attached with the new value.
Revocation. revoke_delegation destroys the only copy on that host.
Rotating a worker's grant key invalidates its grants; rotating its
VGI_SIGNING_KEY invalidates its tickets.
What still can't be done unattended
- Identity-provider claims. A grant's
AuthContexthas the owner'sprincipal, not their groups or tenant. AnyAccessPolicythat reads claims, on a data worker or the report store, must decide from the principal or fail closed. - Scopes are advisory. By default a grant is as powerful as its owner on that worker.
- Non-VGI attachments (local files,
:memory:, other database types) can't be exported. A report that depends on one can't run unattended.
vgi.reports.v1
A report is a standard envelope plus an opaque body, with a linear chain of
immutable numbered revisions, one published pointer, and tombstone redaction.
Every method is required; a host that can't accept writes says
writable = false and refuses them. Shared conventions are in
README.md.
Model
- Revision. An immutable snapshot of envelope and body with
revision_number(1, 2, 3 …, shown as "v7"), author, time,kind(edit,agent,restore,import) and message. The newest is the head. - Publication.
published_revision_idpoints at the revision readers see.publishmoves it; publishing an empty revision id unpublishes. Revision rows recordpublished_atandpublished_bywhen they were published. - Redaction.
redact_revisionreplaces a revision's envelope and body with a tombstone that keeps its number, original author and time, and records the redacting actor, time and reason. Numbering never gaps. The head and the published revision can't be redacted; commit or publish a replacement first. - Deletion.
delete_report. Whether that is soft, and for how long, is implementation policy. - Drafts and restore are client-side. Clients keep their working copy
locally (Cupola already does) and commit with
expected_revision_id; restore is reading an old revision and committing it withkind = "restore".
Envelope
@dataclass
class ReportEnvelope:
title: str
description: str = ""
path: str = "" # library folder, "/"-separated
tags: list[str] = []
body_format: str # "cupola.evidence/1"
data_sources: list[DataSource] = [] # credentials.md
parameters: list[ParameterSpec] = []
@dataclass
class ParameterSpec:
key: str # $key in SQL
label: str
type: str # text | number | date | boolean | select | multi_select | date_range
required: bool = False
default_json: str = "null"
data_sources says which catalogs the report needs, so a renderer, a scheduler
or another browser can rebuild its session. parameters is the report's public
interface, so a schedule editor can show inputs without parsing the body.
Records add report_id, head_revision_id, published_revision_id,
created_at/by, updated_at/by, body_sha256 and allowed_actions.
Revision rows add published_at/by and, once redacted, redacted_at/by and
redaction_reason.
Methods
U = unary, S = producer stream. All are required.
| Method | Kind | Purpose |
|---|---|---|
get_report_service_info() |
U | display_name, writable, body formats, size limits |
list_reports(query, path_prefix, tags, owned_by_me, published_only) |
S | Library rows without bodies, filtered by read permission |
get_report(report_id, revision_id="") |
U | Envelope and body. With no revision id, editors get the head and everyone else the published revision; revision_served says which |
list_revisions(report_id) |
S | History, newest first, tombstones included |
create_report(request_id, envelope, body, message) |
U | New report with revision 1 |
commit_revision(report_id, expected_revision_id, envelope, body, kind, message) |
U | New head; conflict if the head moved |
publish(report_id, revision_id, expected_published_revision_id) |
U | Move the published pointer; empty revision_id unpublishes |
delete_report(report_id, expected_revision_id) |
U | |
redact_revision(report_id, revision_id, reason) |
U | Requires redact |
Errors: not_found (with the optional access hint), action_denied,
read_only_service, conflict, invalid_request, quota_exceeded,
service_unavailable (README.md).
Unattended reads. A scheduler or renderer reads reports with the owner's
grant on the report service. Grants carry no identity-provider claims, so an
AccessPolicy that authorizes by claims refuses them; it must decide from the
principal (credentials.md).
SQL binding
Schema reports. Derived by the rules in README.md; this is the annotation.
| Table | List / get | INSERT |
UPDATE |
DELETE |
|---|---|---|---|---|
reports |
list_reports / get_report (fetches body) |
create_report |
commit_revision, expected = the row's head_revision_id |
delete_report, same |
list_revisions has a required report_id, so it stays a table function.
publish and redact_revision are procedures. Extra columns: request_id and
message on INSERT; message and kind on UPDATE.
SELECT title, path, published_revision_id FROM ops.reports.reports
WHERE path LIKE '/finance/%';
SELECT revision_number, author, kind, message
FROM ops.reports.list_revisions(report_id := 'r_1');
CALL ops.reports.publish(report_id := 'r_1', revision_id := 'rev_7',
expected_published_revision_id := 'rev_5');
Read-only publisher
ReadOnlyReportStore.from_files(path) serves .cupola-reports.json exports
from a package directory and implements the whole protocol: reads serve the
files, writes are refused with read_only_service, writable = false. The
revision id is the content hash and the published revision is the head.
Import
An import of a .cupola-reports.json file is create_report followed by
commit_revision for each historical revision, kind import, with the original
kinds kept in the message. Cupola's "remove from history" maps to
redact_revision.
vgi.report_render.v1
Rendering a report is a job: start it, poll it, collect artifacts. The first
implementation is headless Cupola, because Cupola reports render only in a
browser; other renderers can register for other body formats. The scheduler
uses it for render_report actions and Cupola for large exports. Shared
conventions are in README.md.
Why jobs, not one call. A render runs every query, waits for charts to settle and builds a PDF. That takes longer than HTTP intermediaries tolerate, and the caller needs to cancel and retry it.
@dataclass
class RenderRequest:
body_format: str
body: bytes # the revision to render, passed inline
data_sources: list[DataSource] # the session to rebuild (credentials.md)
delegations: list[Delegation] = [] # owner's grant + ticket per source, this job only
parameter_values: list[ParamValue] = [] # literals; the caller resolves relative tokens
outputs: list[str] = ["application/pdf"] # also text/html, image/png (page-1 preview)
locale: str = "en-US"
time_zone: str = "UTC"
timeout_seconds: int = 600
The caller passes the body inline. A scheduler has already read the revision
(it needs the envelope's data_sources to check grants), so the renderer never
needs credentials for the report store and is a function of body, sources,
parameters and grants.
| Method | Kind | Purpose |
|---|---|---|
get_renderer_info() |
U | Body formats, output media types, timeout ceiling, allowed source locations |
start_render(request_id, request) |
U | Returns job_id, status queued |
get_render(job_id) |
U | Status (queued, running, succeeded, failed, cancelled), poll_after_seconds, artifacts, and on failure the cause's code, kind and data source |
cancel_render(job_id) |
U | Best effort |
Artifacts are {name, media_type, size_bytes, sha256, url, expires_at},
with a short-lived download URL. Callers that keep artifacts copy them.
Credentials. The renderer follows the presenting rule in
credentials.md: a grant goes only to the
location it was minted at, and is dropped when the job ends. It also enforces
an operator allowlist of source locations, published in get_renderer_info,
so a report can't point a session at an arbitrary host.
SQL binding
Schema report_render. Derived by the rules in README.md; this is the annotation.
No tables beyond info: render jobs can't be listed. start_render,
get_render and cancel_render are table functions, so a script can start a
render and poll it; Cupola and schedulers use RPC.
Reference renderer (non-normative)
- Headless Cupola keeps today's isolation: sandboxed components run in an opaque-origin iframe that can't read tokens.
- Always produce tagged, accessible PDFs.
vgi.schedules.v1
A schedule runs an action on a trigger, as its owner, and delivers what the action produced. Rendering a report is one action; running a query is another; implementations may add their own. Each firing is a run that records what ran, with which values, whether its condition held, and where the outputs went. Shared conventions are in README.md; sessions and grants are in credentials.md.
A host of vgi.schedules.v1 also hosts vgi.delegations.v1: a schedule
carries no credentials, and runs reattach each source with the owner's
delegation for it.
Model
@dataclass
class Schedule:
title: str
trigger: Trigger
action: Action
condition_sql: str = "" # empty = always; see Conditions
parameter_values: list[ParamValue] = []
deliveries: list[Delivery] = []
enabled: bool = True
@dataclass
class Trigger:
kind: str # cron | once
cron: str = "" # 5-field cron (croner dialect, vgi-crontimes)
run_at: str = "" # kind=once: ISO 8601
time_zone: str = "UTC" # IANA
start_at: str = ""
end_at: str = ""
@dataclass
class Action:
kind: str # render_report | run_query | <vendor>.<name>
render_report: RenderReportAction = ... # kind = render_report
run_query: RunQueryAction = ... # kind = run_query
data_sources: list[DataSource] = [] # run_query and vendor kinds; credentials.md
custom_json: str = "" # kind = <vendor>.<name>, validated by that implementation
@dataclass
class RenderReportAction:
report: ReportRef # {service_url, report_id, revision_id}
track: str = "published" # pinned | published | head
outputs: list[str] = ["application/pdf"]
@dataclass
class RunQueryAction:
setup_sql: str = ""
sql: str # its result is the action's output
output_format: str = "parquet" # none | arrow | parquet | csv
@dataclass
class ParamValue:
key: str
kind: str # literal | relative
json: str = "" # kind=literal
relative: str = "" # kind=relative, e.g. previous_month, last_n_days:7
@dataclass
class Delivery:
destinations: list[Destination] # vgi.notify.v1 {kind, address}
inline: str = "summary" # none | summary | image (report) | table (query)
attach: list[str] = [] # output media types to attach; empty = link only
Schedule editors offer presets ("weekdays at 8:00"); they compile to cron, and
preview_trigger describes the cron back in plain English. A missed firing
after downtime runs once.
Records add schedule_id, owner, version, next_fire_at, last_run,
disabled_reason, credentials (per required location: ok, expiring with
a time, expired, missing) and allowed_actions.
Actions
| Kind | Session | Outputs |
|---|---|---|
render_report |
The report's data_sources, rendered by vgi.report_render.v1 |
The rendered media types |
run_query |
The action's data_sources |
The result in output_format, plus row_count |
<vendor>.<name> |
The action's data_sources |
Defined by the implementation |
render_reportreads the revision to run on every run (pinned, latest published, or head) with the owner's grant on the report service, takesdata_sourcesfrom that revision's envelope, checks the owner holds grants for them, and sends body, sources and grants to a renderer.run_queryrebuilds the session fromdata_sourcesand the owner's grants, runssetup_sqlthensqlwith parameters bound as$key, and stores the result. Where the session runs is the implementation's choice.- Custom kinds (
acme.refresh_extract) carry their arguments incustom_json, validated at create time. They declaredata_sourcesso grants are checked the same way. A client that doesn't know a kind can still list, pause, run and delete the schedule and read its runs. get_scheduler_infolists the kinds a scheduler supports; any other kind isinvalid_request.
Tracking report versions. pinned runs exactly one revision. published
and head follow the report as it changes, which means later edits by the
report's editors run with the schedule owner's grants and go to the owner's
destinations. That is an accepted risk the owner chooses; schedule editors warn
when someone follows a report they can't edit, and runs record the revision
they rendered.
Conditions
condition_sql decides whether a run proceeds. It is one query that must
return exactly one row with one BOOLEAN column, run in the action's
session with the same parameters.
| Result | Run outcome |
|---|---|
true |
Proceed and deliver |
false / NULL |
skipped (condition_false / condition_null) |
| Any other shape | failed, condition_invalid |
For run_query it runs after the query and can read the result as the view
action_result ("send only if there are late orders" is SELECT count(*) > 0
FROM action_result). For render_report and custom kinds it runs before the
action, so a quiet day costs one small query instead of a render.
Methods
U = unary, S = producer stream. All are required.
| Method | Kind | Purpose |
|---|---|---|
get_scheduler_info() |
U | Action kinds, output formats, minimum interval, relative tokens, limits |
preview_trigger(trigger, count) |
U | Next N fire times, a plain-English description, DST warnings |
list_schedules(action_kind, report_id, owned_by_me) / get_schedule(schedule_id) |
S / U | Read |
create_schedule(request_id, schedule) |
U | Owner is the caller; grant_required if the owner lacks a grant for a required location |
update_schedule(schedule_id, expected_version, schedule) |
U | Includes pausing and resuming via enabled |
delete_schedule(schedule_id, expected_version) |
U | |
test_run(schedule) |
U | Run an unsaved schedule once and return its outputs, row count, condition value and the messages it would send. Delivers nothing |
run_now(schedule_id, request_id, ignore_condition) |
U | Manual run, also the way to retry |
list_runs(schedule_id, status, since) / get_run(run_id) |
S / U | Run history |
cancel_run(run_id) |
U |
Errors: the shared kinds in README.md, notably
grant_required (FAILED_PRECONDITION, one PreconditionFailure violation per
location without a usable grant) and conflict.
Runs
A run records run_id, trigger_kind (schedule, manual), scheduled_for,
started_at, finished_at, status (pending, running, succeeded,
skipped, failed, cancelled), skip_reason, the revision_id rendered,
the resolved parameter values, outputs (media_type, row_count), one result
per destination, and on failure error:
@dataclass
class RunError:
code: str # canonical vgi_rpc code
kind: str # grant_expired | grant_required | source_unavailable | query_failed
# | condition_invalid | result_too_large | policy | render_timeout | internal
location: str = "" # the data source involved, if any
message: str = "" # developer-facing
The protocol guarantees two things about runs, and leaves the rest to the implementation:
- Runs are unique on
(schedule_id, scheduled_for), so a firing happens once however many scheduler replicas there are. - A schedule has at most one active run; a firing during one is
skippedwith reasonoverlap.
Relative values
Tokens resolve in the trigger's time zone: today, yesterday,
last_n_days:N, previous_week (ISO weeks, Monday start), previous_month,
previous_quarter, month_to_date, year_to_date. Pinned with test vectors.
A date_range parameter resolves to {start, end}. Resolved values are stored
on the run and shown in deliveries ("Period: Sep 1–30, 2026").
SQL binding
Schema schedules. Derived by the rules in README.md; this is the annotation.
| Table | List / get | INSERT |
UPDATE |
DELETE |
|---|---|---|---|---|
schedules |
list_schedules / get_schedule |
create_schedule |
update_schedule, expected = the row's version |
delete_schedule, same |
runs |
list_runs / get_run |
list_runs with no schedule_id lists runs across every schedule the caller
can read. run_now, cancel_run, test_run and preview_trigger are
procedures.
UPDATE ops.schedules.schedules SET enabled = false WHERE title LIKE 'Weekly%';
SELECT schedule_id, scheduled_for, error.kind, error.message
FROM ops.schedules.runs WHERE status = 'failed' AND started_at > now() - INTERVAL 1 DAY;
Reference scheduler (non-normative)
How the reference scheduler is built. None of this is required of another implementation.
Driven by tick, so it runs long-running or serverless. The engine keeps no
clock. Two calls on durable state do all the work: tick(budget_seconds) claims
due schedules, due alert rules and runs with a step due; advance_run(run_id,
budget_seconds) moves one run through its persisted, idempotent steps
(condition, action, poll the render, store outputs, deliver to each
destination) until it finishes, waits, or runs out of budget. A long-running
vgi-report-serve calls tick on a loop. Serverless deployments are ticked by
a platform cron's HTTP request (a Cloudflare Cron Trigger, EventBridge, any
cron), and tick fans out one advance_run request per run to its own URL.
Both are methods of vgi.scheduler_driver.v1, an operator protocol callable
only by allowlisted principals (VGI_SCHEDULER_DRIVER_PRINCIPALS); tick takes
no time argument, so a caller can wake the scheduler but not move its clock.
Duplicate ticks are no-ops. Platform cron's one-minute floor is the scheduler's
minimum interval. Queries need native DuckDB with the VGI extension, so on
Cloudflare the Worker is only the clock.
Sessions are fresh per run, built from data_sources plus the owner's delegations, with
enable_external_access = false and lock_configuration = true after the
attaches, and memory, row and time limits. They are dropped with their grants
when the run ends.
Reliability. Per-schedule leases with fencing tokens on FunctionStorage
counters. Retryable failures (by is_retryable()) retry within the run,
honouring RetryInfo; five consecutive failed runs disable the schedule and
notify the owner. Sends record their intent before calling send, with
request_id = hash(run_id, delivery index), so a resumed run never delivers
twice. An expired or rejected grant fails the run with grant_expired.
Outputs are copied into the scheduler's store and kept for the operator's retention (90 days by default). Delivery links point at an authenticated run page; attachments use short-lived signed URLs the notify service can fetch.
State lives in FunctionStorage: sqlite for long-running and development,
Azure SQL or Cloudflare Durable Objects for serverless. The grant encryption key
comes from the platform's secret store.
Deferred
- Change detection ("only if results changed"), fan-out per parameter value, chained actions, business-day and fiscal relative tokens.
vgi.sql_tasks.v1
A SQL task runs SQL that changes data on a trigger, as its owner, in a
session rebuilt from its data sources and the owner's delegations. A task can
load a query's result into a table (replace it, append to it, merge into it,
or keep dated snapshots), fetch only what's new since the last run, or run a
script of statements in one transaction, CALLs included. Each firing is a run
that records what it changed and where the watermark moved. Shared conventions
are in README.md; sessions and delegations are
in credentials.md. A host of vgi.sql_tasks.v1 also hosts
vgi.delegations.v1.
Why not a schedule. A schedule produces outputs and delivers them; it has no destination table and no memory between runs. A task writes, keeps a watermark across runs, and is managed as the dataset it maintains: how fresh it is, what the last run loaded, reset and backfill. Either protocol can be hosted without the other; the reference service runs both on one engine.
Model
@dataclass
class SqlTask:
title: str
description: str = ""
trigger: Trigger # as vgi.schedules.v1: cron | once
data_sources: list[DataSource] # the session; credentials.md
body: TaskBody
incremental: Incremental = ... # default kind = none
condition_sql: str = "" # empty = always; see Conditions
parameter_values: list[ParamValue] = []
notify: TaskNotify = ...
stale_after_seconds: int = 0 # health is `stale` with no success this long; 0 = never
timeout_seconds: int = 3600
enabled: bool = True
@dataclass
class TaskBody:
kind: str # load | script | <vendor>.<name>
load: LoadBody = ... # kind = load
script: ScriptBody = ... # kind = script
custom_json: str = "" # kind = <vendor>.<name>
@dataclass
class LoadBody:
setup_sql: str = ""
query: str # the rows to load; may use $watermark
target: Target
mode: str # replace | append | merge | snapshot
key_columns: list[str] = [] # mode = merge: match on these
delete_missing: bool = False # mode = merge: delete target rows the query no longer returns
snapshot_column: str = "_snapshot_at" # mode = snapshot
retain_snapshots: int = 0 # mode = snapshot: keep the newest N; 0 = all
on_schema_change: str = "fail" # fail | add_columns
@dataclass
class ScriptBody:
statements: list[str] # run in order; may use $watermark and parameters
transaction: str = "single" # single | per_statement | none
watermark_sql: str = "" # incremental.kind = stored: returns the new watermark
@dataclass
class Target:
kind: str # catalog | host
alias: str = "" # kind = catalog: one of data_sources, written under the owner's delegation
schema: str = "main"
table: str
create_if_missing: bool = True # created from the query's result schema
@dataclass
class Incremental:
kind: str = "none" # none | derived | stored
cursor_column: str = "" # load: the column that only grows (updated_at, id)
initial_json: str = "" # the watermark for the first run; empty = NULL
lookback: str = "" # e.g. "PT2H": re-read late-arriving rows; needs mode = merge
@dataclass
class TaskNotify:
destinations: list[Destination] = [] # vgi.notify.v1
on: list[str] = ["failing", "recovered", "disabled"]
# failing | recovered | stale | disabled | every_failure | success
Records add task_id, owner, version, next_fire_at, last_run,
watermark_json, target_row_count where the target reports it,
disabled_reason, credentials (as schedules), allowed_actions, and the
task's health (below).
Bodies
load
Runs setup_sql, then writes query's result to the target in one
transaction:
| Mode | Effect |
|---|---|
replace |
The target holds exactly the result. Readers see the old rows or the new, never neither |
append |
Inserts the result |
merge |
Upserts on key_columns; with delete_missing, deletes target rows whose key the result no longer returns |
snapshot |
Appends the result with snapshot_column set to the run's scheduled_for, then deletes snapshots beyond retain_snapshots |
A missing target is created from the result's schema when create_if_missing.
A result whose schema no longer matches fails the run (schema_mismatch)
unless on_schema_change = add_columns, which adds new columns as nullable and
fills missing ones with NULL; a changed type always fails.
script
Runs statements in order, with parameters bound as $key:
transaction |
Meaning |
|---|---|
single |
One transaction around all statements: all commit or none do |
per_statement |
Each statement commits on its own; a failure stops the script and earlier statements stay committed |
none |
No transaction is opened; for statements that can't run inside one |
A CALL is a statement like any other. CALL sales.main.refresh_extract(...)
invokes a table function on the sales worker under the owner's delegation, which
is how a task asks a worker to do work that isn't SQL. Work outside any worker
is a vendor kind.
Vendor kinds
<vendor>.<name> carries its arguments in custom_json, validated at create
time, as in schedules. It declares data_sources so delegations are checked
the same way. A client that doesn't know a kind can still list, pause, run and
delete the task and read its runs.
Transactions and atomicity
One written database per transaction. DuckDB reads from any number of
attached databases in a transaction but writes to only one. A single script
or a load may read every source; its writes must all go to one database. A
task that writes two databases uses per_statement and accepts that a failure
can leave the first written and the second not. Implementations check this at
create time where they can and otherwise fail the run with
transaction_failed.
A remote target's commit is that worker's. Writing a catalog target
commits through the worker's VGI catalog transaction. Atomicity, isolation and
what readers see during the write are that catalog's guarantees.
Incremental loads
The watermark is the high-water mark of what has been loaded. It's bound as
$watermark in query, statements and condition_sql (NULL on the first
run unless initial_json is set).
kind |
Where the watermark comes from | Guarantee |
|---|---|---|
none |
No watermark | |
derived |
SELECT max(cursor_column) FROM target, read inside the run's transaction just before writing |
Exactly once. The read and the write commit together, and at most one run is active, so a crashed run loads nothing and the next run starts where the target ends |
stored |
Saved by the service after a successful run: max(cursor_column) of the loaded rows (load), or the single value watermark_sql returns, run last inside the transaction (script) |
At least once. The service saves the watermark after the target commits, so a crash between the two re-reads the last window. Make the write idempotent (merge) and that is harmless |
derived needs a load with a target that has cursor_column. stored works
for any body, including a script whose watermark isn't a column, a target the
runner can't query, and a source-side cursor such as an API page token. On a
host target an implementation may store the watermark in the same database as
the data, which makes stored exactly once too; get_tasks_info says whether
it does.
lookback subtracts an ISO 8601 duration from a timestamp watermark to
re-read rows that arrived late, and requires mode = merge so the overlap
doesn't duplicate. A watermark never moves backwards on its own: a run whose
new watermark is lower than the old one fails with watermark_invalid. Moving
it back is a backfill, done explicitly with set_watermark.
Destinations
catalog: one of the task'sdata_sources, written under the owner's delegation for it. The data lands on that worker (a DuckLake catalog, a database worker, anything with VGI write support), and that worker's access control governs both the write and every later read.host: a table in the service's own store, served by the service as a read-only catalog overvgi.v2. The catalog's name and location are inget_tasks_info. Who can read it is the service'sAccessPolicy; by default, the task's owner and whoever the owner shares the task with.
Conditions
condition_sql runs before the body in the task's session and returns one
BOOLEAN, with the results as in schedules (false/NULL skip the run). It
can read $watermark, so "only if anything is newer than what we have" is
SELECT max(updated_at) > $watermark FROM sales.main.orders.
Tracking health
Every task carries a health summary, kept current by the service as runs finish and as time passes:
@dataclass
class TaskHealth:
state: str # healthy | failing | stale | paused | disabled | new
since: str # when the task entered this state
last_success_at: str = ""
last_failure_at: str = ""
last_error: RunError = ... # the most recent failure's error
consecutive_failures: int = 0
| State | Meaning |
|---|---|
new |
Never run |
healthy |
The last finished run succeeded or was skipped by its condition, and the task isn't stale |
failing |
The last finished run failed; consecutive_failures counts the streak |
stale |
No success within stale_after_seconds, even though nothing failed (runs skipped, cancelled, or the trigger stopped firing) |
paused |
enabled = false, set by a person |
disabled |
Disabled by the service, with disabled_reason (for example, repeated failures or an expired delegation) |
Notifications follow state changes, not runs. failing notifies once when
a streak starts, recovered once when it ends, stale and disabled once on
entry. every_failure and success notify per run, for those who want it.
Each message carries the task, the state, the run and its RunError.
History is queryable in SQL. The tasks and runs tables (see SQL
binding) carry health and run history, so an alert rule can watch tasks too
("any task in the finance folder failing for over an hour").
Methods
U = unary, S = producer stream. All are required.
| Method | Kind | Purpose |
|---|---|---|
get_tasks_info() |
U | Body kinds, modes, host store catalog and location, whether host watermarks are atomic, minimum interval, limits |
preview_trigger(trigger, count) |
U | As in schedules |
list_tasks(owned_by_me, target, health_state) / get_task(task_id) |
S / U | Read, with health; health_state filters ("everything failing") |
create_task(request_id, task) |
U | Owner is the caller; grant_required for any source without a usable delegation |
update_task(task_id, expected_version, task) |
U | Includes pausing via enabled. Changing the target, mode or cursor column clears a stored watermark |
delete_task(task_id, expected_version, drop_target) |
U | drop_target drops a host table; a catalog target is never dropped |
test_run(task) |
U | Runs an unsaved task inside a transaction that is always rolled back. Returns row counts per statement, the old and new watermark, the target schema it would create and the condition value. A CALL whose function acts outside DuckDB still has that effect |
run_now(task_id, request_id, ignore_condition, full_refresh) |
U | Manual run, also the way to retry. full_refresh runs with $watermark = NULL and, for load, replace semantics |
set_watermark(task_id, expected_version, watermark_json) |
U | Backfill or reset a stored watermark; empty clears it. Refused for derived, whose watermark is the target's data |
list_runs(task_id, status, since) / get_run(run_id) |
S / U | Run history. task_id empty lists runs across every task the caller can read, so "what failed overnight" is one call |
cancel_run(run_id) |
U | Rolls back an open transaction; per_statement keeps what already committed |
Errors: the shared kinds in README.md.
Runs
As in schedules (unique on (task_id, scheduled_for), at most one active run,
an overlapping firing is skipped with reason overlap), recording in
addition: watermark_before, watermark_after, per statement its index, kind
and rows_affected, and for load the rows inserted, updated and deleted and
the target's row count afterwards, and attempt when the implementation
retried within the run. RunError.kind adds transaction_failed,
schema_mismatch, watermark_invalid and target_unwritable to the schedule
kinds.
SQL binding
Schema sql_tasks. Derived by the rules in README.md; this is the annotation.
| Table | List / get | INSERT |
UPDATE |
DELETE |
|---|---|---|---|---|
tasks |
list_tasks / get_task |
create_task |
update_task, expected = the row's version |
delete_task, same (drop_target false; use CALL to drop) |
runs |
list_runs / get_run |
run_now, cancel_run, test_run, set_watermark and preview_trigger are
procedures. A task's own script can call them, and any other protocol's
procedures (CALL ops.notify.send(…)).
SELECT title, health.state, health.consecutive_failures, health.last_error.message
FROM ops.sql_tasks.tasks WHERE health.state IN ('failing', 'stale');
CALL ops.sql_tasks.run_now(task_id := 't_orders', request_id := 'manual-2026-10-08',
ignore_condition := false, full_refresh := true);
Reference implementation (non-normative)
- Engine. The same
tick/advance_runengine as the reference scheduler, so tasks run long-running or serverless alike. A task's step list is condition, body, save watermark, notify. - Sessions are built like schedule sessions (fresh, hardened, limited). The
hoststore is attached writable before configuration is locked; every other source is attached with the owner's delegation. - The
hoststore is a DuckLake catalog by default, soreplaceandsnapshotkeep time travel for free, and the stored watermark lives in the same catalog, which makeshosttargets exactly once. - Bodies compile to SQL.
mergeisMERGE INTOwhere the target supports it, or a delete-and-insert in one transaction where it doesn't. Runs record the compiled statements.
Deferred
- Dependencies between tasks ("after the orders load succeeds"), partitioned backfills, change data capture from source logs, and per-row quarantine of rows that fail a check.
vgi.alerts.v1
An alert rule is a query that returns the rows meeting a condition. Each row is
an alert instance, identified by the rule's key columns, and its other
columns are the instance's details. Instances move through states, people are
notified on transitions rather than on every evaluation, and anyone with access
can acknowledge, snooze or subscribe. Shared conventions are in
README.md; sessions and grants are in
credentials.md. A host of vgi.alerts.v1 also hosts
vgi.delegations.v1: rules carry no credentials and are evaluated by
reattaching each source with the owner's delegation for it.
Why not a schedule with a condition. A schedule delivers every time its condition holds and has no memory between runs. An alert needs state (notify when West starts falling short and when it recovers, not every five minutes in between), one instance per entity, the triggering values in the message, and people acting on it. Those are this protocol. Either protocol can be hosted without the other; the reference service runs both on one engine.
Rules
@dataclass
class AlertRule:
title: str
description: str = ""
severity: str = "warning" # info | warning | critical
data_sources: list[DataSource] # the session to evaluate in (credentials.md)
setup_sql: str = ""
sql: str # returns the rows meeting the condition
key_columns: list[str] = [] # identify an instance; empty = one instance at most
parameters: list[ParameterSpec] = []
parameter_values: list[ParamValue] = []
evaluate_every_seconds: int = 300 # at least the evaluator's minimum
pending_for_seconds: int = 0 # must keep matching this long before firing
resolve_after_seconds: int = 0 # must stay absent this long before resolving
repeat_every_seconds: int = 0 # re-notify while firing and unacknowledged; 0 = never
max_instances: int = 100
messages: AlertMessages = ...
owner_destinations: list[Destination] = []
share_details_with_subscribers: bool = False
enabled: bool = True
@dataclass
class AlertMessages:
title_template: str = "{{rule.title}}: {{instance.key}}"
summary_template: str = "" # e.g. "{{region}} is {{shortfall | percent}} below target"
resolved_template: str = "" # default: "{{instance.key}} recovered"
field_columns: list[str] = [] # detail columns shown as headline fields
The query is the condition. "Any region below target today" is:
SELECT region, revenue, target, revenue / target - 1 AS shortfall
FROM sales.main.daily_revenue
WHERE day = current_date AND revenue < target
with key_columns = ["region"]. A row for West means West matches; no row for
West means West doesn't. A yes/no alert is a rule with no key columns whose
query returns one row or none, which is the schedule condition's case made
stateful.
Evaluation runs setup_sql and sql in a session rebuilt from
data_sources and the owner's delegations, wherever the implementation runs
it. Parameters bind as in schedules, relative values included, so
last_n_days:1 works in a rule.
Result rules. Key columns must exist and be non-null, and keys must be
unique within one result; a violation is an evaluation error. More than
max_instances rows is an evaluation error (too_many_instances), never a
silent truncation. Detail values are kept as their Arrow types and rendered by
the templates.
Instances and states
An instance's id is a hash of (rule_id, canonical key), where the canonical
key is the RFC 8785 JSON of the key columns in key_columns order. The same
region therefore maps to the same instance across evaluations and restarts.
stateDiagram-v2
[*] --> pending: row appears
pending --> firing: still present after pending_for_seconds
pending --> [*]: row gone before then
firing --> resolving: row gone
resolving --> firing: row back
resolving --> resolved: still gone after resolve_after_seconds
resolved --> pending: row appears again
- With
pending_for_seconds = 0an instance fires on first sight; withresolve_after_seconds = 0it resolves on the first evaluation without it. - Each instance keeps
first_seen_at,fired_at,resolved_at,last_seen_at, the latest detail values, and the detail values at the moment it fired, so "fired at $41k, now $44k" can be shown. - Evaluation errors never resolve anything. If the query fails, a grant is
rejected or the result breaks the rules above, every instance keeps its state,
the rule enters
errorwith aRunError(same kinds and codes as schedule runs), and the owner is notified once on enteringerrorand once on leaving it. Treating a failed query as "no rows" would resolve every alert exactly when the data is least trustworthy. - A resolved instance is kept for history; the reporting catalog shows it.
Notifications
Notifications go out on transitions, never on every evaluation:
| Event | Sent when | Message |
|---|---|---|
firing |
pending → firing, or resolved → … → firing |
title_template / summary_template, with the detail values |
resolved |
resolving → resolved |
resolved_template |
repeat |
Still firing, unacknowledged, not snoozed, every repeat_every_seconds |
As firing, marked as a reminder |
rule_error / rule_recovered |
The rule enters or leaves error |
To the owner only |
- Grouped. Transitions from one evaluation go out as one notification per rule ("3 regions below target: West −18%, East −6%, South −2%").
- Threading.
thread_keyis the instance id, so a Slack thread or email conversation carries one instance's firing, reminders and resolution. - Severity maps to the notify
severity. Fields are thefield_columnsof the instance. Link opens the instance in Cupola.
Templates are logic-less, with {{column}}, {{rule.title}},
{{instance.key}}, {{instance.fired_at}} and a fixed set of formatters:
number, percent, currency:<ISO 4217>, date, datetime, duration,
pinned with test vectors. Channel formatting and escaping are the notify
service's. create_rule and test_rule reject a template that names a column
the query doesn't return.
Who sees the details
A rule is evaluated as its owner, so its rows are the owner's data. The protocol holds no grants for anyone else and can't check what they may see.
- Owner destinations (
owner_destinations, and the owner's own subscriptions) get the full message, with details. - Other subscribers get the rule title, the instance key, state, severity, times and a link. Opening the link shows details only to someone who may see them: the owner, or anyone when the rule shares details. Cupola also offers "view with my access", which re-runs the rule's query for that one key with the viewer's own session.
- Sharing details is the owner's explicit choice.
share_details_with_subscribers = truesends details to every subscriber. The rule records who turned it on and when. Notify's destination policy still applies. - Instance keys appear in every notification, so a rule whose key itself is sensitive should not have subscribers.
Acknowledge, snooze, subscribe
Every action records actor, time and note, and appears in the instance or rule timeline.
- Acknowledge an instance ("I'm on West"): stops
repeatreminders until it resolves, and clears on resolve.set_acknowledged(…, false)reverses it. - Snooze a rule or an instance until a time, or until the instance next
resolves. Evaluation and state changes continue; only notifications are held.
A snooze must end: the evaluator publishes a maximum length, and silencing for
good means disabling the rule. When a snooze ends on a still-firing instance,
one
firingnotification goes out marked as after-snooze. - Subscribe the caller to a rule with destinations they choose, subject to notify's policy. Subscribers manage only their own subscriptions; the owner can remove anyone's.
Methods
U = unary, S = producer stream. All are required.
| Method | Kind | Purpose |
|---|---|---|
get_alerts_info() |
U | Minimum evaluation interval, max_instances ceiling, maximum snooze, limits |
list_rules(query, owned_by_me, state) / get_rule(rule_id) |
S / U | Rules with their state (ok, firing, error, disabled) and instance counts |
create_rule(request_id, rule) |
U | Owner is the caller; grant_required if the owner lacks a grant for a required location |
update_rule(rule_id, expected_version, rule) |
U | Includes pausing via enabled. Changing the key columns starts new instances; the old ones close as superseded |
delete_rule(rule_id, expected_version) |
U | Instances close as superseded |
test_rule(rule) |
U | Evaluate an unsaved rule once: the instances it would create, rendered messages, template errors. Sends nothing |
list_instances(rule_id, state, since) / get_instance(instance_id) |
S / U | Instances with state and times; detail values only for callers who may see them |
set_acknowledged(instance_id, acknowledged, note) |
U | |
snooze(rule_id, instance_id, until, until_resolved, reason) / unsnooze(snooze_id) |
U | Empty instance_id snoozes the rule |
subscribe(rule_id, destinations) / unsubscribe(subscription_id) / list_subscriptions(rule_id) |
U / U / S | |
list_events(rule_id, instance_id, since) |
S | Timeline: transitions, notifications, acknowledgements, snoozes, errors |
Permission hints. Rules and instances carry allowed_actions from the
shared set plus acknowledge, snooze, subscribe, view_details.
Errors use the shared kinds in README.md, plus:
invalid_request covers a bad rule (a template column the query doesn't
return, a missing key column, an interval below the minimum) and a snooze past
the maximum, with a BadRequest naming the field.
SQL binding
Schema alerts. Derived by the rules in README.md; this is the annotation.
| Table | List / get | INSERT |
UPDATE |
DELETE |
|---|---|---|---|---|
rules |
list_rules / get_rule |
create_rule |
update_rule, expected = the row's version |
delete_rule, same |
instances |
list_instances / get_instance |
|||
events |
list_events |
list_instances and list_events with no rule_id cover every rule the
caller can read. list_subscriptions has a required rule_id, so it stays a
table function. set_acknowledged, snooze, unsnooze, subscribe,
unsubscribe and test_rule are procedures. Detail columns are NULL where
the caller may not see them, as over RPC.
SELECT rule_id, instance_id, state FROM ops.alerts.instances
WHERE state = 'firing';
CALL ops.alerts.set_acknowledged(instance_id := 'i_west', acknowledged := true,
note := 'Investigating');
Conformance
The state machine is pinned with vectors. Each vector is a sequence of
(time, rows returned or error) evaluations, plus acknowledgements and
snoozes, with the expected transitions and notifications. The fake clock and
recording notifier in vgi.testing run them, so every implementation agrees on
when West fires, reminds and resolves.
Reference evaluator (non-normative)
- Evaluation is driven by the scheduler's
tick, so alerts run long-running or serverless alike (schedules.md). A rule is claimed with the scheduler's per-rule lease, and a missed interval is evaluated once, not replayed. - A fresh DuckDB session and VGI attach cost about 270 ms before the query runs, so a long-running evaluator keeps a warm session per rule, rebuilt when the rule or the owner's grants change. Serverless evaluators can't, and their published minimum interval reflects it (60 s at best, the cron floor).
- Large groups are summarized after 10 instances ("and 14 more") with a link.
Deferred
- Escalation and on-call (escalate if unacknowledged for 30 minutes): route to PagerDuty or Opsgenie through a signed notify webhook for now.
- Dependencies and inhibition (don't page for each region when the warehouse is down).
- Anomaly and change rules (more than 3σ from the 28-day mean); expressible today in the rule's SQL, without a built-in.
- Event-driven evaluation when a source refreshes, instead of polling.
vgi.notify.v1
One channel-agnostic protocol for delivering a message to email, Slack, Microsoft Teams, Discord or a webhook. The caller says what to say in a portable content model; the service decides how each channel shows it, and owns which destinations are allowed. Shared conventions are in README.md.
Destinations
Destinations are {kind, address} pairs. kind is an open set; v1 names these:
| Kind | address |
|---|---|
email |
ana@example.com |
slack |
Channel or user id, e.g. C024BE91L |
teams |
Name of a registered channel |
discord |
Name of a registered channel |
webhook |
Name of a registered endpoint, e.g. ops-pagerduty; never a raw URL |
Everything URL-shaped is a registered destination the operator configured, so a schedule can't be pointed at an attacker's URL.
Message
@dataclass
class Notification:
title: str # subject line / headline
summary: str # plain text, always delivered
body_markdown: str = "" # CommonMark subset; converted per channel
link_url: str = "" # primary action, e.g. the run or alert page
link_label: str = ""
fields: list[Field] = [] # headline values: {label, value}
image: Attachment = ... # inline preview; empty url = none
attachments: list[Attachment] = [] # {filename, media_type, url, size_bytes, sha256}
severity: str = "info" # info | warning | critical
thread_key: str = "" # groups follow-ups (Slack thread, email In-Reply-To)
on_behalf_of: PrincipalRef = ...
correlation: dict[str, str] = {} # schedule_id, run_id, rule_id, ...
Each service maps the model to the channel's native form, and parts a channel can't carry degrade to text, so callers never branch on channel.
Attachment URLs must be fetchable by the notify service: the caller issues
short-lived signed URLs for them. Links (link_url) are different: they point
at pages that check the reader's own access.
Methods
U = unary, S = producer stream. All are required.
| Method | Kind | Purpose |
|---|---|---|
get_notify_info() |
U | Per kind: enabled, limits (text length, attachment bytes), images, threads; destination policy summary |
list_destinations(kind, query) |
S | Registered and discoverable destinations with display_name; also address autocomplete |
check_destinations(destinations) |
U | Per destination: allowed, display_name, or why it is refused |
send(request_id, destinations, notification) |
U | notification_id and one result per destination: accepted, refused (with reason, e.g. policy, unsubscribed) or failed |
send delivers to the destinations it can and reports the rest. A caller that
wants all-or-nothing calls check_destinations first. A caller not on the
sender allowlist gets action_denied; a provider outage is
service_unavailable with RetryInfo.
SQL binding
Schema notify. Derived by the rules in README.md; this is the annotation.
| Table | List / get | INSERT |
UPDATE |
DELETE |
|---|---|---|---|---|
destinations |
list_destinations |
send and check_destinations are procedures, so a SQL task can notify as a
step of its script.
Policy
- Who may send. An allowlist of principals permitted to call
send(--notify-senders/VGI_NOTIFY_SENDERS), no permissive default: the scheduler's and alert evaluator's service principals, alerting workers. Users configure deliveries; those services send them and recordon_behalf_of. These principals send messages; they never read data. - Destination policy is the service's own (email domains, one Slack
workspace, registered webhooks only), summarized in
get_notify_infoand enforced onsend. - Unsubscribe is the notify service's. It keeps a suppression list keyed by
address and
correlation(e.g.schedule_id), so an unsubscribe link works however the sender is deployed. Suppressed destinations returnrefused: unsubscribed.
Reference notify service (non-normative)
The reference host is a new vgi-notify repo hosting every channel, with a
stdout channel for development.
- Email (SMTP, SES, Resend) renders HTML and adds one-click
List-Unsubscribe(RFC 8058) through a signed link. - Slack Block Kit, Teams Adaptive Cards and Discord embeds, through credentials held by the service.
- Webhooks send a fixed JSON envelope (
type,title,summary,fields,link_url,correlation,sent_at) signed with the Standard Webhooks headers. - Later:
vgi-email's Gmail and Microsoft Graph drivers as an email channel (Graph needs the app-only auth inENHANCEMENT-microsoft-app-only.md).
Plan: attach tickets and vgi_export_session()
Status: plan for review · 2026-10-06
Goal: serialize a user's attached DuckDB session so any runner can reattach it
later as that user, without seeing the user's secrets. Design context is in
credentials.md. This plan covers the worker side (every SDK)
and the extension. vgi-rpc needs no change: tickets are a VGI artifact, sealed
the way VGI already seals attach_opaque_data.
Decisions to confirm
| # | Decision | Recommendation | Why |
|---|---|---|---|
| T1 | A vgi.v2 method, or a separate protocol? |
Separate: vgi.attach_tickets.v1 |
vgi.v2 is version-gated on exact major+minor. A new vgi.v2 method means 2.2.0, and every SDK plus the extension must release in lockstep, as catalog_contents did. A separate protocol is optional per worker (D1), discovered by reflection, and needs no vgi.v2 change. |
| T2 | How the ticket reaches catalog_attach |
A reserved ATTACH option, vgi_attach_ticket, opened by the SDK framework before the catalog's own catalog_attach runs |
CatalogAttachRequest.options is already an open record, so the vgi.v2 wire is unchanged. The extension sends it only to workers whose reflection lists vgi.attach_tickets.v1, so an old worker never silently ignores it. |
| T3 | Where sealing lives | Each SDK's existing attach-envelope sealer (Python _seal_attach, Java OpaqueDataSealer, C++ seal_attachment, …), keyed by VGI_SIGNING_KEY, with a ticket-specific AAD |
Every SDK already seals attach_opaque_data with an AEAD bound to the caller, using vgi-rpc's crypto primitive where the port exposes one. Tickets are the same kind of VGI artifact. No vgi-rpc change, no new key, no new cipher. |
| T4 | What gets sealed | The options the extension attached with, sent by the extension; validated against the catalog's declared AttachOptionSpecs, not by a trial attach |
The extension holds the original values, secrets included. Stateless workers don't keep them. Validation without attaching has no side effects. |
| T5 | Ticket and grant separate | Yes | See credentials.md: a ticket carries no authority, so it needs no fresh login, never enters AuthContext, and adds no authentication path. |
Token format
ticket = "vgia1." base64url_nopad( envelope )
envelope = the SDK's attach envelope: version || nonce || ciphertext || tag, under VGI_SIGNING_KEY
aad = "vgi.attach_ticket.v1" 0x00 || UTF-8(principal)
Same sealer, different AAD. attach_opaque_data binds
(auth domain, principal). A ticket must bind the principal only, because
it's sealed while the user is logged in (domain jwt, say) and opened when a
runner presents their grant (domain grant). The principal is the same in both:
a grant is minted for its caller's principal. Binding the principal in the AAD
also makes the principal check free: a ticket opened under any other principal
fails authentication, so there is no separate comparison to forget.
The prefix keeps tickets distinguishable from other strings and versioned. The distinct AAD domain means an attach envelope can never be opened as a ticket, or the reverse.
Payload (integers little-endian; strings are u16 length plus UTF-8):
| Field | Encoding |
|---|---|
issued_at, expires_at |
int64 seconds (expires_at = 0 means no expiry) |
ticket_id |
string (32 hex) |
catalog_name |
string |
data_version_spec, implementation_version |
string ("" = none) |
options |
u32 length plus Arrow IPC bytes of the options record, exactly as in CatalogAttachRequest.options |
Capped at 16 KiB of options. Verification: prefix, canonical base64url, open
under the caller's principal (failure is attach_ticket_invalid), strict
parse, then lifetime with 60 s skew.
Key and lifetime.
- Tickets live as long as
VGI_SIGNING_KEYdoes. Rotating it invalidates every ticket, as it already invalidates attach envelopes. - The protocol is hosted only when the signing key is configured explicitly. A per-process generated key would make tickets die on restart.
- The maximum lifetime follows the worker's grant maximum, so the two halves of a delegation expire together.
Principal namespaces must be consistent per worker. A ticket sealed for
alice opens for any caller whose principal is alice, whatever domain
authenticated them. That's already true of grants, and a deployment whose
authenticators map the same string to different people has that problem today.
The spec says so.
vgi.attach_tickets.v1
One method. Hosted on HTTP when VGI_SIGNING_KEY is set explicitly and the
worker can issue grants (grant keys, or its own mint_grant), because a ticket
is useless without a grant.
@dataclass
class SealAttachRequest:
catalog_name: str
options: bytes # Arrow IPC record, as CatalogAttachRequest.options
data_version_spec: str = ""
implementation_version: str = ""
ttl_seconds: int = 0 # 0 = as long as the worker allows
@dataclass
class AttachTicket:
ticket: str
expires_at: float # +inf when the worker sets no maximum
| Method | Rules |
|---|---|
seal_attach(request) -> AttachTicket |
Caller must be authenticated (anonymous is action_denied). The principal sealed is the caller's. Options are validated against the catalog's declared options (invalid_request with BadRequest). No freshness requirement. A worker may refuse by policy (action_denied). |
Reattach: what every SDK's catalog_attach does
When options contains vgi_attach_ticket, the framework, before any catalog
code:
- Rejects any other catalog option alongside it (
invalid_request). The sealed options are authoritative, so there is nothing to merge. - Verifies the ticket: an invalid one is
invalid_request(attach_ticket_invalid); an expired one isFAILED_PRECONDITION(attach_ticket_expired). - The principal check is the open itself: the AAD carries the caller's
principal, so a ticket presented by anyone else fails as
attach_ticket_invalid. A stolen ticket is useless without that user's grant. - Calls the catalog's
catalog_attachwith the sealedcatalog_name, options and version specs, as if the user had typed them. Catalog code needs no change.
vgi_attach_ticket becomes a reserved option name: declaring an AttachOption
with it fails at startup. Tickets and restored secret options are never logged,
and loggable_attach_options sees the restored options as usual.
Extension
vgi_export_session(aliases := NULL, ttl_seconds := NULL), a table
function. For each attached catalog (or each alias given):
- Not
TYPE vgi: emit the row withstatus = 'not_vgi'. - Reflection on the worker: if
vgi.attach_tickets.v1, orissue_grantonvgi_rpc.Identity.v1, is missing, emitnot_supported. - Call
issue_grant(purpose = "vgi.unattended", scopes = [], ttl_seconds)using the catalog's existing auth. Astale_authrefusal isstale_login, so Cupola can step up and retry. - Call
seal_attachwith the catalog name, the original option values (secrets included) and the version specs. - Emit
alias, location, catalog_name, grant, ticket, expires_at, status, message.
Per-row failures never abort the export.
attach_ticket ATTACH option.
- It's a secret option, like
bearer_token: redacted fromduckdb_databases(), never logged, and folded into the cache identity only as a salted hash. - Before
catalog_attach, the extension checks reflection forvgi.attach_tickets.v1and fails clearly if it's absent. - It sends the ticket as the
vgi_attach_ticketoption, and nothing else from the ATTACH options.
Prerequisites in the extension:
- Keep the original attach option values per catalog, secrets included,
in memory only. They're needed for
seal_attach. Verify whether today's reconnect path already does; if not, add it. - Call non-
vgi.v2protocols through the existing RPC client, using the per-protocol routing key, forIdentity.v1andvgi.attach_tickets.v1. - Work in the wasm build. Cupola is where exports happen, so the HTTP path must work under Emscripten.
Work and order
| Step | Repo | Work |
|---|---|---|
| 0 | vgi (extension), vgi-python | Reflection in the clients. Every worker hosts vgi_rpc.Reflection.v1, but no VGI client calls it. Extension: a vgi_protocols(...) table function (by location like vgi_catalogs(), or by attached catalog alias) returning protocol_name, protocol_version, protocol_hash, plus a cached per-catalog "hosts protocol X?" check for the extension's own features. vgi-python: Client.list_protocols(). The TypeScript client already has it (vgi-rpc-typescript introspect). Cross-SDK sqllogictests in test/sql/integration/reflection/. |
| 1 | vgi-python (reference SDK) | vgi.attach_tickets.v1, ticket payload, the catalog_attach interception, reserved option name, attach_ticket_vectors.json, fixture worker with a secret attach option. Spec in vgi/docs. |
| 2 | vgi (extension) | vgi_export_session(), the attach_ticket option, retained options, the Identity and tickets clients, wasm. sqllogictests (below). In parallel with step 1, against the Python fixture. |
| 3 | 6 other VGI SDKs | Port step 1 on each SDK's existing attach sealer and pass the same sqllogictests. No vgi-rpc releases are needed first. |
| 4 | Cupola | Export on save and on login renewal; put_delegations to the scheduler and alert hosts. Not part of this plan's release. |
No vgi.v2 version bump and no vgi-rpc release, so the SDKs and the extension release independently.
Tests
Cross-SDK sqllogictests, in ~/Development/vgi/test/sql/integration/attach_ticket/,
over HTTP, against every SDK's fixture worker:
- Export, detach, then reattach with
bearer_tokenplusattach_ticket. Query results match, and a secret option took effect without being in the reattach. - A ticket presented with another principal's grant is refused.
- An expired ticket, a tampered ticket, and a ticket from another worker's key are refused.
- Extra options alongside
attach_ticketare refused. - A worker without the protocol gives a clear error and
status = 'not_supported'. duckdb_databases()shows the ticket redacted.vgi_export_session()on a mix of VGI and:memory:catalogs reports a status per row.
Unit tests: in every port and SDK, attach_ticket_vectors.json with exact
mint, accept and reject cases: fixed key, nonce and clock, including a
principal mismatch and an attach envelope presented as a ticket.
Mutation checks: drop the principal from the AAD (or use the
(domain, principal) attach AAD), skip expiry, drop the "no other options"
rule, and skip the reflection check before sending. Each must turn
a test red.
Risks
- Ticket size. Options travel in the
catalog_attachbody, not a header, so a 16 KiB cap is generous. A runner storing many of them is a storage question, not a wire one. - Upstream secret rotation. A ticket keeps the old value. The catalog's own error surfaces on reattach, and the owner re-exports. Nothing detects this early.
- Workers without grant keys can't issue either half, so their catalogs can't run unattended. That's the intended opt-in.
- The wasm HTTP path is the most likely place for surprises. Prototype
vgi_export_session()in Cupola first.
Prerequisites: multi-protocol hosting across vgi-rpc and the VGI SDKs
Status: done · planned 2026-10-05, released 2026-10-06 (see Outcome)
Before any reporting protocol can ship, every vgi-rpc port and every VGI SDK
must let a worker host additional protocols, classify transient errors with a
retry hint, and host vgi_rpc.Identity.v1. This plan covers 1 spec, 7 vgi-rpc
ports and 7 VGI SDKs. Rowfence is deferred.
Decisions
| # | Decision | Consequence |
|---|---|---|
| D1 | The protocol is the unit of optionality. No method subsets and no feature tokens for application protocols. A capability that is optional becomes its own protocol. | No narrowing API. Reflection features stays emitted-empty; the spec says so explicitly. Identity keeps its existing hook-based narrowing, untouched. |
| D2 | A worker supplies a list of (protocol, implementation) pairs when its server is built. The list may be computed from configuration or environment, and is fixed for the life of the process. |
Reflection output and protocol hashes stay stable per process. The same list is hosted on every transport. |
| D3 | Identity stays framework-owned, and lands in all seven SDKs now. It keeps the reserved vgi_rpc. name and its guards, is opted into through resolve_token / mint_grant hooks rather than the worker's list, and is hosted on transports that authenticate callers (HTTP). |
Every SDK gets the same opt-in and the same refuse-to-start rule when introspection is enabled without an allowlist. |
| D4 | Errors take gRPC's shape. A closed set of canonical codes (vgi_rpc.error_code), the existing open error_kind as the fine-grained reason, and a list of typed details from a fixed catalog (vgi_rpc.error_details). Rules below. |
Clients get a uniform retry and handling policy from the code alone. Transient errors carry RetryInfo. Later protocols reuse the catalog (e.g. a revision conflict as ABORTED + ResourceInfo). |
| D5 | Rowfence routing is out of scope. | Rowfence still forwards only Identity.v1/issue_grant. |
| D6 | The cross-SDK checks live in vgi-rpc-python's conformance suite. Every SDK fixture worker hosts the same conformance.Secondary.v1 the ports use. |
One fixture protocol and one test group serve both the ports and the SDKs. |
Error model (adopted from gRPC)
gRPC's google.rpc.Status model (canonical codes, a reason, and typed
details from error_details.proto, with
AIP-193 as the usage guide) has held up across
many languages for years. We adopt its shape, and fix the parts that caused
trouble in practice.
Three layers on every EXCEPTION batch:
| Layer | Wire | Set | Purpose |
|---|---|---|---|
| Code | vgi_rpc.error_code, the code's name, e.g. UNAVAILABLE |
Closed: gRPC's 16 codes minus OK |
Generic handling: retry or not, how to show it, and later the HTTP status a proxy maps it to |
| Reason | vgi_rpc.error_kind, unchanged, e.g. identity_unavailable |
Open; unique within the protocol that raised it | What a client branches on. The pair (protocol, kind) is gRPC's (domain, reason) |
| Details | vgi_rpc.error_details, a JSON array of typed objects |
Fixed catalog below | Machine-readable specifics: retry delay, which field, which resource |
error_message stays developer-facing English, as in gRPC.
Detail catalog. Each object names its type in @type, mirroring
protobuf's JSON form for Any. Field names follow gRPC's, in snake_case:
@type |
Fields | Use |
|---|---|---|
vgi_rpc.ErrorInfo |
metadata: {string: string} |
Extra context for the reason. Reason and domain are already error_kind and the protocol, so they aren't repeated |
vgi_rpc.RetryInfo |
retry_delay_seconds: number |
How long to wait before retrying |
vgi_rpc.BadRequest |
field_violations: [{field, description}] |
Which inputs were wrong |
vgi_rpc.PreconditionFailure |
violations: [{type, subject, description}] |
What state must change first |
vgi_rpc.QuotaFailure |
violations: [{subject, description}] |
Which limit was hit |
vgi_rpc.ResourceInfo |
resource_type, resource_name, owner, description |
Which object the error concerns |
vgi_rpc.Help |
links: [{description, url}] |
Where to read more |
vgi_rpc.LocalizedMessage |
locale, message |
Text safe to show an end user |
DebugInfo is deliberately absent (see Tracebacks below).
Rules:
- The code is required whenever
error_kindis set. Each kind maps to exactly one code, fixed where the kind is defined. Errors with no kind getUNKNOWN; framework faults getINTERNAL. - Details come only from the catalog. A protocol may define its own detail
type only under its own name (
vgi.reports.v1.SomeDetail), and should preferErrorInfo.metadata. This avoids the gRPC failure of every service inventing its own detail messages. - Each type appears at most once in a details list, as AIP-193 requires.
- Clients ignore detail types they don't know, and never require details.
- Bounded. The serialized details array is at most 4 KiB. A server that would exceed it drops the whole array, not part of it.
- No secrets. Details MUST NOT carry credentials, tokens or user data.
- Retryability follows the code.
UNAVAILABLE, andRESOURCE_EXHAUSTEDwhen it carriesRetryInfo, are retryable; whenRetryInfois present a retry waits at least that long.ABORTEDmeans retry the whole operation at a higher level. Everything else is final. Clients expose anis_retryablecheck but do not retry RPC errors automatically: as in gRPC, automatic retry is opt-in, because a method may not be idempotent.
What we fix from gRPC's experience:
- Details invisible to clients and logs. gRPC put details in a
binary trailer (
grpc-status-details-bin), so many clients, proxies and logs never decoded them. Here the code, kind and details are plain top-level metadata, written into the access log, and conformance-tested in every client. - Truncation. gRPC proxies silently drop oversized trailers. Our 4 KiB cap drops the whole array explicitly.
UNKNOWNoveruse. gRPC services returnUNKNOWNfor everything they didn't classify. Every kind we define names its code, and conformance pins the mapping.- Stack traces. gRPC keeps
DebugInfoout of production responses. We chose the opposite default deliberately: tracebacks are included by default on every transport, behind one operator switch, because the DuckDB extension surfaces the remote traceback in the error a user sees, and hiding it on HTTP hid the cause of real failures. A language with no native stack synthesizes one, solog_extra.tracebackis never empty while the switch is on.
Codes for the existing kinds:
error_kind |
Code | Details |
|---|---|---|
method_not_implemented |
UNIMPLEMENTED |
|
protocol_not_supported |
UNIMPLEMENTED |
|
protocol_not_specified |
INVALID_ARGUMENT |
|
protocol_version_mismatch |
FAILED_PRECONDITION |
PreconditionFailure |
session_lost |
ABORTED |
|
server_draining |
UNAVAILABLE |
RetryInfo |
identity_unavailable |
UNAVAILABLE |
RetryInfo (required) |
stale_auth |
UNAUTHENTICATED |
|
introspection_refused |
PERMISSION_DENIED |
|
grant_refused |
PERMISSION_DENIED |
|
token_unresolved |
NOT_FOUND |
What the audit found
Spec rules ports already break. These need no spec change, only fixes:
- §3.1 says a server MUST refuse an application protocol named
vgi_rpc.*. Rust never checks the application name, C++ doesn't check at all, and C# checks only names declared with[ProtocolName]. - §8 says the client MUST surface
error_kind. The Python, TypeScript and Rust clients drop it. - Root cause:
CLIENT_DRIVER_PROTOCOL.md's error object has noerror_kindfield, so client conformance never checked it.
The spec doesn't say these. All seven ports get them wrong, which is a sign the spec is the cause:
retry_afterexists onIdentityUnavailableErrorin every port but reaches the wire in none.- When an identity hook raises the port's "auth unavailable" error, only
TypeScript and Rust translate it to
identity_unavailable. Python, Go, C#, Java and C++ send it unclassified. - vgi-python's own docs tell workers to raise that error type
(
vgi/worker.py:1460,CLAUDE.md). - Nothing says how an application registers several protocols. Only Python
(
extra_protocols) and TypeScript (addProtocol) can. Go'sAddProtocoltakes an unexported type, and Rust, C#, Java and C++ host exactly primary + Reflection + Identity.
SDKs:
- No SDK lets a worker add a protocol.
- Only vgi-python hosts Identity, and only over HTTP.
- vgi-python and vgi-typescript construct their server in 5 places each.
- Go, Rust, C#, Java and C++ each have one builder.
Out of scope:
- vgi-rpc-swift contains only a Claude Code settings file.
- vgi-kotlin is retired.
Phase 0 — Spec (vgi-rpc repo)
docs/WIRE_PROTOCOL.md§3.1, "Hosting several application protocols." - A server MUST let an application register any number of application protocols at construction, each(name, version, implementation). - The registered set is fixed for the server's lifetime and is the same on every transport the server is offered on. - The reserved-prefix rule applies to every registered protocol, however its name was derived. -list_protocolslists application protocols in registration order, the primary first. The client driver'sdescribeop already relies on "first non-reserved protocol".- §8, "Error model."
- New keys
vgi_rpc.error_codeandvgi_rpc.error_details, EXCEPTION batches only, also mirrored inlog_extra. - Normative text for the three layers, the detail catalog, the rules and the retry policy above. - The error-kinds table gains code and details columns, as above. - The traceback switch, on by default on every transport. - §16, "Identity."
-
identity_unavailableMUST carryRetryInfo. - Aresolve_tokenormint_granthook that raises the port's transport-auth "unavailable" error MUST be emitted asidentity_unavailablewith that error's retry hint, never unclassified. That is the TypeScript and Rust behaviour today. - §14, "Reflection."
featuresis reserved and MUST be emitted as an empty list in this version; clients MUST ignore its contents. tools/cross-port/specs/CLIENT_DRIVER_PROTOCOL.md. Adderror_code(string,""when absent),error_kind(string,""when absent) anderror_details(array,[]when absent) to the structured error object.tools/cross-port/specs/IDENTITY_CONFORMANCE_FIXTURE.md. Add a reserved test token that makes the fixture's resolver raise the port's auth-unavailable error with a retry hint of 7 seconds.- New decision record,
tools/cross-port/specs/MULTI_PROTOCOL_HOSTING.md, in theIDENTITY_V1_SPEC.mdstyle: - the decisions above, with reasons - the conformance fixture protocol (Phase 1) - per-port deliverables - the pinned protocol hash for the fixture protocol docs/porting-guide.md. A checklist entry for each item above.
Commit titles name the finding, e.g. "§8 never said how a retry hint reaches the client, and all seven ports dropped it".
Phase 1 — Python reference (vgi-rpc)
- Error model.
- A
Codeenum and the detail catalog as dataclasses invgi_rpc, withto_json/from_json. - Every error class with anerror_kindalso declares itserror_code. Classes may supplyerror_details. -Message.from_exception(vgi_rpc/log.py:207) writes code, kind and details to the top-level keys andlog_extra, enforcing the cap. -IdentityUnavailableErrorandAuthUnavailableErrorsupplyRetryInfo. - One server switch for tracebacks, on by default on every transport. - Access-log records gainerror_code. - Client.
-
RpcError(vgi_rpc/rpc/_common.py:688) gainserror_code,error_kindanderror_details, plus typed accessors (retry_info(),bad_request(), …), filled on every decode path: pipe, HTTP, stream, and external. -is_retryable()onRpcError, following the rule above. No automatic retry. -client_driver.pyreports all three. - Identity translation.
IdentityImplcatchesAuthUnavailableErrorfrom either hook and raisesIdentityUnavailableError(detail, retry_after=...). This also makes vgi-python's documented advice correct. - Reflection. No code change;
features=[]is now what the spec requires. - Conformance fixture protocol. The conformance worker hosts a second
application protocol,
conformance.Secondary.v1, throughextra_protocols. -echo(value: str) -> str. The primary also has a method of this name, so routing by(protocol, method)is exercised. -fail(code: str, kind: str, retry_delay_seconds: float): raises an error with that code and kind, plusRetryInfowhen the delay is positive. -fail_oversized(): raises an error whose details exceed 4 KiB. - Conformance tests (
vgi_rpc/conformance/, run against every port on every transport): - The secondary is listed second bylist_protocols, anddescribeworks. -echoroutes by pair and the same-named primary method stays distinct. -failround-tripserror_code,error_kindanderror_details. - The Identity fixture's auth-unavailable token yieldsUNAVAILABLE/identity_unavailablewithRetryInfoof 7 seconds. - Every existing kind carries the code in the mapping table. - Tracebacks are present by default on every transport and absent when the switch is off. - Detail rules: unknown detail types are ignored by clients; an array over 4 KiB is dropped whole. - Client role: throughclient_driver, each port's client reportserror_code,error_kindanderror_detailsfrom the reference server. - Hosted-protocols group, for SDK workers. Takes a worker command or URL plus the expected protocol list. It asserts that reflection lists every expected protocol in order, thatconformance.Secondary.v1passes the group above, and, over HTTP when the fixture opts in, thatvgi_rpc.Identity.v1is listed and passes the Identity fixture tests. It runs on stdio, unix and HTTP. - Tooling.
tools/cross-port/describe_diff.pycompares the secondary's hash across ports. - Release. vgi-rpc to PyPI (remote
upstream), before anything consumes it.
Phase 2 — The six other vgi-rpc ports
Same deliverables in each, verified by the Phase 1 suite in both directions: the port's server against the reference client, and the port's client against the reference server.
| Port | Public hosting API to add | Reserved prefix | Emit code + details; traceback setting | Translate auth-unavailable | Client exposes code / kind / details |
|---|---|---|---|---|---|
| TypeScript | Launchers (serveTcp, serveUnix, serveStream) accept a protocol host with extra protocols, closing the gap noted at server.ts:177 |
ok | add | ok | add both (log-batch.ts:33) |
| Go | Exported binding constructor, so AddProtocol works outside the package |
ok | add | add (identity_v1.go:587) |
detail; use errors.As for kind (wire.go:251) |
| Rust | Builder method to add a protocol with its dispatcher | add on the application name (server.rs:561) |
add (server.rs:1989) |
ok | add both (envelope.rs:57) |
| C# | Constructor or builder accepting additional (interface, implementation) pairs |
extend to derived names | add | add (IdentityImpl.cs:134) |
detail |
| Java | addProtocol(Class<?>, Object) beside setIdentity |
ok | add (Wire.java:232) |
add (IdentityImpl.java:198) |
detail |
| C++ | ServerBuilder::add_protocol(...) |
add (server.cpp:129) |
add (result.cpp:80) |
add (token_identity.cpp:481) |
detail |
Each port's conformance worker hosts conformance.Secondary.v1, and its client
driver reports error_code, error_kind and error_details. Releases follow the per-repo
mechanics (crates.io and Go on tag push, npm and Maven on release). The VGI SDKs
that depend on these ports wait for each release.
Phase 3 — The seven VGI SDKs
Common contract.
- Hook. A worker overrides one hook that returns its additional protocols as
(protocol, implementation)pairs. It is called once when the server is built, may consult configuration, and its result is hosted on every transport the worker serves. - Identity. Hosted when the worker implements
resolve_tokenand/ormint_grant, on HTTP. Enabling introspection without an allowlist of principals refuses to start. This is vgi-python's existing rule, ported.
| SDK | Work |
|---|---|
| vgi-python | build_rpc_server(worker, transport) replaces the 5 RpcServer(...) calls (vgi/serve.py:321, :612; vgi/worker.py:1727, :5565, :5644). Add Worker.hosted_protocols(cls) -> Sequence[tuple[type, object]]. Iroh gets the same set. Docs: AuthUnavailableError is now correct advice; say why. |
| vgi-typescript | Consolidate its 5 sites (worker.ts:198, :223, :248, :289; http/fetch.ts:200) into one builder; add the hook; add Identity opt-in with the allowlist rule. |
| vgi-go | Hook and Identity in buildServer (vgi/worker.go:1235). |
| vgi-rust | Hook and Identity in build_parts (vgi/src/worker.rs:297). |
| vgi-csharp | Hook and Identity in NewRpcServer (Worker.cs:381). |
| vgi-java | Hook and Identity in buildServer (Worker.java:1138); bump the vgirpc dependency and the VGI_RPC_JAVA_REF CI pins. |
| vgi-c++ | Hook and Identity in Worker::run (worker.cpp:231). |
Fixture and test.
- Each SDK's example fixture worker hosts
conformance.Secondary.v1through its new hook. This is additive: no existing fixture function changes. - Over HTTP, the fixture worker also opts into Identity with the policy from
IDENTITY_CONFORMANCE_FIXTURE.md, including the auth-unavailable test token. - Acceptance is vgi-rpc-python's hosted-protocols group (Phase 1, item 6), run
against each SDK's fixture worker with the expected list
vgi.v2,conformance.Secondary.v1. - Each SDK's CI runs that group, the way the ports already run the shared suite.
Order and parallelism
flowchart LR
S[Phase 0: spec] --> P[Phase 1: Python vgi-rpc + suite] --> R1[release vgi-rpc]
R1 --> TS[vgi-rpc-typescript] & GO[vgi-rpc-go] & RS[vgi-rpc-rust] & CS[vgi-rpc-csharp] & JV[vgi-rpc-java] & CP[vgi-rpc-c++]
R1 --> VP[vgi-python]
TS --> VT[vgi-typescript]
GO --> VG[vgi-go]
RS --> VR[vgi-rust]
CS --> VC[vgi-csharp]
JV --> VJ[vgi-java]
CP --> VX[vgi-c++]
- Phases 0 and 1 are sequential and done in this session.
- The six ports are independent. Each can run as its own agent in its own repo, with the shared suite as the acceptance test.
- Agents must not share a scratch directory or run suites concurrently on one machine; that has corrupted results before.
- Each SDK follows its transport's release.
Definition of done
- The cross-port conformance run prints
CONFORMANCE PASSED (all transports), including the new secondary-protocol and error-detail groups. This holds in both directions for all seven ports. describe_diff.pyreports identical hashes forconformance.Secondary.v1.- Each of the seven SDKs:
- hosts a worker-supplied protocol on every transport
- hosts Identity over HTTP when opted in
- passes the hosted-protocols group
- Every repo's own CI is green, including vgi-python's
mypy tests/,ruff format --checkand the Windows matrix.
Outcome
Everything above shipped on 2026-10-06, followed the same day by grants
accepted as bearer credentials (vgi-rpc GRANT_AUTHENTICATION.md), which the
reporting design needs for unattended runs.
| Language | vgi-rpc port | VGI SDK |
|---|---|---|
| Python | vgi-rpc 0.49.0 | vgi-python 0.42.0 |
| TypeScript | vgi-rpc-typescript 0.27.0 | vgi-typescript 0.40.0 |
| Go | vgi-rpc-go 0.32.0 | vgi-go 0.31.0 |
| C# | vgi-rpc-csharp 0.14.0 | vgi-csharp 0.15.0 |
| Java | vgi-rpc-java 0.29.0 | vgi-java 0.38.0 |
| Rust | vgi-rpc-rust 0.30.0 | vgi-rust 0.40.0 |
| C++ | vgi-rpc-cpp 0.7.0 | vgi-cpp 0.7.0 |
Versions are the grant-acceptance releases; the hosting and error-model work landed in earlier releases.
Differences from this plan:
- Tracebacks default on everywhere (above), not off on HTTP.
revoke_grantand aGrantAuthenticatorwere not built. Sealed grants are stateless and not individually revocable; a short framework maximum lifetime and key rotation stood in. The reporting design now makes lifetime the worker's policy (see credentials.md).
Still open from this round:
IDENTITY_V1_SPEC.md§9.3 doesn't say what happens when a deployment's own authenticator returns anonymous for a bearer it doesn't recognise. The reference stays anonymous; Java and Rust return 401.- vgi-rpc
TestSticky::test_expired_session_surfaces_session_lostis flaky on macOS CI.
Implementing the reporting protocols
The build plan for the protocols in this directory, written so an agent with no other context can pick up where the last one stopped. Read this file, then the README, credentials.md and the spec of the phase you're on. Update the status table when a phase lands, in the same commit.
Status
| Phase | Status | Landed in |
|---|---|---|
| P0 Specs committed, HTML builder in the repo | done | this commit |
| P1 Unlimited default grant lifetime | not started | |
| P2 SQL-binding framework (vgi-python) | not started | |
P3 vgi.reports.v1 |
not started | |
P4 vgi.delegations.v1 |
not started | |
P5 Run engine and vgi.schedules.v1 (run_query) |
not started | |
P6 vgi.sql_tasks.v1 |
not started | |
P7 vgi.notify.v1 reference and deliveries |
not started | |
P8 vgi.alerts.v1 |
not started | |
P9 vgi.report_render.v1 and render_report |
not started | |
| P10 Cupola client | not started (can start after P3) |
Already shipped and not part of this plan: multi-protocol hosting, the gRPC
error model, Identity.v1, grants as bearers, attach tickets
(vgi.attach_tickets.v1, vgi_export_session(), the attach_ticket ATTACH
option), full vgi.v2 parity across the seven SDKs.
Decisions
Settled; don't reopen them without the user. The README's Decisions table has the protocol-level ones; these are the build-level ones and late changes.
| # | Decision |
|---|---|
| D1 | The protocol is the unit of optionality; every method is required. |
| D2 | vgi.delegations.v1 (not "held attachments"): Delegation, put_delegations, list_delegations, revoke_delegation. |
| D3 | vgi.sql_tasks.v1 is its own protocol on the shared run engine, for SQL that changes data. Both destination kinds (catalog, host) and both watermark kinds (derived, stored). Scripts run statements in one transaction by default; CALL is a statement. |
| D4 | vgi.schedules.v1's run_query stays as specified (it may write). The overlap with SQL tasks is accepted. |
| D5 | The delegation field stays grant. DuckDB accepts it unquoted (the extension's attach_ticket/round_trip.test does SELECT grant FROM …). |
| D6 | The SQL binding is normative and derived from the RPC definition by a generic adapter plus a per-protocol annotation; never hand-written (README). |
| D7 | Protocol-table writes are refused inside an explicit multi-statement transaction (each row is an RPC that ROLLBACK can't undo). Revisit only if a real use needs it. |
| D8 | Specs stay in docs/design/reporting-protocols/ until each protocol ships; a shipped protocol's spec then moves to docs/protocol/. |
| D9 | These are service protocols. The reference implementation is Python; other SDKs implement one only when someone wants to host it natively. The extension tests the SQL bindings cross-language. |
| D10 | Build order is by dependency, P1 to P9. Each phase ends usable, with conformance over both surfaces. |
Working rules
Where work happens
- Build and test on the EC2 host
ec2-user@13.217.25.2in a directory you create and own, not the shared Mac. Debug builds, no LTO, at most 8 jobs. Kill only processes you started, by PID; never a broadpkill. - Never edit
~/Development/vgior~/Development/vgi-typescriptin place; use a git worktree. vgi-python's main checkout may hold the user's uncommitted files: never stage what you didn't write. - Implement, never skip: missing SDK functionality is ported, not excluded from a cross-SDK test.
Testing
- vgi-python:
uv run ruff check --fix . && uv run ruff format . && uv run mypy vgi/, thenuv run pytest -n auto. - Feature tests that cross languages go in the extension as sqllogictests
(
~/Development/vgi/test/sql/integration/<protocol>/). Every localunittestrun passes--test-dir <tree>(otherwise it silently runs its compiled-in tree) and--test-config test/configs/no_error_skip.json(otherwise HTTP-lane errors become skips). - Anything a test starts in the background, the test stops: record PIDs in the
parent shell, never inside
$( … ).
Commits and releases
- Agents commit on a branch and report; they don't push or tag.
- Docs and script-only changes go straight to
main, no PR. - A package release is CI-gated: push the release commit to
main, wait for every push-triggered workflow on that SHA to succeed (zero runs is a failure: check the repo name fromgit remote), then create the GitHub release or push the tag, whichever that repo publishes on. Retry GitHub 5xx. - Commit messages are findings-style (what was wrong or missing, how it was
found, what changed, how it was verified) and end with the session's
Co-Authored-Bytrailer.
Phases
P0 Specs committed (done)
docs/design/reporting-protocols/ committed, with
scripts/build_reporting_protocols_html.py, which regenerates
reporting-protocols.html from the markdown. Rerun it after editing a spec.
P1 Unlimited default grant lifetime
The design says a grant's lifetime is the issuing worker's policy and unlimited
unless set. vgi-rpc still defaults to 7 days (DEFAULT_MAX_TTL_SECONDS in
vgi_rpc/grants.py), so every unattended job would stop a week after its
owner's last login.
- Decide the wire form of "no expiry" in the grant spec first (an absent or
zero
exp), with test vectors. - Unset
VGI_RPC_GRANT_MAX_TTL_SECONDSmeans no maximum, in vgi-rpc and all six ports. Attach tickets follow the worker's grant maximum, so check the SDKs' ticket expiry the same way. - Done when the shared vectors pass in every port,
expires_atis+inf(orNULLinvgi_export_session()) for an unlimited worker, and the ports are released.
P1 doesn't block P2 to P4; development can set an explicit TTL meanwhile.
P2 SQL-binding framework (vgi-python)
The generic adapter that serves a hosted RPC protocol as a VGI catalog schema, built before any protocol so each ships both surfaces.
- Protocol classes for every protocol: Arrow-typed records and the
Protocolclass, plus the SQL annotation (tables, get methods for fetched columns, DML methods). Proposed homevgi/reporting/; adjust if the codebase suggests better. - The adapter, implementing the README's derivation rules
1 to 9 and the semantics (conflict,
request_id, autocommit, error text). scripts/regen_generated.py --checkrenders each spec's SQL-binding table from the annotations.- Done when each derivation rule and semantic has a test against a toy protocol, and a protocol with an annotation that names a missing method or a column the row lacks fails at startup.
P3 vgi.reports.v1
- The store on
FunctionStorage,AccessPolicy(by principal: grant callers have no claims),ReadOnlyReportStore,vgi-report-serve. - Pin every row schema in reports.md exactly; the SQL binding makes them a contract.
- A conformance suite that runs against any implementation over RPC and over SQL.
- Extension sqllogictests
test/sql/integration/reports/against the Python service. - Done when the conformance suite passes over both surfaces and Cupola could save, list, publish and read history with no localStorage.
P4 vgi.delegations.v1
put_delegations,list_delegations,revoke_delegation; grants and tickets encrypted under a key outside the store, never returned.- Done when this round trip passes as a sqllogictest:
vgi_export_session(),INSERT INTO … delegations BY NAME …, then a runner reattaches each source withATTACH … (bearer_token, attach_ticket)from the stored delegation and reads data, andrevoke_delegationmakes the next reattach fail.
P5 Run engine and vgi.schedules.v1
- The shared engine from schedules.md:
tick/advance_run,vgi.scheduler_driver.v1(allowlisted), leases, idempotent steps, hardened per-run sessions built from delegations. vgi.schedules.v1withrun_query; deliveries wait for P7.- Done when a schedule runs as its owner from delegations, runs are unique
on
(schedule_id, scheduled_for)across two replicas, overlap isskipped, and a missing delegation isgrant_required.
P6 vgi.sql_tasks.v1
load(four modes),script(three transaction kinds), both watermark kinds,lookback, thehoststore (DuckLake by default), health and transition notifications (deliveries once P7 lands).- Done when
derivedloads exactly once across a crash injected between write and commit,storedreplays at most the last window, a script that writes two databases insinglefails withtransaction_failed, andtest_runleaves the target untouched.
P7 vgi.notify.v1 reference and deliveries
- A reference service with a recording fake channel and a webhook channel;
destination policy and suppression list. Wire schedule deliveries and task
notifications to it. The production service is the separate
vgi-notifyrepo, later.
P8 vgi.alerts.v1
- Rules, instances and states, notifications on transitions, acknowledge, snooze, subscriptions, on the shared engine. Pin the instance row schema (it's loose in alerts.md today).
P9 vgi.report_render.v1 and render_report
- Depends on a headless Cupola renderer. Then enable the
render_reportschedule action.
P10 Cupola client
- Starts after P3: reports client and migration off localStorage, then delegation on save and renewal on login (P4), schedule and task editors (P5, P6), "Alert me when…" (P8). TypeScript types and clients come from codegen.
Known gaps to close while building
- Row schemas are prose in places ("instances with state and times"). Each phase pins its protocol's rows; the regen check keeps the spec in step.
get_*_infolimits (sizes, minimum intervals) have no values yet; choose them in the phase and record them in the spec.