Hub Export Workflow

This page defines the sender-side workflow for lx-annotate when a site node exports anonymized resources to a hub deployment.

It is the local companion to the upstream hub receive contract in endoreg_db. The goal is to make outbound transfer restart-safe, idempotent, and explicit for operators in a high-stakes clinical environment.

Scope

This workflow applies to:

  • anonymized videos

  • anonymized reports

  • processed media transfer from a site node to a hub

This workflow does not apply to:

  • watcher or API ingest into the local node

  • raw-media transfer

  • the legacy annotation-segment export screen

Core Decisions

  • center_key is the canonical machine-facing center identifier

  • transfer is permitted only for anonymized resources

  • the sender exports only anonymized processed media

  • transfer is explicit: a resource must be marked for upload before it may be queued

  • retries must reuse the same logical transfer identity

Marked For Upload

marked for upload means:

  • the resource is locally approved for outbound transfer to the configured hub

  • the resource is eligible for transfer under the anonymization policy

  • the resource has not necessarily been queued or transferred yet

This is a local sender decision, not a remote hub status.

The mark must be:

  • persisted durably

  • attributable to a user or explicit system action

  • reversible before queueing

For the initial implementation, the mark should be per resource. Bulk marking is allowed as a UI convenience, but it must be implemented as repeated per-resource state changes, not as a separate domain concept.

Eligible Resources

A resource is eligible for outbound hub transfer only when all of the following are true:

  • it is a video or report supported by the upstream transfer contract

  • it belongs to the local sender center scope

  • it has been explicitly marked for upload

  • processed media exists locally

  • the anonymization state is one of:

    • ANONYMIZED

    • DONE_PROCESSING_ANONYMIZATION

    • VALIDATED

The following states are not export-eligible:

  • NOT_STARTED

  • STARTED

  • EXTRACTING_FRAMES

  • PROCESSING_ANONYMIZING

  • FAILED

If the local state becomes ineligible after marking, the sender must refuse to queue or retry the transfer until the resource becomes eligible again.

Sender State Machine

The local outbound transfer ledger should use this state model:

  • not_marked The default state. No outbound transfer intent exists.

  • marked Operator-approved for transfer, but not yet queued for delivery.

  • queued Ready for the sender worker to attempt registration with the hub.

  • registering The sender is submitting transfer metadata to the hub.

  • awaiting_media The hub accepted metadata and is waiting for processed media upload.

  • uploading The sender is uploading processed media to the hub.

  • completed The sender has received a terminal successful hub state.

  • failed The last transfer attempt failed. The resource may be retried.

Allowed transitions:

  • not_marked -> marked

  • marked -> not_marked

  • marked -> queued

  • queued -> registering

  • registering -> awaiting_media

  • registering -> completed

  • registering -> failed

  • awaiting_media -> uploading

  • awaiting_media -> failed

  • uploading -> completed

  • uploading -> failed

  • failed -> queued

Terminal behavior:

  • completed is terminal for the current transfer intent

  • any retry after completed must first resolve to the existing logical transfer, not create a new one

Transfer Identity

The sender must compute a deterministic transfer_key.

Recommended shape:

  • "{source_node_key}__{resource_kind}__{resource_hash}__processed_v1"

Required properties:

  • stable across retries

  • unique for a resource content hash and transfer mode

  • independent of local database primary keys

This key must be reused whenever the sender retries the same logical transfer.

Transfer Mode Policy

The sender must use:

  • metadata_and_processed_media

The sender must not use:

  • metadata_and_raw_media

  • metadata_raw_and_processed_media

That keeps the sender aligned with the upstream policy that only anonymized data may be transferred.

Retry And Reuse Rules

The sender must treat the following as idempotent reuse cases:

  • the local outbound job restarts after a crash

  • the metadata registration request times out and is retried

  • the hub responds with an already-existing transfer for the same transfer_key

  • processed media upload is retried after partial network failure

The sender must not create a new logical transfer when:

  • the resource hash is unchanged

  • the target node is unchanged

  • the transfer mode is unchanged

A new logical transfer is required only when the transfer intent changes in a way that would materially alter the hub-side contract, such as:

  • a different target hub node

  • a different resource hash

  • a different transfer mode

Sender Payload Responsibilities

Before contacting the hub, the sender must build a canonical payload that matches the upstream transfer serializer.

At minimum, the payload must include:

  • transfer_key

  • source_node_key

  • target_node_key

  • source_center_key

  • resource_kind

  • resource_hash

  • transfer_mode

  • processing_policy

  • processing_intent

  • cleanup_policy

  • resource_rows

  • processing_snapshot

For video transfer, the sender must provide:

  • resource_rows.video_file

  • resource_rows.video_state

  • resource_rows.sensitive_meta where available

For report transfer, the sender must provide:

  • resource_rows.raw_pdf_file

  • resource_rows.raw_pdf_state

  • resource_rows.sensitive_meta where available

The sender must validate the payload locally before any network request is issued.

UI Workflow

The operator-facing export workflow should be derived from the anonymization overview.

The new export page should:

  • show anonymization readiness first

  • allow marking and unmarking eligible resources

  • show current outbound sender state

  • keep the legacy annotation export page separate

The default filtered view should show only resources that are:

  • anonymized

  • locally complete enough to export

  • not already completed for the current transfer intent

Operational Requirements

Before queueing any transfer, the local node must have:

  • one active local site NetworkNode

  • one active target central_hub NetworkNode

  • a configured source center scope

  • a valid hub base URL

  • valid node authentication material

If any of these are missing, the UI and sender worker must refuse to start transfer work.

Audit Requirements

The sender must record:

  • who marked the resource for upload

  • when it was queued

  • when metadata registration started and finished

  • when media upload started and finished

  • the last failure reason

  • the final sender-visible hub outcome

Local Cleanup Policy

Sender-side cleanup is separate from hub receive-side cleanup.

The operational default is conservative:

  • retain local processed artifacts after verified hub apply

An optional sender-side policy may mark local processed artifacts as cleanup-eligible only after the hub has returned a successful terminal state.

Allowed sender policies:

  • retain_processed_media

  • eligible_after_verified_apply

This policy must never delete local artifacts before the sender has a verified successful hub outcome.

Summary

The sender-side workflow is intentionally explicit:

  • processing completion makes a resource eligible

  • operator marking authorizes export

  • a local transfer ledger tracks progress

  • retries reuse the same deterministic transfer identity

  • only anonymized processed media is transferred