Skip to content
Back to top
View this page

# Hub Export Workflow

This page defines the sender-side workflow for `lx-annotate` when a site node exports anonymized resources to a hub deployment.

It is the local companion to the upstream hub receive contract in `endoreg_db`. The goal is to make outbound transfer restart-safe, idempotent, and explicit for operators in a high-stakes clinical environment.

## Scope

This workflow applies to:

- anonymized videos

- anonymized reports

- processed media transfer from a site node to a hub

This workflow does not apply to:

- watcher or API ingest into the local node

- raw-media transfer

- the legacy annotation-segment export screen

## Core Decisions

- `center_key` is the canonical machine-facing center identifier

- transfer is permitted only for anonymized resources

- the sender exports only anonymized processed media

- transfer is explicit: a resource must be marked for upload before it may be queued

- retries must reuse the same logical transfer identity

## Marked For Upload

`marked for upload` means:

- the resource is locally approved for outbound transfer to the configured hub

- the resource is eligible for transfer under the anonymization policy

- the resource has not necessarily been queued or transferred yet

This is a local sender decision, not a remote hub status.

The mark must be:

- persisted durably

- attributable to a user or explicit system action

- reversible before queueing

For the initial implementation, the mark should be per resource. Bulk marking is allowed as a UI convenience, but it must be implemented as repeated per-resource state changes, not as a separate domain concept.

## Eligible Resources

A resource is eligible for outbound hub transfer only when all of the following are true:

- it is a video or report supported by the upstream transfer contract

- it belongs to the local sender center scope

- it has been explicitly marked for upload

- processed media exists locally

- the anonymization state is one of:

  - `ANONYMIZED`

  - `DONE_PROCESSING_ANONYMIZATION`

  - `VALIDATED`

The following states are not export-eligible:

- `NOT_STARTED`

- `STARTED`

- `EXTRACTING_FRAMES`

- `PROCESSING_ANONYMIZING`

- `FAILED`

If the local state becomes ineligible after marking, the sender must refuse to queue or retry the transfer until the resource becomes eligible again.

## Sender State Machine

The local outbound transfer ledger should use this state model:

- `not_marked` The default state. No outbound transfer intent exists.

- `marked` Operator-approved for transfer, but not yet queued for delivery.

- `queued` Ready for the sender worker to attempt registration with the hub.

- `registering` The sender is submitting transfer metadata to the hub.

- `awaiting_media` The hub accepted metadata and is waiting for processed media upload.

- `uploading` The sender is uploading processed media to the hub.

- `completed` The sender has received a terminal successful hub state.

- `failed` The last transfer attempt failed. The resource may be retried.

Allowed transitions:

- `not_marked -> marked`

- `marked -> not_marked`

- `marked -> queued`

- `queued -> registering`

- `registering -> awaiting_media`

- `registering -> completed`

- `registering -> failed`

- `awaiting_media -> uploading`

- `awaiting_media -> failed`

- `uploading -> completed`

- `uploading -> failed`

- `failed -> queued`

Terminal behavior:

- `completed` is terminal for the current transfer intent

- any retry after `completed` must first resolve to the existing logical transfer, not create a new one

## Transfer Identity

The sender must compute a deterministic `transfer_key`.

Recommended shape:

- `"{source_node_key}__{resource_kind}__{resource_hash}__processed_v1"`

Required properties:

- stable across retries

- unique for a resource content hash and transfer mode

- independent of local database primary keys

This key must be reused whenever the sender retries the same logical transfer.

## Transfer Mode Policy

The sender must use:

- `metadata_and_processed_media`

The sender must not use:

- `metadata_and_raw_media`

- `metadata_raw_and_processed_media`

That keeps the sender aligned with the upstream policy that only anonymized data may be transferred.

## Retry And Reuse Rules

The sender must treat the following as idempotent reuse cases:

- the local outbound job restarts after a crash

- the metadata registration request times out and is retried

- the hub responds with an already-existing transfer for the same `transfer_key`

- processed media upload is retried after partial network failure

The sender must not create a new logical transfer when:

- the resource hash is unchanged

- the target node is unchanged

- the transfer mode is unchanged

A new logical transfer is required only when the transfer intent changes in a way that would materially alter the hub-side contract, such as:

- a different target hub node

- a different resource hash

- a different transfer mode

## Sender Payload Responsibilities

Before contacting the hub, the sender must build a canonical payload that matches the upstream transfer serializer.

At minimum, the payload must include:

- `transfer_key`

- `source_node_key`

- `target_node_key`

- `source_center_key`

- `resource_kind`

- `resource_hash`

- `transfer_mode`

- `processing_policy`

- `processing_intent`

- `cleanup_policy`

- `resource_rows`

- `processing_snapshot`

For video transfer, the sender must provide:

- `resource_rows.video_file`

- `resource_rows.video_state`

- `resource_rows.sensitive_meta` where available

For report transfer, the sender must provide:

- `resource_rows.raw_pdf_file`

- `resource_rows.raw_pdf_state`

- `resource_rows.sensitive_meta` where available

The sender must validate the payload locally before any network request is issued.

## UI Workflow

The operator-facing export workflow should be derived from the anonymization overview.

The new export page should:

- show anonymization readiness first

- allow marking and unmarking eligible resources

- show current outbound sender state

- keep the legacy annotation export page separate

The default filtered view should show only resources that are:

- anonymized

- locally complete enough to export

- not already completed for the current transfer intent

## Operational Requirements

Before queueing any transfer, the local node must have:

- one active local site `NetworkNode`

- one active target `central_hub` `NetworkNode`

- a configured source center scope

- a valid hub base URL

- valid node authentication material

If any of these are missing, the UI and sender worker must refuse to start transfer work.

## Audit Requirements

The sender must record:

- who marked the resource for upload

- when it was queued

- when metadata registration started and finished

- when media upload started and finished

- the last failure reason

- the final sender-visible hub outcome

## Local Cleanup Policy

Sender-side cleanup is separate from hub receive-side cleanup.

The operational default is conservative:

- retain local processed artifacts after verified hub apply

An optional sender-side policy may mark local processed artifacts as cleanup-eligible only after the hub has returned a successful terminal state.

Allowed sender policies:

- `retain_processed_media`

- `eligible_after_verified_apply`

This policy must never delete local artifacts before the sender has a verified successful hub outcome.

## Summary

The sender-side workflow is intentionally explicit:

- processing completion makes a resource eligible

- operator marking authorizes export

- a local transfer ledger tracks progress

- retries reuse the same deterministic transfer identity

- only anonymized processed media is transferred