Metadata-Version: 2.3
Name: whitebox-plugin-transfer-manager
Version: 0.1.5
Summary: A plugin for whitebox to manage file transfers from devices
License: GNU Affero General Public License v3
Author: avilabss
Author-email: contact@avilabs.net
Requires-Python: >=3.10.0
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Description-Content-Type: text/markdown

# Whitebox Plugin - Transfer Manager

Manages file transfers from external devices (cameras, etc.) with a two-phase workflow: download from device, then process (transcode/stitch/convert). Tracks persistent state, emits real-time progress to the frontend, and handles retries and crash recovery automatically.

## Installation

```
poetry add whitebox-plugin-transfer-manager
```

## Capabilities

| Provides | Requires |
|----------|----------|
| `transfer-manager` | `device` |

## Exposed to Other Plugins

### Plugin Classes (`get_plugin_classes_map`)

Accessible via `import_whitebox_plugin_class()`:

| Key | Type | Description |
|-----|------|-------------|
| `transfer.TransferManager` | `TransferManager` instance | Singleton service for queueing, dispatching, and tracking transfers |
| `transfer.TransferAdapter` | `Protocol` class | Interface that device plugins implement for download/processing |
| `transfer.RemoteFile` | `dataclass` class | Represents a file on a remote device |

### Models (`model_registry`)

| Key | Model | Description |
|-----|-------|-------------|
| `transfer.Transfer` | `Transfer` | Persistent record for each file transfer |

### Frontend Components (`exposed_component_map`)

| Category | Key | Component | Description |
|----------|-----|-----------|-------------|
| `service-component` | `transfer-service` | `TransferServiceComponent` | Background service that listens for WebSocket transfer events, updates the Zustand store, and triggers flight session re-fetches on completion |
| `transfer-manager` | `transfer-panel` | `TransferPanel` | UI panel showing active/completed transfers with progress bars, cancel, and retry actions |

### Frontend Slots (`slot_component_map`)

| Slot | Component |
|------|-----------|
| `transfer.panel` | `TransferPanel` |

### State Stores (`state_store_map`)

| Key | Module | Description |
|-----|--------|-------------|
| `transfer` | `stores/transfer` | Zustand store tracking transfer state (`transfers`, `activeCount`, `upsertTransfer`, `removeTransfer`, `clearCompleted`, `dismissAll`) |

## Daemon

**Command:** `daemon_transfer_manager`

Runs as a `BaseDaemonCommand` that orchestrates all transfers.

### Subscribed Events

| Event | Action |
|-------|--------|
| `observation.flight.end` | Dispatch any pending transfers |
| `observation.device.connection_status.update` | On reconnect, dispatch pending transfers for that device |
| `command.transfer.cancel` | Cancel a transfer (sets Redis flag for in-progress workers) |
| `command.transfer.retry` | Retry a failed/cancelled transfer |
| `command.transfer.queue_batch` | Create transfer records from a batch of files and dispatch |

### Periodic Checks (every 5s)

- **Stale transfer recovery**: Resets transfers stuck in active states with no DB update for 2+ minutes (worker likely dead)
- **Dispatch pending**: Picks up PENDING transfers respecting concurrency limits

### Startup Recovery

- Resets any in-progress transfers (DOWNLOAD_STARTED, DOWNLOAD_COMPLETED, PROCESSING_STARTED) back to PENDING
- Clears retry_after on PENDING transfers that were in backoff when daemon stopped

## Emitted Events

All events are sent to the `management` WebSocket group.

| Event | When |
|-------|------|
| `observation.transfer.queued` | Transfer created or reset to pending |
| `observation.transfer.download.started` | Download begins |
| `observation.transfer.download.progress` | Download progress (throttled: 1s or 5% delta) |
| `observation.transfer.download.completed` | Download finished |
| `observation.transfer.processing.started` | Processing begins |
| `observation.transfer.processing.progress` | Processing progress (throttled: 1s or 5% delta) |
| `observation.transfer.processing.completed` | Processing finished, FlightSessionRecording created |
| `observation.transfer.download.failed` | Download failed permanently |
| `observation.transfer.processing.failed` | Processing failed permanently |
| `observation.transfer.retry_scheduled` | Transfer scheduled for retry with backoff |
| `observation.transfer.cancelled` | User cancelled transfer |
| `observation.transfer.paused` | User paused transfer |

### Registering an Adapter

Device plugins expose adapters via their plugin class:

```python
# In device plugin's whitebox plugin class
def get_transfer_adapters(self):
    from .transfer_adapter import MyDeviceTransferAdapter
    return [("my_device_codename", MyDeviceTransferAdapter())]
```

The daemon discovers these on startup; workers discover lazily on first use.

## Process file recording command

**Command:** `process_file_recording [device_codename] [source_path] [destination_path]`

This command is used to process a file recording for a transfer manually.

Example:
```bash
docker exec backend-dev poetry run python whitebox/manage.py process_file_recording \
  insta360_x4 \
  /opt/whitebox/media/transfers/downloads/VID_001.insv \
  /opt/whitebox/media/transfers/processed/VID_001.m3u8
```

Running this command will process the specified `source path` with the transfer
adapter defined by the device plugin for the given `device_codename`, and store
it into the `destination path`, just like the transfer manager would do it.

It can be used for debugging or testing purposes.

## TransferManager API

### Queueing

```python
from plugin.registry import import_whitebox_plugin_class

transfer_manager = import_whitebox_plugin_class("transfer.TransferManager")
RemoteFile = import_whitebox_plugin_class("transfer.RemoteFile")

# Queue a batch of files (sync, for use in RQ tasks or daemon)
files = [RemoteFile(path="/DCIM/VID_001.insv", size=1_000_000)]
transfers = transfer_manager.queue_batch_sync(
    device_connection=conn,
    files=files,
    download_dir=Path("/media/transfers/downloads"),
    processed_dir=Path("/media/transfers/processed"),
    associate_with=flight_session,
)
```

### Concurrency

| Setting | Default | Description |
|---------|---------|-------------|
| Max concurrent global | 4 | Total active transfers across all devices |
| Max concurrent per device | 2 | Active transfers per device connection |

### Retry Logic

- Transient errors (network, device) retry with exponential backoff: 2s, 4s, 8s, ..., capped at 60s
- Retry window: 7 days from initial queue time
- Safety cap: 500 retries max
- Non-transient errors (disk full, corrupt file) do not retry

## Additional Instructions

- [Plugin Development Guide](https://docs.whitebox.aero/plugin_guide/#plugin-development-workflow)
- [Plugin Testing Guide](https://docs.whitebox.aero/plugin_guide/#testing-plugins)
- [Contributing Guidelines](https://docs.whitebox.aero/development_guide/#contributing)

