Metadata-Version: 2.5
Name: gi-ingest
Version: 0.6.0
Summary: Upload robotics recordings from directories and reclaim confirmed files
License: Proprietary
Requires-Python: >=3.11
Requires-Dist: boto3>=1.34
Requires-Dist: httpx>=0.27
Requires-Dist: mcap>=1.2
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# gi-ingest

Upload robotics recordings from vendor-managed directories and reclaim only
recordings whose delivery has been confirmed by GILabs.

Vendors copy SD cards to SSDs or local storage themselves. `gi-ingest` uploads
those files in place: there is no `stage` command and no staging disk to configure.
Keep each recording's directory, part filenames, and metadata sidecar together.

## Install and authenticate

macOS and Linux, Python 3.11 or newer:

```bash
uv tool install gi-ingest
gi-ingest login --api-key gik_...
gi-ingest doctor
```

Get an upload token from the vendor portal at
[vendor.gilabs.xyz](https://vendor.gilabs.xyz). Select a destination when your
vendor has more than one:

```bash
gi-ingest destinations --select
```

## Upload from any number of paths

Pass a recording directory, a copied card directory, or a parent containing
recordings. Paths may span multiple SSDs and the computer's local disk:

```bash
gi-ingest upload /Volumes/SSD1/recordings /Volumes/SSD2/recordings ~/recordings \
  --jobs 4 --max-bandwidth 50M
```

Discovery is recursive. Overlapping input paths are deduplicated. Symlink paths
and recordings containing symlinks are refused. Missing or disconnected input
directories are reported rather than silently omitted.

`--jobs` controls concurrent recording inspection and uploads (default 4, range
1–32). `--max-bandwidth` is an aggregate limit across the process, in bytes per
second; `50M` means 50 MiB/s. Stop recorder activity and finish copying before
uploading. Keep source disks connected until the command finishes.

```bash
gi-ingest upload /Volumes/SSD1/recordings --project <project-id>
gi-ingest upload /Volumes/SSD1/recordings --dry-run
```

Recordings with bad or unset clocks require an individual collection date.
An interactive upload shows each affected recording's full path, device, and
reported timestamp, then asks for its date (YYYY-MM-DD). Type `skip` to leave that
recording unresolved. No answer is shared across recordings, even on the same SSD
or with the same reported timestamp. Healthy timestamps remain unchanged.

Unattended uploads hold these recordings, save their paths and content identities
in the ledger, and return nonzero. Run `gi-ingest attribute` from a terminal with
the SSD mounted to resolve each date, followed by any missing operator/environment
labels. Corrections are retained per recording for retries. If files move, run
upload with their new paths first. Reclaim keeps unresolved recordings.
There is no upload-wide `--collection-date` option. Recordings with a confirmed
missing IMU are held; `--allow-missing-imu` explicitly permits them. Short clips
are uploaded for the platform to evaluate.

Operator and environment attribution is required. Interactive upload shows a
numbered inventory with full paths, device, date, and existing labels. It offers:

1. Select recordings and label a group (`1,3-5` or `all`).
2. Label recordings individually.
3. Continue with ready recordings and leave the rest for `attribute`.

The inventory uses natural full-path order (`session_2` before `session_10`).
Numbers stay fixed throughout the labeling session. No grouping is inferred from
paths, devices, or dates. A review and confirmation precede every group write.
Only missing fields are filled; conflicting existing labels exclude a recording
from that group and are never overwritten. Sidecars are updated in place.

To label before uploading, supply paths directly to `attribute`. It scans
recursively and saves source registrations without uploading. With paths it only
labels that selection; without paths it uses the saved queue:

```bash
gi-ingest attribute /Volumes/SSD1/data /Volumes/SSD2/data
gi-ingest upload /Volumes/SSD1/data /Volumes/SSD2/data
```

`upload --no-prompt` holds missing dates and labels for later attribution even in
an interactive terminal. Ready recordings still upload, and any held recordings
cause a nonzero exit code. Unattended upload uses the same behavior:

```bash
gi-ingest upload /Volumes/SSD1/recordings --no-prompt
gi-ingest attribute                   # select groups and resolve held dates
gi-ingest upload /Volumes/SSD1/recordings
```

Plain MCAP recordings can recover a missing `session.json` from embedded
metadata. The recovered sidecar is created alongside the recording; an existing
unreadable sidecar must be repaired. Recording payloads are never rewritten.

A dry run reads and hashes the selected recordings using a temporary copy of
ledger state. It does not upload, change source files, or update the saved ledger.
Missing labels are checked against recovered metadata in memory too. If a preview
finds missing dates or labels, rerun the same upload command without `--dry-run`
from a terminal to answer the prompts. Running `attribute` immediately after a
dry run cannot resolve those findings: the preview has not saved them to the queue.

## Interruptions and retries

Run the same upload command again after a crash or network failure. An open
delivery is reused. Objects with matching sizes and completed-file checkpoints
are skipped; uncertain transfers are sent again. You can also resume the local queue without supplying paths:

```bash
gi-ingest upload
```

If a recording moves to another disk, supply its new path with its original
session directory name. Its content identity retains its upload state. To resume
an open delivery with explicit paths, include all of that delivery's recordings.
The CLI refuses to seal only part of an existing delivery.

A failed transfer leaves the delivery open and returns a nonzero exit code.
Recordings held for missing metadata or attribution also make the command return
nonzero, even when other recordings were delivered successfully. Read the named
failures; do not treat a partial delivery as permission to delete the whole input.
Changed files in an open delivery must be restored, or the delivery explicitly
abandoned before starting again:

```bash
gi-ingest abandon --batch <batch-id>
gi-ingest upload /Volumes/SSD1/recordings
```

Only one mutating CLI operation may use a ledger at a time. Use `--jobs` for
native concurrency, rather than multiple processes sharing a ledger.

## Reclaim only the paths you choose

**Keep at least one verified local copy until delivery is confirmed.**
`reclaim` is the supported way to enforce that rule. A transfer completing is
not enough: the server must have accepted the sealed delivery.

```bash
gi-ingest reclaim /Volumes/SSD1/recordings --dry-run
gi-ingest reclaim /Volumes/SSD1/recordings
```

Reclaim searches only the supplied paths. It compares each recording's part
inventory, sizes, SHA-256 hashes, and device metadata against a confirmed entry
in the local ledger before deleting its session directory. A second copy, such
as the original SD card, can be reclaimed by supplying that copy's path too.
Physical card identity or a particular mountpoint is not required.

Unconfirmed recordings, changed contents, missing proof, and directories with
unknown files are kept. There is no option to wipe undelivered or unusable
recordings through this command. Reclaim never purges another disk or every
saved source path as a side effect. There is no default scan of mounted drives.

Confirmation means the server accepted the raw delivery, not that downstream
processing or quality review has finished. The current server seal verifies
object presence and sizes; the CLI records SHA-256 values and uses them for
local content matching. Manually deleting the last local copy before confirmation
bypasses these safeguards and can lose data.

## Status and local state

```bash
gi-ingest status
gi-ingest history
gi-ingest history --batch <batch-id>
gi-ingest retry
gi-ingest config show
```

Authentication and defaults live in `~/.config/gi-ingest/config.json`; upload and
delivery records live in `~/.local/state/gi-ingest/ledger.db`. Keep the ledger:
without a matching confirmation record, reclaim preserves the files. Use
`--ledger PATH` on upload and reclaim to use another ledger, and use the same
ledger for both commands.

Existing ledgers and staged files remain usable. Upload their directories
directly, or run `upload` without paths to resume the old queue. `staged` remains
an internal state meaning ready to upload; it no longer implies a separate copy
step. The old `staging-dir` setting is unused by upload and reclaim.

`reset` and `drop` preserve files registered by path-based upload. Their legacy
cleanup behavior still applies to CLI-owned staging copies from older versions;
inspect their dry-run output before changing old queue entries.

## Encrypted recordings

VREC-encrypted recordings retain the recorder's `ego_*.mcap` filenames. Upload
preserves ciphertext, including sizes and SHA-256 values in the manifest.
Vendors do not need a recording key. The backend handles decryption and IMU
inspection; accepting ciphertext does not certify that it contains IMU data.

Keep `session.json` beside encrypted parts. Without it, the CLI cannot recover
metadata from ciphertext and holds the recording until the sidecar is restored.

## Development

```bash
uv sync --extra dev
uv run pytest -q
uv run ruff check gi_ingest tests
```
