Metadata-Version: 2.5
Name: gi-ingest
Version: 0.9.0
Summary: Upload robotics recordings from directories and reclaim confirmed files
License: Proprietary
Requires-Python: >=3.11
Requires-Dist: boto3>=1.34
Requires-Dist: httpx>=0.27
Requires-Dist: mcap>=1.2
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: desktop
Requires-Dist: pyside6<7,>=6.8; extra == 'desktop'
Provides-Extra: desktop-build
Requires-Dist: pyinstaller<7,>=6.20; extra == 'desktop-build'
Provides-Extra: dev
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# gi-ingest

See [CHANGELOG.md](CHANGELOG.md) for release notes and unreleased changes.

Upload robotics recordings from vendor-managed directories and reclaim only
recordings whose delivery has been confirmed by GILabs.

For a native Windows UI with verified SD-card-to-SSD staging, persistent caching,
labels, and uploads, see [GI Ingest for Windows](WINDOWS.md).

With the CLI, vendors copy SD cards to SSDs or local storage themselves. `gi-ingest` uploads
those files in place: there is no `stage` command and no staging disk to configure.
Keep each recording's directory, part filenames, and metadata sidecar together.

## Install and authenticate

macOS and Linux, Python 3.11 or newer:

```bash
uv tool install gi-ingest
gi-ingest login --api-key gik_...
gi-ingest doctor
```

Get an upload token from the vendor portal at
[vendor.gilabs.xyz](https://vendor.gilabs.xyz). Select a destination when your
vendor has more than one:

```bash
gi-ingest destinations --select
```

## Upload from any number of paths

Pass a recording directory, a copied card directory, or a parent containing
recordings. Paths may span multiple SSDs and the computer's local disk:

```bash
gi-ingest upload /Volumes/SSD1/recordings /Volumes/SSD2/recordings ~/recordings \
  --jobs 4 --max-bandwidth 50M
```

Discovery is recursive. Overlapping input paths are deduplicated. Symlink paths
and recordings containing symlinks are refused. Missing or disconnected input
directories are reported rather than silently omitted.

`--jobs` controls concurrent recording inspection and uploads (default 4, range
1–32). `--max-bandwidth` is an aggregate limit across the process, in bytes per
second; `50M` means 50 MiB/s. Stop recorder activity and finish copying before
uploading. Keep source disks connected until the command finishes.

`--accelerate auto` (the default) measures standard and accelerated S3 endpoints
and uses acceleration when it is at least 1.15× faster. Successful comparisons
are cached for three days across deliveries; the endpoint message shows the
cached result's age. A changed public egress IP, bucket/region, or bandwidth
limit triggers a new comparison. The CLI checks the public IP through AWS's
`checkip.amazonaws.com`; if that lookup fails, it probes without using the cache.
Failed comparisons are not cached. Cache storage follows `XDG_CACHE_HOME`,
defaulting to `~/.cache/gi-ingest/acceleration.json`.

Use `--refresh-acceleration` to ignore the cache and measure again, or
`--accelerate always|never` to force an endpoint without probing or reading the
cache. Forced modes take precedence over `--refresh-acceleration`.

```bash
gi-ingest upload /Volumes/SSD1/recordings --project <project-id>
gi-ingest upload /Volumes/SSD1/recordings --dry-run
```

Recordings with bad or unset clocks require an individual collection date.
An interactive upload shows each affected recording's full path, device, and
reported timestamp, then asks for its date (YYYY-MM-DD). Type `skip` to leave that
recording unresolved. No answer is shared across recordings, even on the same SSD
or with the same reported timestamp. Healthy timestamps remain unchanged.

Unattended uploads hold these recordings, save their paths and content identities
in the ledger, and return nonzero. Run `gi-ingest attribute` from a terminal with
the SSD mounted to resolve each date, followed by any missing operator/environment
labels. Corrections are retained per recording for retries. If files move, run
upload with their new paths first. Reclaim keeps unresolved recordings.
There is no upload-wide `--collection-date` option. Recordings with a confirmed
missing IMU are held; `--allow-missing-imu` explicitly permits them. Short clips
are uploaded for the platform to evaluate.

Operator and environment attribution is required. Interactive upload shows a
numbered inventory with full paths, device, date, and existing labels. It offers:

1. Select recordings and label a group (`1,3-5` or `all`).
2. Label recordings individually.
3. Continue with ready recordings and leave the rest for `attribute`.

The inventory uses natural full-path order (`session_2` before `session_10`).
Numbers stay fixed throughout the labeling session. No grouping is inferred from
paths, devices, or dates. A review and confirmation precede every group write.
Only missing fields are filled; conflicting existing labels exclude a recording
from that group and are never overwritten. Sidecars are updated in place.

To label before uploading, supply paths directly to `attribute`. It scans
recursively and saves source registrations without uploading. With paths it only
labels that selection; without paths it uses the saved queue:

```bash
gi-ingest attribute /Volumes/SSD1/data /Volumes/SSD2/data
gi-ingest upload /Volumes/SSD1/data /Volumes/SSD2/data
```

`upload --no-prompt` holds missing dates and labels for later attribution even in
an interactive terminal. Ready recordings still upload, and any held recordings
cause a nonzero exit code. Unattended upload uses the same behavior:

```bash
gi-ingest upload /Volumes/SSD1/recordings --no-prompt
gi-ingest attribute                   # select groups and resolve held dates
gi-ingest upload /Volumes/SSD1/recordings
```

Plain MCAP recordings can recover a missing `session.json` from embedded
metadata. The recovered sidecar is created alongside the recording; an existing
unreadable sidecar must be repaired. Recording payloads are never rewritten.

A dry run reads and hashes the selected recordings using a temporary copy of
ledger state. It does not upload, change source files, or update the saved ledger.
Missing labels are checked against recovered metadata in memory too. If a preview
finds missing dates or labels, rerun the same upload command without `--dry-run`
from a terminal to answer the prompts. Running `attribute` immediately after a
dry run cannot resolve those findings: the preview has not saved them to the queue.

## Interruptions and retries

During a running upload, transient connection failures in file/part uploads and
the resume listing are retried with backoff for up to 15 minutes per request,
starting at its first connection failure. This includes TLS connections cut off
mid-write. Multipart retries keep chunks already sent in that running transfer.
The log shows the retry delay and remaining window. Certificate and permission
errors do not receive the extended retry window.

On an unstable connection, reduce simultaneous recording uploads with `--jobs 1`
(or set upload concurrency to 1 in the desktop app). Each recording still sends
up to eight chunks concurrently. If retries are exhausted, retry the same
delivery with the same cache/ledger; the desktop app preserves delivery membership
when you retry its pending queue.

Files of 16 MiB or larger now retain multipart upload checkpoints across failures
and app/process restarts. On retry, the CLI reads the source again, verifies the
size and SHA-256 of chunks already in S3, and sends only missing or mismatched
chunks. Reused chunks count toward progress without inflating transfer speed.
Completed files are also reused. Keep the original source files and ledger (the
desktop app uses `delivery.db` in its cache); no new command or option is needed.

This applies to uploads started with a version supporting multipart checkpoints.
Older uploads without checkpoints, partial uploads removed by S3 lifecycle
cleanup, and uploads whose completion succeeded but could not be confirmed
locally restart the affected file conservatively. A partial upload does not
confirm a delivery or make its source eligible for reclaim.

Run the same upload command again after a crash or network failure. An open
delivery is reused. Objects with matching sizes and completed-file checkpoints
are skipped; uncertain transfers are sent again. You can also resume the local queue without supplying paths:

```bash
gi-ingest upload
```

If a recording moves to another disk, supply its new path with its original
session directory name. Its content identity retains its upload state. To resume
an open delivery with explicit paths, include all of that delivery's recordings.
The CLI refuses to seal only part of an existing delivery.

A failed transfer leaves the delivery open and returns a nonzero exit code.
Recordings held for missing metadata or attribution also make the command return
nonzero, even when other recordings were delivered successfully. Read the named
failures; do not treat a partial delivery as permission to delete the whole input.
Changed files in an open delivery must be restored, or the delivery explicitly
abandoned before starting again:

```bash
gi-ingest abandon --batch <batch-id>
gi-ingest upload /Volumes/SSD1/recordings
```

Only one mutating CLI operation may use a ledger at a time. Use `--jobs` for
native concurrency, rather than multiple processes sharing a ledger.

## Reclaim only the paths you choose

**Keep at least one verified local copy until delivery is confirmed.**
`reclaim` is the supported way to enforce that rule. A transfer completing is
not enough: the server must have accepted the sealed delivery.

```bash
gi-ingest reclaim /Volumes/SSD1/recordings --dry-run
gi-ingest reclaim /Volumes/SSD1/recordings
```

Reclaim searches only the supplied paths. It compares each recording's part
inventory, sizes, SHA-256 hashes, and device metadata against a confirmed entry
in the local ledger before deleting its session directory. A second copy, such
as the original SD card, can be reclaimed by supplying that copy's path too.
Physical card identity or a particular mountpoint is not required.

Unconfirmed recordings, changed contents, missing proof, and directories with
unknown files are kept. There is no option to wipe undelivered or unusable
recordings through this command. Reclaim never purges another disk or every
saved source path as a side effect. There is no default scan of mounted drives.

Confirmation means the server accepted the raw delivery, not that downstream
processing or quality review has finished. The current server seal verifies
object presence and sizes; the CLI records SHA-256 values and uses them for
local content matching. Manually deleting the last local copy before confirmation
bypasses these safeguards and can lose data.

## Status and local state

```bash
gi-ingest status
gi-ingest history
gi-ingest history --batch <batch-id>
gi-ingest retry
gi-ingest config show
```

Authentication and defaults live in `~/.config/gi-ingest/config.json`; upload and
delivery records live in `~/.local/state/gi-ingest/ledger.db`. Keep the ledger:
without a matching confirmation record, reclaim preserves the files. Use
`--ledger PATH` on upload and reclaim to use another ledger, and use the same
ledger for both commands.

Existing ledgers and staged files remain usable. Upload their directories
directly, or run `upload` without paths to resume the old queue. `staged` remains
an internal state meaning ready to upload; it no longer implies a separate copy
step. The old `staging-dir` setting is unused by upload and reclaim.

`reset` and `drop` preserve files registered by path-based upload. Their legacy
cleanup behavior still applies to CLI-owned staging copies from older versions;
inspect their dry-run output before changing old queue entries.

## Encrypted recordings

VREC recordings retain their `ego_*.mcap` names and use the same
`gi-ingest upload <paths...>` workflow. Upload sends ciphertext unchanged and
reports that decryption and content checks run on the backend. No recording key
is needed locally. The ledger persists `source_encrypted`; older entries remain
unknown until rescanned.

If `session.json` is missing, the CLI warns and uploads the encrypted parts with
metadata recovery pending. DPS decrypts the MCAP, reconstructs the sidecar from
embedded capture metadata, and writes both to the episodes bucket. No placeholder
sidecar is created locally. An existing unreadable sidecar still requires repair.

Resume, dry-run, and scoped reclaim use the original ciphertext hashes.

## Development

```bash
uv sync --extra dev
uv run pytest -q
uv run ruff check gi_ingest tests
```


### Upload progress and ETA

`gi-ingest upload` displays aggregate byte progress, transfer speed, remaining
upload time, and completed/total files across all concurrent recording uploads.
Resumed objects count toward completion but do not inflate transfer speed.
ETA becomes unknown when transfers stall or S3 is still finalizing a file.
Redirected output (including `nohup`) gets a progress line about every 10 seconds
and a final summary.

The CLI sends a best-effort snapshot about every 10 seconds to
`PUT /api/v1/ingest/batches/{batch_id}/progress`, plus a final `uploaded`, `failed`,
or `interrupted` snapshot. Reporting uses a background thread and short request
timeouts; an unavailable or older backend does not stop the upload. A process
killed without cleanup leaves its last heartbeat behind.

Batch responses expose `upload_progress`: `total_bytes`, `uploaded_bytes`
(includes in-flight and resumed bytes), `resumed_bytes`, `total_files`,
`completed_files`, `bytes_per_second`, `eta_seconds`, `phase`, and the server's
`received_at`. Use the timestamp to detect stale reports. These values are
client-reported and independent of manifest-verified batch totals and lifecycle
status; `uploaded` means ready for sealing, not delivered. On resume the latest
snapshot replaces the previous attempt's progress. Multiple processes uploading
the same batch are not aggregated; the latest received snapshot wins.

Apply the `add_ingest_upload_progress` database migration before deploying the
backend. Existing clients continue to work and leave `upload_progress` null.

## Troubleshooting upload connections

Uploads automatically record structured diagnostics in `upload-diagnostics/`
beside the ledger. Each attempt logs a run ID and records stages, request failure
categories, executed retries, recovered requests, progress, and reused multipart
chunks. Reports also include standard and accelerated endpoint probe speeds in
bytes per second, whether the reading was fresh or cached, and its cache age.
Missing readings stay null. Current delivery speed is reported separately in
`progress.bytes_per_second`. The shared Windows engine records the same data.

Reports are sent automatically to the backend for platform engineering analysis;
vendors do not need to upload them manually. They are stored in the
`ingest_batches.upload_diagnostics` JSONB column, with up to ten runs per delivery.
There is no portal diagnostics UI or read endpoint. See the
[engineering query guide](../docs/developer/vendor-upload-diagnostics.md).
Run timestamps come from the vendor's clock; `received_at` is the server's clock.
A stale `running` report does not prove the client is still running. A forced
process kill can lose events since the previous heartbeat (normally 30 seconds).

Reports are saved locally before background delivery. A network outage retains
unsent reports, and subsequent heartbeats or upload attempts replay them using
the same server and upload-token login. Only 20 runs per login are kept locally;
very old reports can be evicted, including unsent ones. Switching tokens/server
isolates the outbox. Reports from failures before a batch ID exists stay local.
Validation and version checks before an upload attempt are not instrumented.

For optional support investigation while a machine remains offline, export to a
new filename:

```bash
gi-ingest diagnostics --output upload-diagnostics.json
# Include --ledger /path/to/ledger.db if uploads use a custom ledger.
```

This JSON contains batch/run IDs and structured statistics, not raw logs,
recording contents, paths, API keys, S3 credentials, request URLs/headers, IP
addresses, or Wi-Fi names. The file is safe for your support team to inspect;
share it through your normal support channel. No third-party telemetry service
receives these reports. DNS/connect, timeout, TLS, and server error categories
help narrow the investigation but cannot conclusively distinguish bad Wi-Fi from
an ISP or upstream network problem.

Server rollout: apply the `vendor_upload_diagnostics` migration, deploy the
backend, then release the updated CLI/desktop app. With an older backend,
the CLI keeps local reports and continues uploading normally. Diagnostic writes
are advisory, vendor-scoped, and accepted after sealing so delayed reports are
not discarded. They never change upload permissions or verified delivery totals.
