Metadata-Version: 2.5
Name: gi-ingest
Version: 0.2.0
Summary: Offload robotics recordings from TF cards and deliver them to GILabs
License: Proprietary
Requires-Python: >=3.11
Requires-Dist: boto3>=1.34
Requires-Dist: httpx>=0.27
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# gi-ingest

Offload robotics recordings from TF cards and deliver them to GILabs.

## Install

**Not on PyPI yet** — there is no release workflow for this package (
`.github/workflows/cli-release.yml` publishes `gidata` only). Install from a
checkout:

```bash
git clone https://github.com/General-Intelligence-Labs/gilabs-data-center.git
uv tool install ./gilabs-data-center/vendor-cli    # or: pipx install ./…/vendor-cli
gi-ingest --help
```

Needs Python ≥ 3.11. Both commands put `gi-ingest` on your PATH in its own
isolated environment; `uv tool install --force …` upgrades in place after a
`git pull`.

Once the package is published this becomes `uv tool install gi-ingest` (or
`pipx install gi-ingest`) with no path — until then that command fails with
"No matching distribution found", which means it is not published, not that
anything is wrong with your setup.

Then authenticate:

```bash
gi-ingest login --api-key gik_...
gi-ingest doctor      # verifies connectivity, clock skew, disk, credentials
```

Get your token from the vendor portal at
[vendor.gilabs.xyz](https://vendor.gilabs.xyz) → **Upload tokens**.

## The three phases

They are three commands because they run on two different timescales, and fusing
them turns a 30-minute attended task into an all-night one.

| Phase | When | What it does |
|---|---|---|
| `stage` | attended, minutes per card | Copies cards to a local disk and verifies each copy |
| `upload` | unattended, hours | Pushes everything staged as **one delivery**, overnight |
| `reclaim` | attended, next morning | Wipes only the cards the server confirmed |

```bash
# 1. Drain every card. Swap cards as each finishes; nothing is wiped yet.
gi-ingest stage --device umi_gripper
gi-ingest stage --watch                 # or let it auto-copy on insert

# 2. Leave this running. One delivery, resumable, safe to nohup.
gi-ingest upload --max-bandwidth 50M

# 3. Next morning, with the cards back in the reader.
gi-ingest reclaim
```

### Why the wipe is a separate step

`upload` finishes hours after the cards were pulled, so it cannot wipe them —
they are back in the rig. `stage` writes a `.staged.json` marker and leaves the
card intact; `reclaim` deletes part files only for episodes the server reports as
delivered, and writes a tombstone so a re-scan skips them.

**Between `stage` and `reclaim`, the staging disk holds the only copy.** Size it
for a full collection round; `gi-ingest doctor` warns under 50 GiB free.

### Choosing where cards are copied to

By default cards land in `~/gilabs-staging`. That is rarely where you want a
multi-terabyte collection round, so it is settable two ways:

```bash
# Persist it — do this once per machine.
gi-ingest config set staging-dir /Volumes/BigDisk/gilabs
gi-ingest config show

# Or override for a single run, without changing the stored setting.
gi-ingest stage --staging /Volumes/OtherDisk/gilabs
```

`config set` also accepts `ledger-path`, `api-url` and `project-id`. Commands
that read the ledger take `--ledger` for a one-off override, which is how you
keep two independent offload runs from sharing state on one machine.

## Other commands

```bash
gi-ingest whoami          # vendor, scopes, and where you may deliver
gi-ingest destinations    # the org/project list, with ids for --project
gi-ingest doctor          # token, connectivity, disk — run before a night
gi-ingest queue           # what is staged / in flight / delivered
gi-ingest status          # deliveries and their QC verdicts
gi-ingest status --batch btch_...
gi-ingest retry           # re-queue anything that failed
```

## How a delivery works

1. `upload` opens a **batch** and receives AWS credentials scoped to that batch's
   folder alone — they cannot read or write anything else, including your own
   other deliveries.
2. Files go straight to S3. Credentials expire hourly and are re-minted
   automatically, so a multi-hour run and a laptop that sleeps both just work.
3. `upload` **seals** the batch with a manifest listing every episode, part, size
   and SHA-256. GILabs verifies every declared file is present at its declared
   size before ingesting anything, so a half-finished upload can never be
   processed as if it were complete.

Interrupted? Re-run `upload`. It re-uses the same open delivery and skips files
already sent — a restart costs a listing, not the bytes.

## Choosing a destination

Every delivery is filed against one project. With a single granted destination,
or one marked default, `upload` needs no flag. With several and no default it
will refuse to guess:

```bash
gi-ingest destinations
gi-ingest upload --project proj_abc123
```

## Duplicates

`stage` asks GILabs which episodes it already has before copying, so a card a
colleague already delivered costs one request instead of its bytes. If you are
offline it stages anyway and the duplicate is caught server-side at seal.

## Local development

```bash
uv venv && uv pip install -e ".[dev]"
uv run pytest
uv run ruff check gi_ingest
```

Design doc: [`docs/product-plans/vendor-data-ingestion.md`](../docs/product-plans/vendor-data-ingestion.md).
