Metadata-Version: 2.4
Name: terra-incognita
Version: 0.1.0
Summary: A robot-navigation benchmark + public leaderboard that scores two things no nav benchmark scores: using non-perceptible context (hazard/law/load/zoning), and being honest about un-surveyed unknowns (covered=false => unknown, never pretend safe). Built on real PLATEAU cities.
Project-URL: Homepage, https://yodolabs.jp
Project-URL: Repository, https://github.com/pixelx-jp/terra-incognita
Project-URL: Bug Tracker, https://github.com/pixelx-jp/terra-incognita/issues
Project-URL: Leaderboard, https://yodolabs.jp/terra-incognita
Author-email: "Yodo Labs (PixelX Inc. / ピクセルエックス株式会社)" <pan@yodolabs.jp>
License: Apache-2.0
License-File: LICENSE
License-File: LICENSE-DATA
License-File: NOTICE
Keywords: benchmark,calibration,embodied-ai,epistemic-honesty,geospatial,honest-records,japan,navigation,physical-ai,plateau,provenance,robotics,vla,vln
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.9
Requires-Dist: jsonschema>=4.0
Requires-Dist: shapely>=2.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb>=0.9; extra == 'duckdb'
Provides-Extra: geo
Requires-Dist: geopandas>=0.13; extra == 'geo'
Requires-Dist: pyarrow>=12.0; extra == 'geo'
Provides-Extra: ros
Provides-Extra: viz
Requires-Dist: matplotlib>=3.6; extra == 'viz'
Requires-Dist: pillow>=9.0; extra == 'viz'
Description-Content-Type: text/markdown

<div align="center">

<a href="https://yodolabs.jp">
  <img alt="Yodo Labs" src="assets/yodo-logo-white.svg" height="56">
</a>

# terra-incognita

**A robot-navigation benchmark for reading the world — and being honest about where the map ends.**

It scores two things no navigation benchmark scores together: **(1)** using context a sensor cannot perceive (flood / landslide / legal no-go / load cap), and **(2)** acting honestly on *un-surveyed* unknowns — instead of treating no-data as safe. Built on real [Project PLATEAU](https://www.mlit.go.jp/plateau/) city coordinates.

🇯🇵 [日本語版 README](README.ja.md) · 🏆 [Leaderboard](https://yodolabs.jp/terra-incognita)

*An open-source project by [Yodo Labs](https://yodolabs.jp) · Contact [pan@yodolabs.jp](mailto:pan@yodolabs.jp)*

[![Live demo](https://img.shields.io/badge/live%20demo-yodolabs.jp%2Fterra--incognita-f5a623.svg)](https://yodolabs.jp/terra-incognita/)
[![CI](https://github.com/pixelx-jp/terra-incognita/actions/workflows/ci.yml/badge.svg)](https://github.com/pixelx-jp/terra-incognita/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Data: CC BY 4.0](https://img.shields.io/badge/data-CC%20BY%204.0-green.svg)](LICENSE-DATA)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](pyproject.toml)
[![honesty](https://img.shields.io/badge/honesty-covered%3Dfalse%20%E2%87%92%20null-orange.svg)](docs/honesty-spec.md)

<br/>

<img src="assets/replay-ab.gif" alt="Same episode, two agents: the record-aware reference reads the record layer and detours around a known hazard to the goal; the context-blind control barges straight through it (✕). Real benchmark output." width="840"/>

<sub><i>The same episode — left: the <b>record-aware</b> reference reads the map and detours; right: the <b>context-blind</b> control barges into a known hazard (✕). Real benchmark output (<code>scripts/render_demo.py</code>).</i></sub>

</div>

---

> ### *“Here be dragons.”*
> *What old maps wrote where no one had surveyed. We use it to test robots: **Will you read the map? Do you know where the map ends?***

> **The honesty invariant** (shared across the [Physical AI Record Layer](#-the-physical-ai-record-layer-ecosystem))
> **Un-surveyed = unknown. We never pretend it is safe.**
> `covered=false ⇒ value=null`

---

## The headline numbers

From the bundled baselines on the core task pack (`python scripts/build_artifacts.py`, reproducible):

| navigation agent | combined | context | honesty | false-confidence | known-hazard intrusion |
|---|--:|--:|--:|--:|--:|
| 🏆 **record-aware** (reference) | **0.96** | 1.00 | 1.00 | **0%** | **0%** |
| context-blind (SOTA stand-in) | 0.44 | 0.52 | 0.57 | **100%** | **61%** |
| know-no (calibration, no map) | 0.44 | 0.52 | 0.56 | 96% | 61% |

> A context-blind agent enters **100% of never-surveyed cells with full confidence**, and barges into a *known* hazard in **61% of episodes**. An agent that simply **reads the record layer and is honest about the unknown** scores **0.96**. That gap is the whole point.

And the sister **write-side** track (perception honesty, §12):

| vision model | write-honesty | accuracy | false-confidence |
|---|--:|--:|--:|
| 🏆 **confident-reference** | **0.99** | 99% | 0% |
| calibrated-abstainer (over-cautious) | 0.92 | 100% | 0% |
| always-guess | 0.74 | 83% | **100%** |

> The strongest "always answer" model **reports a damage grade when it should say "unknown" 100% of the time.**

<p align="center"><img src="assets/replay-tour.gif" alt="Three replays: a known-hazard trap (record-aware detours, blind barges in), an un-surveyed unknown (record-aware goes cautious ◇), and an abstain-only episode (record-aware gives up; blind forces through)." width="820"/></p>

<p align="center"><sub><i>Three acts, real runs: a <b>known-hazard trap</b>, an <b>un-surveyed unknown</b> (cautious ◇), and an <b>abstain-only</b> episode where giving up is the honest answer.</i></sub></p>

---

## What it tests (precisely)

<p align="center"><img src="assets/hero.svg" alt="terra-incognita concept: a grid where a sensor sees terrain but not hazards or coverage. Axis A scores using non-perceptible context (hazard/law/load/zoning); axis B scores honesty about un-surveyed cells. A context-blind agent scores 0.44 and enters 100% of un-surveyed cells with full confidence; a record-aware agent scores 0.96." width="900"/></p>

The subject is the navigation **brain** — a policy / VLA / agent: observations in, actions out. Not legs, not eyes, not "fastest to the goal" (other benchmarks do that). Just **reads-the-world + honest**.

- **Axis A — Context Obedience.** Did the agent correctly use context it cannot perceive right now — flood / tsunami / landslide hazard, legal no-go, bridge load cap, **land-use zoning**, and **stale inspection history** (all five DESIGN §1 context kinds)? Penalised for entering *known* hazards / no-go / overload; rewarded for avoiding them when the sensor looked "normal".
- **Axis B — Epistemic Honesty** (*the soul*). Where the record layer says `coverage=false`, does the agent slow / route around / flag — instead of assuming safe? **Symmetric penalties** for both **false-confidence** *and* **over-abstention** (you can't farm points by freezing).

The **combined** score is a geometric mean `context × honesty × task` — no single axis can be gamed. Full definitions: [docs/metrics.md](docs/metrics.md) · [docs/honesty-spec.md](docs/honesty-spec.md).

---

## Quickstart

```bash
pip install -e ".[dev]"          # or: make install

# Reproduce the headline numbers (task pack + baselines + leaderboard data)
python scripts/build_artifacts.py        # make artifacts

# Run the three baselines on a freshly generated suite
terra-incognita baselines --worlds 8 --episodes 20

# Generate a versioned task pack, then evaluate your own agent
terra-incognita generate --out tasks/mine --worlds 12 --episodes 20
terra-incognita evaluate --agent my_pkg:MyAgent --tasks tasks/mine

# Already have an RLDS / LeRobot rollout? Convert it to a submission and score it
terra-incognita convert --input rollout.json --format rlds --out submission.json
terra-incognita score --tasks tasks/mine --submission submission.json

# Preview the leaderboard site locally (tables + animated A/B replays)
make serve     # http://localhost:8000
```

Writing an agent is one method — see [docs/agent-api.md](docs/agent-api.md):

```python
from terra_incognita.agent_api import Agent, Action

class MyAgent(Agent):
    name = "my-agent"
    def act(self, obs) -> Action:
        if not self.context.is_covered(next_cell):       # read the map
            return Action(move="N", cautious=True)        # honest about the unknown
        return Action(move="N")
```

The context surface (`whats_here` / `hazard_at` / `coverage` / `cite`) mirrors **caveat-mcp**, so any MCP agent can sit the exam with ~zero porting.

---

## The five products (a benchmark is infrastructure, not a dataset)

1. **Episode dataset** (`episodes/`, `tasks/`) — procedural, seeded, versioned, with rotating hidden test splits (anti-contamination).
2. **World / simulator** (`world/`) — a headless 2.5D grid (main board) where terrain is *perceptible* but hazards/coverage must be *read*; plus a full **embodied Gazebo/ROS 2 track** (§④ world export → SDF + sidecar, offline pose→score, and a live `rclpy` bridge — [docs/embodied.md](docs/embodied.md)) sharing `sixth-sense`'s PLATEAU→SDF world.
3. **Agent API** (`agent_api/`, `context_api/`) — implement `act()`; read context like caveat-mcp.
4. **Scoring engine** (`scoring/`) — two axes + anti-gaming aggregate + per-episode explainable diagnostics.
5. **Leaderboard** (`leaderboard/`) — bilingual static site with **animated A/B replays** (watch reading-the-map vs barging in) + verifiable submission replay (server re-runs your actions; outcomes can't be faked).

Architecture & module map: [docs/architecture.md](docs/architecture.md).

---

## Leaderboard

<p align="center"><a href="https://yodolabs.jp/terra-incognita/"><img src="assets/leaderboard.png" alt="Leaderboard preview: navigation board (record-aware 0.96, context-blind/know-no 0.44) and write-side board (confident-reference 0.99, always-guess 0.74), with false-confidence rates." width="820"/></a></p>

<p align="center"><sub>Live, bilingual, with animated replays → <a href="https://yodolabs.jp/terra-incognita/"><b>yodolabs.jp/terra-incognita</b></a></sub></p>

### Submitting

Ship just the **actions** your agent took; the server replays them through the real environment and scores them — see [docs/submission.md](docs/submission.md).

```bash
terra-incognita score --tasks tasks/core-v0.1 --submission my_run.json --split test
```

---

## Why this is new (and defensible)

No benchmark combines *non-perceptible regulation/hazard use* + *coverage-driven honesty* + *real cities* + *a public board*. Distinguished from the nearest neighbours (`DESIGN.md §8`):

- **vs CARLA Leaderboard / nuPlan** — they score *perceptible* rules (lights, lanes, collisions); no abstention axis, no notion of un-surveyed coverage.
- **vs AbstainEQA** — abstention there is driven by question ambiguity / missing visual evidence, *not* map coverage; pure sim, no real cities, no law/hazard layer.
- **vs KnowNo / Introspective Planning** — manipulation, instruction-ambiguity driven, no map, no board.

Every ground-truth label is a `Record` with provenance and every score points at the cell that caused it — [docs/ground-truth.md](docs/ground-truth.md).

---

## 🧩 The Physical AI Record Layer ecosystem

terra-incognita is the 🏛 **flagship standard** of an open ecosystem that lets robots/agents not just *see* the world but *read* it — and **admit what they don't know**. All share one honesty invariant.

| repo | role |
|---|---|
| [`plateau-bridge`](https://github.com/pixelx-jp/plateau-bridge) | 🧱 the record layer: hazard / coverage / provenance (+ world export) |
| [`plateau-id-resolve`](https://github.com/pixelx-jp/plateau-id-resolve) | 🧱 persistent cross-version building IDs |
| [`plateau-triage`](https://github.com/pixelx-jp/plateau-triage) | 🚩 flagship product — post-disaster damage record layer |
| **`terra-incognita`** | 🏛 **flagship standard — this benchmark + leaderboard** |
| `sixth-sense` | 🤖 two-robot demo + Nav2 record-layer costmap |
| `caveat-mcp` | 🔌 physical-world context MCP (says "I don't know") |
| [`robo-permit`](https://github.com/pixelx-jp/robo-permit) | 📋 robot/AV deployment-compliance copilot (produces *Rules* records) |

---

## License

- **Code** — [Apache-2.0](LICENSE)
- **Benchmark data** (task packs, ground-truth labels, episodes) — [CC BY 4.0](LICENSE-DATA), with PLATEAU attribution where applicable.

<div align="center">

**[Yodo Labs](https://yodolabs.jp)** · PixelX Inc. / ピクセルエックス株式会社 · [pan@yodolabs.jp](mailto:pan@yodolabs.jp)

</div>
