Metadata-Version: 2.4
Name: kindergarden
Version: 0.0.2
Summary: A robotics benchmark for physical reasoning.
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: gymnasium
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: Pillow
Requires-Dist: shapely>=2.1.2
Requires-Dist: scipy>=1.14.0
Requires-Dist: relational_structs>=0.0.1
Requires-Dist: prpl_utils>=0.0.1
Provides-Extra: all
Requires-Dist: pygame>=2.5.2; extra == "all"
Requires-Dist: pymunk==7.1.0; extra == "all"
Requires-Dist: mujoco==3.2.4; extra == "all"
Requires-Dist: opencv-python>=4.9.0; extra == "all"
Requires-Dist: pybullet>=3.2.5; extra == "all"
Requires-Dist: tomsgeoms2d>=0.0.1; extra == "all"
Requires-Dist: pybullet_helpers>=0.0.1; extra == "all"
Provides-Extra: dynamic2d
Requires-Dist: pygame>=2.5.2; extra == "dynamic2d"
Requires-Dist: pymunk==7.1.0; extra == "dynamic2d"
Requires-Dist: tomsgeoms2d>=0.0.1; extra == "dynamic2d"
Provides-Extra: kinematic2d
Requires-Dist: tomsgeoms2d>=0.0.1; extra == "kinematic2d"
Provides-Extra: tidybot
Requires-Dist: mujoco==3.2.4; extra == "tidybot"
Requires-Dist: opencv-python>=4.9.0; extra == "tidybot"
Requires-Dist: gdown; extra == "tidybot"
Provides-Extra: kinematic3d
Requires-Dist: pybullet>=3.2.5; extra == "kinematic3d"
Requires-Dist: pybullet_helpers>=0.0.1; extra == "kinematic3d"
Provides-Extra: develop
Requires-Dist: pygame>=2.5.2; extra == "develop"
Requires-Dist: pymunk==7.1.0; extra == "develop"
Requires-Dist: mujoco==3.2.4; extra == "develop"
Requires-Dist: opencv-python>=4.9.0; extra == "develop"
Requires-Dist: pybullet>=3.2.5; extra == "develop"
Requires-Dist: tomsgeoms2d>=0.0.1; extra == "develop"
Requires-Dist: pybullet_helpers>=0.0.1; extra == "develop"
Requires-Dist: black[jupyter]; extra == "develop"
Requires-Dist: docformatter; extra == "develop"
Requires-Dist: isort; extra == "develop"
Requires-Dist: mypy; extra == "develop"
Requires-Dist: pylint>=2.14.5; extra == "develop"
Requires-Dist: pytest-pylint>=0.18.0; extra == "develop"
Requires-Dist: pytest>=7.2.2; extra == "develop"
Requires-Dist: coverage; extra == "develop"
Requires-Dist: gdown; extra == "develop"
Requires-Dist: scipy-stubs>=1.15.3.0; extra == "develop"
Requires-Dist: types-shapely>=2.1.0; extra == "develop"
Requires-Dist: nbmake; extra == "develop"
Requires-Dist: ipykernel; extra == "develop"
Dynamic: license-file

# KinDER

A **p**hysical **r**easoning **bench**mark for robotics.

There's growing excitement around large language models and their ability to "reason"—but reasoning isn't just about tokens and text. **Robots must reason too**: over long horizons, under uncertainty, and with sparse feedback. And unlike purely symbolic systems, **robotic reasoning is physical**: it's grounded in low-level, continuous state and action spaces. It requires understanding kinematics, geometry, dynamics, contact, force, tool use, and more.

This benchmark is designed for this kind of physical reasoning with robots. We invite researchers to try their best task and motion planning, reinforcement learning, imitation learning, and foundation models approaches. We hope that KinDER will bridge perspectives and foster shared progress toward physically intelligent robots.

### Environments

| Environment | Category | Example Environment ID |
|---|---|---|
| ClutteredRetrieval2D | Kinematic2D | `kinder/ClutteredRetrieval2D-o10-v0` |
| Motion2D | Kinematic2D | `kinder/Motion2D-p5-v0` |
| Obstruction2D | Kinematic2D | `kinder/Obstruction2D-o4-v0` |
| PushPullHook2D | Kinematic2D | `kinder/PushPullHook2D-v0` |
| ClutteredStorage2D | Kinematic2D | `kinder/ClutteredStorage2D-b15-v0` |
| StickButton2D | Kinematic2D | `kinder/StickButton2D-b10-v0` |
| DynObstruction2D | Dynamic2D | `kinder/DynObstruction2D-o3-v0` |
| DynPushPullHook2D | Dynamic2D | `kinder/DynPushPullHook2D-o5-v0` |
| DynPushT2D | Dynamic2D | `kinder/DynPushT2D-t1-v0` |
| DynScoopPour2D | Dynamic2D | `kinder/DynScoopPour2D-o50-v0` |
| Obstruction3D | Kinematic3D | `kinder/Obstruction3D-o4-v0` |
| Packing3D | Kinematic3D | `kinder/Packing3D-p3-v0` |
| Table3D | Kinematic3D | `kinder/Table3D-o3-v0` |
| Transport3D | Kinematic3D | `kinder/Transport3D-o2-v0` |
| BaseMotion3D | Kinematic3D | `kinder/BaseMotion3D-v0` |
| Shelf3D | Kinematic3D | `kinder/Shelf3D-o10-v0` |
| ConstrainedCupboard3D | Dynamic3D | `kinder/ConstrainedCupboard3D-o6-v0` |
| SortClutteredBlocks3D | Dynamic3D | `kinder/SortClutteredBlocks3D-o20-sort_the_cluttered_blocks_into_bowls-v0` |
| Rearrange3D | Dynamic3D | `kinder/Rearrange3D-o2-put_the_boxed_drink_and_the_can_next_to_the_bowl-v0` |
| SweepSimple3D | Dynamic3D | `kinder/SweepSimple3D-o50-sweep_the_blocks_to_the_left_side_of_the_kitchen_island-v0` |
| Dynamo3D | Dynamic3D | `kinder/Dynamo3D-o1-v0` |
| Tossing3D | Dynamic3D | `kinder/Tossing3D-o1-v0` |
| ScoopPour3D | Dynamic3D | `kinder/ScoopPour3D-o10-v0` |
| BalanceBeam3D | Dynamic3D | `kinder/BalanceBeam3D-o3-v0` |
| SweepIntoDrawer3D | Dynamic3D | `kinder/SweepIntoDrawer3D-o5-v0` |

## :zap: Usage Example

### Basic Usage (Gym API)

```python
import kinder
kinder.register_all_environments()
env = kinder.make("kinder/Obstruction2D-o3-v0")  # 3 obstructions
obs, info = env.reset()  # procedural generation
action = env.action_space.sample()
next_obs, reward, terminated, truncated, info = env.step(action)
img = env.render()  
```

### Object-Centric States

All environments in KinDER use object-centric states. For example:

```python
from kinder.envs.kinematic2d.obstruction2d import ObjectCentricObstruction2DEnv
env = ObjectCentricObstruction2DEnv(num_obstructions=3)
obs, _ = env.reset(seed=123)
print(obs.pretty_str())
```
Here, `obs` is an [ObjectCentricState](https://github.com/tomsilver/relational-structs/blob/main/src/relational_structs/object_centric_state.py#L25), and the printout is:
```
############################################################### STATE ###############################################################
type: crv_robot           x         y    theta    base_radius    arm_joint    arm_length    vacuum    gripper_height    gripper_width
-----------------  --------  --------  -------  -------------  -----------  ------------  --------  ----------------  ---------------
robot              0.885039  0.803795  -1.5708            0.1          0.1           0.2         0              0.07             0.01

type: rectangle           x         y    theta    static    color_r    color_g    color_b    z_order      width     height
-----------------  --------  --------  -------  --------  ---------  ---------  ---------  ---------  ---------  ---------
obstruction0       0.422462  0.100001        0         0       0.75        0.1        0.1        100  0.132224   0.0766399
obstruction1       0.804663  0.100001        0         0       0.75        0.1        0.1        100  0.0805652  0.0955062
obstruction2       0.559246  0.100001        0         0       0.75        0.1        0.1        100  0.12608    0.180172

type: target_block          x         y    theta    static    color_r    color_g    color_b    z_order     width    height
--------------------  -------  --------  -------  --------  ---------  ---------  ---------  ---------  --------  --------
target_block          1.20082  0.100001        0         0   0.501961          0   0.501961        100  0.138302  0.155183

type: target_surface           x    y    theta    static    color_r    color_g    color_b    z_order     width    height
----------------------  --------  ---  -------  --------  ---------  ---------  ---------  ---------  --------  --------
target_surface          0.499675    0        0         1   0.501961          0   0.501961        101  0.180286       0.1
#####################################################################################################################################
```

For compatibility with baselines, the observations provided by the main environments are vectors. It is easy to convert between vectors and object-centric states. For example:
```python
import kinder
kinder.register_all_environments()
env = kinder.make("kinder/Obstruction2D-o3-v0")
vec_obs, _ = env.reset(seed=123)
object_centric_obs = env.observation_space.devectorize(vec_obs)
recovered_vec_obs = env.observation_space.vectorize(object_centric_obs)
```

## :muscle: Challenges for Existing Approaches

What makes KinDER challenging?

### For Reinforcement Learning

Environments have long horizons and sparse rewards. Users are welcome to engineer dense rewards, but doing so may be nontrivial. Environments also have very diverse task distributions (as in the `reset()` function), so learned policies must generalize.

### For Imitation Learning

As with RL, generalization across tasks is a major challenge for imitation learning. Furthermore, we supply some demonstrations, but they are typically suboptimal, multimodal, and limited in quantity. Users are welcome to collect their own demonstrations.

### For Language Models

The physical reasoning required in KinDER is not easy to represent in natural language alone. Vision-language and vision-language-action models may fare better, but the tasks in KinDER are beyond the capabilities of current VLMs and VLAs.* (*This is an empirical claim that we will test!)

### For Hierarchical Approaches

Approaches that first decide "what to do" and then decide "how to do it" will run into difficulties in KinDER when there are couplings between these high-level and low-level decisions. For example, the exact grasp of an object may determine whether the object can later be placed into a tight space.

### For Task and Motion Planning

KinDER does not provide any models for TAMP. Users are welcome to engineer their own, but doing so may be nontrivial. Furthermore, some environments in KinDER are meant to strain the assumptions that are sometimes made in TAMP. Finally, some environments contain many objects, which may make planning slow even when models are available.

## :octocat: Contributing

### :ballot_box_with_check: Requirements
1. Python >=3.10, <3.13
2. Tested on MacOS Monterey and Ubuntu 22.04 (but we aim to support most platforms)

### :wrench: Installation
We strongly recommend [uv](https://docs.astral.sh/uv/getting-started/installation/). The steps below assume that you have `uv` installed. If you do not, just remove `uv` from the commands and the installation should still work.

Then, choose one of the following based on you need:
- `uv pip install -r optional_prpl_requirements/core.txt && uv pip install -e .` - Installs only core dependencies (matplotlib, numpy, relational_structs, prpl_utils)
- `uv pip install -r prpl_requirements.txt && uv pip install -e .uv pip install -e ".[all]"` - Installs everything (excluding develop)
- `uv pip install -r optional_prpl_requirements/kinematic2d.txt && uv pip install -e ".[kinematic2d]"` - Installs only core + kinematic2d dependencies (no pybullet)
- `uv pip install -r optional_prpl_requirements/dynamic2d.txt && uv pip install -e ".[dynamic2d]"` - Installs only core + dynamic2d dependencies
- `uv pip install -e ".[tidybot]"` - Installs only core + tidybot dependencies
- `uv pip install -r optional_prpl_requirements/kinematic3d.txt && uv pip install -e ".[kinematic3d]"` - Installs only core + kinematic3d dependencies
- `uv pip install -r prpl_requirements.txt && uv pip install -e ".[develop]"` - Installs all + development tools
- Compositionally install the dependencies like `[kinematic2d,kinematic3d]`

### :microscope: Check Installation
Run `./run_ci_checks.sh`. It should complete with all green successes.

### :mag: General Guidelines
* All checks must pass before code is merged (see `./run_ci_checks.sh`)
* All code goes through the pull request review process

### :new: Adding New Environments
Some new environment requests are in Issues. To add a new environment, please see the examples in `src/kinder/env`. Also consider:
* Environments are registered in `src/kinder/__init__.py`
* Each environment should have at least one demonstration (see `scripts/collect_demos.py`)
* After collecting a demonstraction, create a video with `scripts/generate_demo_video.py`, which will be used in the autogenerated documentation
