Metadata-Version: 2.4
Name: eden_grasp_gen
Version: 0.1.5
Summary: A fork of GraspGen for EDEN project
Home-page: https://github.com/NVlabs/GraspGen
Author: Kashu Yamazaki
Author-email: Kashu Yamazaki <kyamazak@andrew.cmu.edu>
License: NVIDIA License
        
        1. Definitions
        
        “Licensor” means any person or entity that distributes its Work.
        “Work” means (a) the original work of authorship made available under this license, which may include software, documentation, or other files, and (b) any additions to or derivative works  thereof  that are made available under this license.
        The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this license, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work.
        Works are “made available” under this license by including in or with the Work either (a) a copyright notice referencing the applicability of this license to the Work, or (b) a copy of this license.
        
        2. License Grant
        
        2.1 Copyright Grant. Subject to the terms and conditions of this license, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to use, reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form.
        
        3. Limitations
        
        3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this license, (b) you include a complete copy of this license with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work.
        
        3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this license (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself.
        
        3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use non-commercially. Notwithstanding the foregoing, NVIDIA Corporation and its affiliates may use the Work and any derivative works commercially. As used herein, “non-commercially” means for research or evaluation purposes only.
        
        3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this license from such Licensor (including the grant in Section 2.1) will terminate immediately.
        
        3.5 Trademarks. This license does not grant any rights to use any Licensor’s or its affiliates’ names, logos, or trademarks, except as necessary to reproduce the notices described in this license.
        
        3.6 Termination. If you violate any term of this license, then your rights under this license (including the grant in Section 2.1) will terminate immediately.
        
        4. Disclaimer of Warranty.
        
        THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF 
        MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE. 
        
        5. Limitation of Liability.
        
        EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
        
        
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE_ASSETS.md
Requires-Dist: h5py
Requires-Dist: hydra-core
Requires-Dist: matplotlib
Requires-Dist: meshcat
Requires-Dist: numpy==1.26.4
Requires-Dist: webdataset
Requires-Dist: scikit-learn
Requires-Dist: scipy
Requires-Dist: tensorboard
Requires-Dist: trimesh==4.5.3
Requires-Dist: transformers
Requires-Dist: tensordict
Requires-Dist: diffusers==0.11.1
Requires-Dist: timm==1.0.15
Requires-Dist: huggingface-hub==0.25.2
Requires-Dist: PyOpenGL==3.1.5
Requires-Dist: addict
Requires-Dist: spconv-cu126
Requires-Dist: yapf==0.40.1
Requires-Dist: tensorboardx
Requires-Dist: sharedarray
Requires-Dist: torch-geometric
Requires-Dist: yourdfpy==0.0.56
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# GraspGen: A Diffusion-based Framework for 6-DOF Grasping

> ⚠️ LICENSE WARNING: This project is a derivative of GraspGen and is licensed under the NVIDIA Source Code License. It is restricted to Non-Commercial Research and Evaluation purposes only. Commercial use is strictly prohibited.

## Installation
```bash
pip install eden_grasp_gen
```

NOTE: When compiling `pointnet2_ops`, if you are facing issues such as finding CUDA runtime headers or missing C++ compiler, try to manually set the following before installing:
```bash
export CC=/usr/bin/g++ && export CXX=/usr/bin/g++ && export CUDAHOSTCXX=/usr/bin/g++ && export TORCH_CUDA_ARCH_LIST="8.6"
```
## Download Checkpoints

The checkpoints can be downloaded from [HuggingFace](https://huggingface.co/adithyamurali/GraspGenModels):
```bash
grasp_gen download
```

## Inference Demos

We have added scripts for visualizing grasp predictions on real world point clouds using the models. The sample dataset is in the models repository in the `sample_data` folder. Please see the script args for use. For plotting just the topk grasps (used on the real robot, `k=100` by default) pass in the `--return_topk` flag. To visualize for different grippers, modify the `--gripper_config` argument.

### Prerequisites

1. **Dataset:** Please [download checkpoints](#download-checkpoints) first - this will be the `<path_to_models_repo>` below.
2. **MeshCat:** All the examples below are visualized on MeshCat in a browser. You can start a MeshCat server in a new terminal (in any environment, install with `pip install meshcat`) with the following command: `meshcat-server`. You can also just run a dedicated docker container in the background `bash docker/run_meshcat.sh`. Navigate to the corresponding url on the browser (it should be printed when you start the server) - the results will be visualized here.
3. **Docker:** The first argument is the path to where you have locally cloned the GraspGen repository (always required). Use `--models` flag for the models directory. These will be mounted at `/code` and `/models` paths inside the container respectively. 
```bash
# For inference only
bash docker/run.sh <path_to_graspgen_code> --models <path_to_models_repo>
```

### Predicting grasps for segmented object point clouds

```bash
cd /code/ && python scripts/demo_object_pc.py --sample_data_dir /models/sample_data/real_object_pc --gripper_config /models/checkpoints/graspgen_robotiq_2f_140.yml
```
<img src="fig/pc/1.png" width="240" height="200" title="objpc1"> <img src="fig/pc/2.png" width="240" height="200" title="objpc2"> <img src="fig/pc/3.png" width="240" height="200" title="objpc3"> <img src="fig/pc/4.png" width="200" height="200" title="objpc4"> <img src="fig/pc/5.png" width="240" height="200" title="objpc5"> <img src="fig/pc/6.png" width="200" height="200" title="objpc6">

### Predicting grasps for object meshes
```bash
cd /code/ && python scripts/demo_object_mesh.py --mesh_file /models/sample_data/meshes/box.obj --mesh_scale 1.0 --gripper_config /models/checkpoints/graspgen_robotiq_2f_140.yml
```
<img src="fig/meshes/1.png" width="240" height="200" title="objpc1"> <img src="fig/meshes/2.png" width="240" height="200" title="objpc2"> <img src="fig/meshes/3.png" width="240" height="200" title="objpc3">

### **[Advanced]** Predicting grasps for objects from scene point clouds
```bash
cd /code/ && python scripts/demo_scene_pc.py --sample_data_dir /models/sample_data/real_scene_pc --gripper_config /models/checkpoints/graspgen_robotiq_2f_140.yml
```
<img src="fig/pc/scene1.png" width="400" height="300" title="scenepc1"> <img src="fig/pc/scene2.png" width="400" height="300" title="scenepc2">

### **[Advanced]** Predicting grasps for objects from scene point clouds, with collision checking
If you would like to filter the inferred grasps based on collisions, use the `--filter_collisions` flag. This uses a simple point cloud based collision checker. On the real robot, we suggest using [NVBlox](https://github.com/NVlabs/nvblox_torch). The grasp is in collision if it is <span style="color:red">**red**</span>, and <span style="color:green">**green**</span> if collision free. See `scripts/demo_collision_free_grasps.py` if you want to pass in your own scene using the depth and segmentation image as commandline arguments.
```bash
cd /code/ && python scripts/demo_scene_pc.py --filter_collisions --sample_data_dir /models/sample_data/real_scene_pc --gripper_config /models/checkpoints/graspgen_franka_panda.yml
```
<img src="fig/pc/collision1.png" width="400" height="300" title="collision1"> <img src="fig/pc/collision2.png" width="400" height="300" title="collision2"> <img src="fig/pc/collision3.png" width="400" height="300" title="collision3"> <img src="fig/pc/collision4.png" width="400" height="300" title="collision4"> <img src="fig/pc/collision5.png" width="400" height="300" title="collision5">

<!-- An example of a grasp that is colliding (left) vs collision-free (right) is show below.
<!-- <img src="fig/pc/collision4.png" width="400" height="300" title="collision4"> <img src="fig/pc/collision5.png" width="400" height="300" title="collision5"> -->

<small>Note: At the time of release of this repo, the suction checkpoint was not trained with on-generator training, hence may not output the best grasp scores.</small>

## Dataset

There are two datasets to download:
1. **Grasp Dataset**: This can be cloned from [HuggingFace](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GraspGen). Where you clone this would be `<path_to_grasp_dataset>`.
```
git clone https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GraspGen
```
2. **Object Dataset**: We have included a [script](scripts/download_objects.py) for downloading the object dataset below. We recommend running this inside the docker container (the `simplify` arg would not work otherwise). You'll have to specify the directory to save the dataset `<path_to_object_dataset>`. We have only tested the training with simplified meshes (hence the `--simplify` arg), which was crucial to increasing rendering and simulation speed. This script may take a few hours to complete downloading and will take up a lot of CPU. If you are running inside a docker container, you'll need to mount a location to save this data.

First start the docker:
```bash
# For Dataset download only
mkdir -p <object_dataset>
bash docker/run.sh <path_to_graspgen_code> --grasp_dataset <path_to_grasp_dataset> --object_dataset <path_to_object_dataset>
```

```bash
cd /code && python scripts/download_objects.py --uuid_list /grasp_dataset/splits/franka_panda/ --output_dir /object_dataset --simplify
```

In total, we release over 57 million grasps, computed for a subset of 8515 objects from the [Objaverse XL](https://objaverse.allenai.org/) (LVIS) dataset. These grasps are specific to three grippers: Franka Panda, the Robotiq-2f-140 industrial gripper, and a single-contact suction gripper (30mm radius). 

<img src="fig/montage2.png" width="1000" height="500" title="readme2"> 

## Training with Existing Datasets

This section covers training on existing pre-generated datasets for the three grippers. For a more detailed tutorial on generating your own dataset as well as training a model, please see [TUTORIAL.md](tutorials/TUTORIAL.md)

### Prerequisites

1. **Dataset:** Please see the [Dataset](#dataset) section and download the grasp and object datasets first.
2. **Path setup**: You will need to find out the following paths for the next step
- `<path_to_graspgen_code>`: Local path to where the graspgen repo was cloned
- `<path_to_grasp_dataset>`: Local path to where grasp dataset was cloned.
- `<path_to_object_dataset>`: Local path to where object dataset was downloaded.
- `<path_to_results>`: Local path to where training logs and cache would be saved.
3. **Docker:** Start the docker container with the correct paths:
```bash
# For training only.
mkdir -p <path_to_results>
bash docker/run.sh <path_to_graspgen_code> --grasp_dataset <path_to_grasp_dataset> --object_dataset <path_to_object_dataset> --results <path_to_results>
```

See the training scripts in `runs/`. For each gripper, there are models to train separately - the generator (diffusion model) as well as the discriminator.

```bash
# Example usage for training the generator
cd /code && bash runs/train_graspgen_robotiq_2f_140_gen.sh

# Example usage for training the discriminator
cd /code && bash runs/train_graspgen_robotiq_2f_140_dis.sh
```

Things to note regarding training:
- The experiments in this paper were run with 8 X A100 machines. We have tested this on V100, A100, H100 and L40s
- **Dataset Caching:** Before starting the actual training, the script builds a cache of the dataset and saves to a hdf5 `.h5` file in the specified cache directory. Both caching and training is handled in the same `train_graspgen.py` script with same arguments. If a cache does not exist or is incomplete, the script will build the cache and will automatically continue with training once caching is complete. If the cache already exists, the script will immediately start the training.
- **On-Generator training:** On-Generator training is not released for the discriminator training (yet). It will be released when the data generation repo is released. This is needed for the best performance and scoring of the predicted grasps (see paper).

### Important Training Arguments
- `NGPU`: Set to the number of GPUs you have for training
- `LOG_DIR`: Specifies where the tensorboard logs, checkpoints and console logs are saved to
- `NWORKERS`: A rule of thumb is to set to a non-zero number, roughly `Number of CPU of Cores` / `Number of GPUs`
- `NUM_REDUNDANT_DATAPOINTS`: parameter controls the redundancy (of camera viewpoints) in cache building. The higher the number, the better the domain randomization and sim2real transfer. Default is 7. There will be a `OOM` error if it is too high.
- `debug` mode: To run the job on a single GPU job and 1 worker, you can set the arg `train.debug=True`

### Monitoring Training and Estimates:
- **Generator**: Grasp reconstruction error `reconstruction/error_trans_l2` on the validation set should converge to a few `cm`; this run will take at least 3K epochs to converge; on a 8 X A100 node, it takes about 40 hrs for 3K epochs
- **Discriminator**: Validation AP score should be > 0.8 and `bce_topk` loss should go down; this run will take at least 3K epochs to converge; on a 8 X A100 node, it takes about 90 hrs for 3K epochs

## Training & Data Generation (for new objects and grippers)

Please see [TUTORIAL.md](tutorials/TUTORIAL.md) for a detailed walkthrough of training a model from scratch, including grasp data generation. Currently, we include an example for suction grippers. See the [GraspDataGen](https://github.com/NVlabs/GraspDataGen) package for Isaac Lab based grasp data generation for pinch grippers.

## GraspGen Conventions

Please see the following files for documentation on the formats we adopted:
- Gripper configuration: [GRIPPER_DESCRIPTION.md](docs/GRIPPER_DESCRIPTION.md)
- Grasp Dataset format: [GRASP_DATASET_FORMAT.md](docs/GRASP_DATASET_FORMAT.md)

See the [GraspDataGen](https://github.com/NVlabs/GraspDataGen) package for Isaac Lab based grasp data generation in the above format.

## FAQ

### How do I train for a new gripper?

Please let us know what gripper you are interested in this [short survey](https://docs.google.com/forms/d/e/1FAIpQLSdTCstEtaeZz5iSyjAhYFuJqSpMF671ftPylkS3ZJFhRIg3dg/viewform?usp=dialog).

For optimal performance on a new gripper, we recommend re-training the model with our specified training recipie. You will need following to achieve that:

* Gripper URDF. See [assets/](assets/) for examples.
* Gripper description in the GraspGen format. See [GRIPPER_DESCRIPTION.md](docs/GRIPPER_DESCRIPTION.md).
* Object-Grasp dataset for this gripper consisting of successful and unsuccessful grasps. See [GRASP_DATASET_FORMAT.md](docs/GRASP_DATASET_FORMAT.md)

Please see [TUTORIAL.md](tutorials/TUTORIAL.md) for a detailed walkthrough of training a model from scratch, including grasp data generation. See [GraspDataGen](https://github.com/NVlabs/GraspDataGen) for data generation.

### My gripper is very similar to one of the existing grippers. Could I re-target model for my gripper?

In most cases, we recommend re-training a new model specific to your gripper as the physics would have changed.

If your gripper is antipodal and has a similar stroke length (i.e. width) to one of the existing grippers (Franka/Robotiq), feel free to re-target the model. You may have to apply a offset along the z direction `import trimesh.transformations as tra; new_grasp = grasp @ tra.translation_matrix([0,0,-Z_OFFSET])` to align the base link frames of both grippers.

If you are using a single-cup suction gripper, you could retarget our suction model trained for a 30mm suction seal.  You could rescale the object point cloud/mesh input before inference `import trimesh.transformations as tra; mat = tra.scale_matrix(r/0.030)` where `r` is the radius of the suction cup for your gripper.


### How do I finetune on new object dataset?

The graspgen model is meant to generalize zero-shot to unknown objects. If you would like to further finetune the model on a new object/grasp dataset combination or train on a larger dataset, you will need to 1) pass in the pretrained checkpoint in the `train.checkpoint` argument in the train script and 2) change the paths to the new grasp/object dataset. Please check the [GRASP_DATASET_FORMAT.md](docs/GRASP_DATASET_FORMAT.md) convention.

### Why is my train script hanging/getting killed without any errors?
Make sure your docker container has sufficient CPU, swap and GPU memory. Please post a github issue otherwise.

### How do I run this on the robot?

You will need instance segmentation (e.g. [SAM2](https://ai.meta.com/sam2/)) and motion planning (e.g. [cuRobo](https://curobo.org/)) to run this model. More details can be found in the experiments section of the paper.

### You did not include the gripper I have/want with your dataset!
Sorry we missed your gripper! Please consider completing this quick [survey](https://docs.google.com/forms/d/e/1FAIpQLSdTCstEtaeZz5iSyjAhYFuJqSpMF671ftPylkS3ZJFhRIg3dg/viewform?usp=dialog) to describe your gripper. You can optionally leave a your URDF.

### How do I report a bug or ask more detailed questions?
Please post a github issue and we will follow up! Or feel free to email us.


## License
License Copyright © 2025, NVIDIA Corporation & affiliates. All rights reserved.

For business inquiries, please submit the form [NVIDIA Research Licensing](https://www.nvidia.com/en-us/research/inquiries/).

## Citation

If you found this work to be useful, please considering citing:

```
@article{murali2025graspgen,
  title={GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training},
  author={Murali, Adithyavairavan and Sundaralingam, Balakumar and Chao, Yu-Wei and Yamada, Jun and Yuan, Wentao and Carlson, Mark and Ramos, Fabio and Birchfield, Stan and Fox, Dieter and Eppner, Clemens},
  journal={arXiv preprint arXiv:2507.13097},
  url={https://arxiv.org/abs/2507.13097},
  year={2025},
}
```
