More Works from Our Lab

Extending Task and Motion Planning with Feasibility Prediction: Towards Multi-Robot Manipulation Planning of Realistic Objects

We introduce a multi-robot TAMP algorithm that leverages feasibility prediction to address complex manipulation tasks. Our approach extends previous methods by enabling the handling of mesh-shaped objects and collaborative multi-robot settings, demonstrating significant improvements over baseline methods.

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)
Simultaneous Action and Grasp Feasibility Prediction for Task and Motion Planning through Multi-Task Learning

We propose AGFP-Net, a multi-task neural network that predicts action feasibility and grasp type feasibility, significantly improving task and motion planning (TAMP) by reducing geometric planning time and solving more complex problems.

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)
Learning to Predict Action Feasibility for Task and Motion Planning in 3D Environments

We introduce a novel approach for Task and Motion Planning (TAMP) in 3D environments, combining an efficient 3D scene representation with a deep neural network to predict action feasibility, significantly reducing geometric planning time by up to 90% on complex problems.

2023 IEEE International Conference on Robotics and Automation (ICRA 2023)

Learning Geometric Reasoning Networks for Robot Task and Motion Planning

Smail Ait Bouhsain, Rachid Alami, Thierry Siméon
LAAS-CNRS
The Thirteenth International Conference on Learning Representations (ICLR 2025)
Paper Poster Code HAL 🤗 Models 🤗 Datasets
Teaser Image

Our method tackles the computational challenges of Task and Motion Planning (TAMP) by introducing Geometric Reasoning Networks (GRN), a graph neural network-based model for action and grasp feasibility prediction. By reducing reliance on geometric planners, GRN accelerates planning in complex 3D environments. Additionally, the model incorporates interpretability mechanisms, such as inverse kinematics (IK) feasibility prediction and grasp obstruction (GO) estimation, to enhance prediction accuracy and provide insights into infeasibility. This approach improves efficiency, generalizability, and explainability, enabling effective planning in diverse object shapes, multi-robot systems, and real-world scenarios.

Abstract

Task and Motion Planning (TAMP) is a computationally challenging robotics problem due to the tight coupling of discrete symbolic planning and continuous geometric planning of robot motions. In particular, planning manipulation tasks in complex 3D environments leads to a large number of costly geometric planner queries to verify the feasibility of considered actions and plan their motions. To address this issue, we propose Geometric Reasoning Networks (GRN), a graph neural network (GNN)-based model for action and grasp feasibility prediction, designed to significantly reduce the dependency on the geometric planner. Moreover, we introduce two key interpretability mechanisms: inverse kinematics (IK) feasibility prediction and grasp obstruction (GO) estimation. These modules not only improve feasibility predictions accuracy, but also explain why certain actions or grasps are infeasible, thus allowing a more efficient search for a feasible solution. Through extensive experimental results, we show that our model outperforms state-of-the-art methods, while maintaining generalizability to more complex environments, diverse object shapes, multi-robot settings, and real-world robots.

Proposed Model

Diagram of Geometric Reasoning Networks (GRN)

The Geometric Reasoning Networks (GRN) architecture. A scene graph is constructed from the input 3D environment. Node features of movable objects are given to IK feasibility module which outputs are used to update self-loop edge features. The concatenated features of nodes linked through a proximity edge are fed to the GO module to get grasp obstruction estimations, which are appended to proximity edge features. Finally, the updated graph is given to the AGF module to predict the action and grasp types feasibility for each movable object in the environment.

🤗 Models

Datasets

Visualization of the datasets used for training and evaluation

The datasets used for training and evaluation are fully synthetic and generated to simulate diverse 3D environments. Each dataset is characterized by the number of movable objects, structures (e.g., racks, counters, baskets), obstacles, and the robot used for annotation. For instance, the Panda-3D-4 dataset includes 3D environments with 4 movable objects, up to 4 structures, and up to 4 obstacles, annotated using a Panda robot. Similarly, the Panda-Tabletop-4 dataset focuses on tabletop environments with 4 movable objects and up to 4 obstacles, designed for comparison with tabletop methods. The PR2-3D-4 dataset demonstrates applicability to the PR2 robot, using parameters similar to Panda-3D-4. Additional test sets, such as Panda-3D-10, Panda-3D-15, and Panda-3D-20, evaluate generalizability to environments with higher object counts and increased complexity.

🤗 Datasets

Results

Combined Results

Comparison with SOTA methods trained and tested on different datasets. For grasp types feasibility prediction, the mean (+/- standard deviation) of F1 scores of the 5 grasp types are reported.

Combined Results

Evaluation of the generalizability to 3D environments with a higher number of objects compared to SOTA methods, when trained on the Panda-3D-4 dataset.

Real Robot Experiment Results

Ablation Study on the Panda-3D-4 dataset. For each task related to grasp types, we report the mean ($\pm$ standard deviation) across all grasp types.

Real Robot Experiment Results

Comparison of the number of parameters and inference time on a 3D environment with 4 movable objects and 15 fixed objects (4 queries).

Visualizations

Visualization of GRN predictions on the Access Problem

Visualization of GRN predictions on the initial state of a larger version of the Access problem. The goal is to have the robot grasp the pink object, which is initially blocked by many objects.

Visualization of GRN predictions on the Handover Problem

Visualization of GRN predictions on the initial state of the Handover problem. The goal is to perform a handover through a wall, with only three openings available for passing the object.

Qualitative results of GRN predictions
Qualitative results of GRN predictions

Application to TAMP Problems

Comparison of TAMP results using GRN and a geometric planner

Performance of GRN planner compared to a non-informed planner on the Access and Clutter problems. Results are average over 10 runs on 10 different instances of each problem.

Visualization of the solution to the Access Problem

Access Problem

Visualization of the solution to the Clutter Problem

Clutter Problem

BibTeX

          
            @inproceedings{ait2025learning,
              title={Learning Geometric Reasoning Networks for Robot Task and Motion Planning},
              author={Ait Bouhsain, Smail and Alami, Rachid and Simeon, Thierry},
              booktitle={The Thirteenth International Conference on Learning Representations (ICLR 2025)},
              year={2025}
            }