Agile and Dynamic Interactions for Mobile Manipulation

This research project is focused on the development of robots that can safely and efficiently work alongside people in crowded spaces, such as warehouses, homes, hospitals, and stores. Working in crowded spaces means that the robot comes into direct physical contact with humans and objects, such as doors, shelves, and wheelchairs. Safety is therefore of paramount importance, and the robot needs to give way to humans while also helping them. The robot’s interactions with people and objects are often dynamic in nature. For example, a robot may need to physically support a stumbling person while guiding them by the hand, or it may need to use the weight of its own body to open a heavy door. Dynamic interactions like these are beyond the current capabilities of robots. The project uses a new type of robot that balances on a single ball while being able to safely interact with people. The project performs fundamental research into dynamic interactions, by humans and robots, for accomplishing challenging tasks in an efficient, robust, and safe manner. The project provides a deeper understanding of how humans interact with their surroundings, which can also be used in the future for improving ergonomic designs and prosthetic arms. The research team is also developing new algorithms, designs, and controllers for collaborative and assistive robots, which have the potential for widespread societal benefits.

Robots capable of safe dynamic interactions, with continuous adaptation and fast reactions, represents a paradigm shift from current  approaches. The research provides an end-to-end study of full-body dynamic interaction tasks. The research contributes a series of human subject studies that will provide knowledge of human dynamic interactions. It also develops new dynamic interaction models and novel methods for planning, machine learning, and real-time control of dynamic interactions. To make this research possible, the research team will be utilizing a unique dynamically stable and agile mobile manipulator: the Carnegie Mellon University ballbot. The person-size ballbot locomotes on a single ball wheel, providing omnidirectional motion with intrinsic compliance, and is equipped with two human-scale 7-degree-of-freedom arms with hands.  Research contributions to dynamic interaction, planning, learning, and control are demonstrated on this robot platform and rigorously evaluated through a diverse set of integrative tasks, including cooperative carrying, maneuvering a manual wheelchair, cooperative task teaching, sit-to-stand maneuvers, and dynamic navigation in cluttered spaces.

Project Groups:


  • Development of a Humanoid Dual Arm System for a Single Spherical Wheeled Balancing Mobile Robot; Roberto Shu and Ralph Hollis (Humanoids), 2019
  • Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap; Jacky Liang, Saumya Saxena and Oliver Kroemer (RSS), 2020
  • The Elliott and Connolly Benchmark: A Test for Evaluating the In-Hand Dexterity of Robot Hands; Ryan Coulson, Chao Li, Carmel Majidi and Nancy Pollard  (Humanoids) 2020
  • Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation; Mohit Sharma, Jacky Liang, Jialiang Zhao, Alex LaGrassa, Oliver Kroemer (CoRL) 2020
  • Contact Tracing: A Low Cost Reconstruction Framework for Surface Contact Interpolation; Arjun Lakshmipathy; Dominik Bauer; Nancy S. Pollard (IROS) 2021
  • Learning Model Preconditions for Planning with Multiple Models; Alex LaGrassa, Oliver Kroemer (CoRL), 2021
  • Relational Learning for Skill Preconditions; Mohit Sharma, Oliver Kroemer (CoRL) 2021
  • Learning Reactive and Predictive Differentiable Controllers for Switching Linear Dynamical Models; Saumya Saxena, Alex LaGrassa, Oliver Kroemer (ICRA) 2021
  • Generalizing Object-Centric Task-Axes Controllers using Keypoints;  Mohit Sharma and Oliver Kroemer (ICRA) 2021
  • Momentum based Whole-Body Optimal Planning for a Single-Spherical-Wheeled Balancing Mobile Manipulator; Roberto Shu, Ralph Hollis (IROS) 2021
  • Characterizing Continuous Manipulation Families for Dexterous Soft Robot Hands;  Jiatian Sun, Jonathan P. King, Nancy S. Pollard (Frontiers in Robotics and AI) 2021
  • Visual Identification of Articulated Object Parts; Vicky Zeng, Timothy E. Lee, Jacky Liang, Oliver Kroemer (IROS) 2021
  • Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation; Jacky Liang, Mohit Sharma, Alex LaGrassa, Shivam Vats, Saumya Saxena, Oliver Kroemer (ICRA) 2022
  • An Agile and Dexterous Balancing Mobile Manipulator Robot; Shu, Roberto (CMU Thesis), 2022
  • Contact Transfer: A Direct, User-Driven Method for Human to Robot Transfer of Grasps and Manipulations; Arjun Lakshmipathy, Dominik Bauer, Cornelia Bauer, Nancy S. Pollard (ICRA), 2022
  • Learning to Navigate by Pushing; Cornelia Bauer, Dominik Bauer (ICRA) 2022
  • Towards Very Low-Cost Iterative Prototyping for Fully Printable Dexterous Soft Robotic Hands; Dominik Bauer, Cornelia Bauer, Arjun Lakshmipathy, Roberto Shu, Nancy S. Pollard (RoboSoft) 2022
  • Dynamic Inference on Graphs using Structured Transition Models; Saumya Saxena, Oliver Kroemer (IROS) 2022