Towards human-level deep robotic cloth manipulation using vision-based methods with implicit representations

Abstract

This PhD study investigates the application of learning-based neural controllers to cloth-shaping tasks, including flattening and folding fabrics, T-shirts, and other garment types. We constrain our problem to a single-gripper Pick-and-Place (PnP) action output, using a top-down RGB-D camera as the observation input. We explore both Deep Reinforcement Learning (DRL) and Deep Imitation Learning (DIL) algorithms that learn implicit latent representations and do not require explicit intermediate feature extraction.

In general, we developed the proposed controllers in this PhD study initially in simulation, and subsequently proposed a series of Simulation-to-Reality (Sim2Real) techniques for real-world deployment. We first investigated DRL algorithms for square-fabric flattening in simulation. We then explored DIL methods to perform towel folding from both flattened and crumpled initial states. Following this, we compared and contrasted all the developed controllers from this PhD in the real world on two different robotic platforms using our proposed real-world deployment framework for cloth manipulation. Lastly, we extended our study to more complex garment types, such as T-shirts, trousers, skirts, and dresses. We successfully demonstrate that a single goal-conditioned model-based Deep Reinforcement Learning (DRL) method can flatten all four garments in simulation, and it can flatten a long-sleeve T-shirt in our real-robot setup. In addition, we developed a general Python framework for benchmarking various types of control algorithms on many different simulation and real-world environments to facilitate our research.

We advance the State-of-The-Art (SoTA) performance of vision-based neural controllers in the investigated domain. Our research outcomes demonstrate that action diversity, pick-position precision, specialised reward engineering, and high-quality expert demonstrations are keys to the success of these general data-driven controllers. However, our research also indicates that robotic control of cloth-shaping still cannot reach human-level manipulation capabilities, which continues to support the Moravec paradox in robotic control. In the future, we intend to integrate implicit topological structure learning for improved performance and to extend these methods to other action primitives.

Date of Award	2 Dec 2025
Original language	English
Awarding Institution	University of St Andrews
Supervisor	Kasim Terzić (Supervisor)