publications

2024

1. 

   Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision

   Alberta Longhini, Marcel Büsching, Bardienus Pieter Duisterhof, and 4 more authors

   8th Annual Conference on Robot Learning (CoRL), Nov 2024

   Abs HTML

   We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth’s state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by 85%.

2. 

   FlowIBR: Leveraging Pre-Training for Efficient Neural Image-Based Rendering of Dynamic Scenes

   Marcel Büsching, Josef Bengtson, David Nilsson, and 1 more author

   CVPR 2024 Workshop on Efficient Deep Learning for Computer Vision, Jun 2024

   Abs HTML

   We introduce a novel approach for monocular novel view synthesis of dynamic scenes. Existing techniques already show impressive rendering quality but tend to focus on optimization within a single scene without leveraging prior knowledge. This limitation has been primarily attributed to the lack of datasets of dynamic scenes available for training and the diversity of scene dynamics. Our method FlowIBR circumvents these issues by integrating a neural image-based rendering method, pre-trained on a large corpus of widely available static scenes, with a per-scene optimized scene flow field. Utilizing this flow field, we bend the camera rays to counteract the scene dynamics, thereby presenting the dynamic scene as if it were static to the rendering network. The proposed method reduces per-scene optimization time by an order of magnitude, achieving comparable results to existing methods - all on a single consumer-grade GPU.

3. 

   Distilling Semantic Features for 3D Cloth Representations from Vision Foundation Models

   Alberta Longhini, Marcel Büsching, Bardienus Pieter Duisterhof, and 3 more authors

   ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, Jun 2024

   Abs HTML

   This study explores the use of vision foundation models to enhance 3D representations of cloth-like deformable objects. By focusing on the distillation of semantic information from RGB images, we examine the potential of pre-trained Visual-Language Models in capturing complex folded configurations of cloth. Our investigation reveals the challenges and preliminary successes in leveraging semantic information to improve the understanding and tracking of deformable object states.