The future of Artificial Intelligence demands a paradigm shift towards multisensory perception—to systems that can digest ongoing multisensory observations, that can discover structure in unlabeled raw sensory data, and that can intelligently fuse useful information from different sensory modalities for decision making. While we humans perceive the world by looking, listening, touching, smelling, and tasting, traditional form of machine intelligence mostly focuses on a single sensory modality, particularly vision. My research aims to teach machines to see, hear, and feel like humans to perceive, understand, and interact with the multisensory world. In this talk, I will present my research of multisensory machine intelligence that studies two important aspects of the multisensory world: 1) multisensory objects, and 2) multisensory space. In both aspects, I will talk about how I design systems to reliably capture multisensory data, how I effectively model them with new differentiable simulation algorithms and deep learning models, and how I explore creative cross-modal/multi-modal applications with sight, sound, and touch. In the end, I will conclude with my future plans.
Ruohan Gao is a Postdoctoral Research Fellow working with Prof. Fei-Fei Li, Prof. Jiajun Wu, and Prof. Silvio Savarese in the Vision and Learning Lab at Stanford University. He obtained his Ph.D. advised by Prof. Kristen Grauman at The University of Texas at Austin and B.Eng. at The Chinese University of Hong Kong. Ruohan mainly works in the fields of computer vision and machine learning with particular interests in multisensory learning with sight, sound, and touch. His research has been recognized by the Michael H. Granof Award which is designated for UT Austin's Top 1 Doctoral Dissertation, the Google PhD Fellowship, the Adobe Research Fellowship, a Best Paper Award Runner Up at British Machine Vision Conference (BMVC) 2021, and a Best Paper Award Finalist at Conference on Computer Vision and Pattern Recognition (CVPR) 2019.