Multimodal Large Language Models (MLLMs) have shown impressive results on tasks that require visual and textual reasoning. However, using these systems on assistive technologies raises important questions around privacy protection and the real capabilities of these models when deployed in challenging scenarios in the real world. In this talk, I will explore these limitations in two settings: document understanding and spatial reasoning in 3D environments. I will first present our work on developing an agentic approach for fine-grained privacy protection that selectively masks personal information while preserving contextual visual content for visual question answering. Then, I will present SceneCritic, a symbolic evaluator that uses explicit spatial heuristics to assess the semantic and geometric consistency of generated indoor layouts. Together, these works aim to ask the now prevalent question: can we harness MLLMs and circumvent their limitations as reasoners and evaluators?
[1] SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis
Paola Cascante-Bonilla is an Assistant Professor in the Department of Computer Science at Stony Brook University (SUNY). Her research focuses on developing systems that enable compositional reasoning and common-sense inference through vision and language, while tackling issues such as cultural biases, data distribution, explainability, and trustworthy AI. She was the recipient of the Ken Kennedy Institute SLB Graduate Fellowship (2022/23), she was selected as a Future Faculty Fellow by Rice's George R. Brown School of Engineering (2023) and as a Rising Star in EECS (2023).