Robot manipulation is learned mostly from action-labeled robot data, typically teleoperated and collected anew for every task and embodiment, most expensively for dexterous hands and aerial manipulators. An action-labeled demonstration bundles three kinds of supervision: motion (how hands, tools, and objects should move), judgment (what to try, whether it worked, why it failed, what is where), and embodiment grounding (which commands make a particular robot produce that motion). The goal of this proposal is to make robot manipulation scale with abundant non-robot knowledge: robots should learn motion from human and cross-embodiment video, obtain judgment from foundation-model agents, and spend action-labeled data only on grounding these priors, ideally once per embodiment rather than once per task. Knowledge passes through embodiment-agnostic interfaces (3D traces, relational scene graphs, executable programs, and spatially grounded 3D memory) and is grounded at the end. In Pillar I (motion from video), TraceGen, MuZero, and Dex-One2Many learn from action-free video: TraceGen reached 67.5% success on four real-robot tasks after adapting on five handheld-phone human videos per task, and Dex-One2Many reached 75% mean success on five real tasks in configurations absent from each task's single human video. In Pillar II (judgment from agents), IVE, Fly-by-Code, and Sequential EQA let agents imagine, verify, probe, and remember; Fly-by-Code's Active Feedback raised final success on six simulated aerial-manipulation tasks from 31.7% with terminal feedback to 75%. Ultimately, this proposal presents an efficient methodology for robot automation that minimizes the need for action-labeled data by learning behaviors from video and leveraging Vision-Language Models (VLMs) as autonomous agents.
Seungjae Lee is a Ph.D. student in the Department of Computer Science at the University of Maryland, College Park, which he joined in August 2024. He is co-advised by Dr. Furong Huang and Dr. Jia-Bin Huang. His research focuses on robot manipulation with minimal action-labeled data: learning motion from human and cross-embodiment video, and using vision-language-model agents to supply the judgment that robot learning otherwise gets from human demonstrations.

