log in  |  register  |  feedback?  |  help  |  web accessibility
PhD Proposal: Motion from Video, Judgement from Agents: Robot Manipulation with Minimal Action-Labeled Data
Seungjae Lee
IRB-5105 https://umd.zoom.us/j/7528443694?omn=93255705105
Monday, November 16, 2026, 3:00-4:30 pm
  • You are subscribed to this talk through .
  • You are watching this talk through .
  • You are subscribed to this talk. (unsubscribe, watch)
  • You are watching this talk. (unwatch, subscribe)
  • You are not subscribed to this talk. (watch, subscribe)
Abstract

Robot manipulation is learned mostly from action-labeled robot data, typically teleoperated and collected anew for every task and embodiment, most expensively for dexterous hands and aerial manipulators. An action-labeled demonstration bundles three kinds of supervision: motion (how hands, tools, and objects should move), judgment (what to try, whether it worked, why it failed, what is where), and embodiment grounding (which commands make a particular robot produce that motion). The goal of this proposal is to make robot manipulation scale with abundant non-robot knowledge: robots should learn motion from human and cross-embodiment video, obtain judgment from foundation-model agents, and spend action-labeled data only on grounding these priors, ideally once per embodiment rather than once per task. Knowledge passes through embodiment-agnostic interfaces (3D traces, relational scene graphs, executable programs, and spatially grounded 3D memory) and is grounded at the end. In Pillar I (motion from video), TraceGen, MuZero, and Dex-One2Many learn from action-free video: TraceGen reached 67.5% success on four real-robot tasks after adapting on five handheld-phone human videos per task, and Dex-One2Many reached 75% mean success on five real tasks in configurations absent from each task's single human video. In Pillar II (judgment from agents), IVE, Fly-by-Code, and Sequential EQA let agents imagine, verify, probe, and remember; Fly-by-Code's Active Feedback raised final success on six simulated aerial-manipulation tasks from 31.7% with terminal feedback to 75%. Ultimately, this proposal presents an efficient methodology for robot automation that minimizes the need for action-labeled data by learning behaviors from video and leveraging Vision-Language Models (VLMs) as autonomous agents.

Bio

Seungjae Lee is a Ph.D. student in the Department of Computer Science at the University of Maryland, College Park, which he joined in August 2024. He is co-advised by Dr. Furong Huang and Dr. Jia-Bin Huang. His research focuses on robot manipulation with minimal action-labeled data: learning motion from human and cross-embodiment video, and using vision-language-model agents to supply the judgment that robot learning otherwise gets from human demonstrations.

This talk is organized by Migo Gui