Human action provides the common basis for connecting visual, linguistic, and motoric representations of the world: the same action can be seen, described, felt, and executed. This dissertation develops a progression of representations and systems for understanding human manipulation action from egocentric video, grounded not only in appearance but in the physical and motoric structure of the action.

The dissertation begins with contact. Contact Anticipation Maps and Next Active Object segmentations capture where and when near-future interactions will occur, and Egocentric Object Manipulation Graphs use these cues for state-of-the-art action anticipation. We then build on contact to define Therbligs, a compositional vocabulary of motion primitives, and compose these primitives into action programs with LEAP, which uses Large Language Models to generate video-grounded programs with sub-actions and control flow.

Vision, however, observes manipulation only through its consequences, while contact and force are often hidden by the hands themselves. The rest of the dissertation therefore grounds visual representations in direct measurements of the physical signals that drive action. FEEL pairs egocentric video with force measurements from custom piezoresistive gloves, providing contact supervision without manual annotation. MotorSense goes beyond fingertip forces, in pairing egocentric video with bimanual wrist-worn electromyography (EMG), from which we learn discrete motor codes that improve action recognition and 3D hand-object reconstruction and enable controllable video generation.

Together, these works seek to understand how people physically interact with the world, how the structure of those interactions can be represented, and how those representations can ultimately support robotic systems.