log in  |  register  |  feedback?  |  help  |  web accessibility
PhD Proposal: Segmentation Before Structure: Real-Time Moving-Object Segmentation and Egomotion Estimation with Event Cameras
Zhiyuan Hua
IRB-3137
Friday, September 18, 2026, 2:00-3:30 pm
  • You are subscribed to this talk through .
  • You are watching this talk through .
  • You are subscribed to this talk. (unsubscribe, watch)
  • You are watching this talk. (unwatch, subscribe)
  • You are not subscribed to this talk. (watch, subscribe)
Abstract

Interpreting a dynamic world from a moving camera requires answering two entangled questions: how is the observer moving, and what else is moving independently? Classical structure-from-motion, whether frame-based or event-based, typically resolves this chicken-and-egg problem in a fixed order: estimate egomotion first under a static-scene assumption, then explain away independently moving objects as outliers of that model. This ordering is fragile exactly where autonomous systems need vision the most—in cluttered, fast, dynamic scenes—and it has been inherited, largely unexamined, by the event-camera literature. This thesis proposes to reverse the pipeline. Building on the observation, both biological and geometric, that independent motion is detectable before and without egomotion, we develop four tightly connected components: a universal motion signal: per-event normal flow with calibrated confidence, strictly local so that it transfers across scenes and sensors; moving-object segmentation solved first, from events alone, without egomotion priors; egomotion estimation as a learned minimizer that keeps hard geometric constraints inside the loop, learning how to optimize rather than what scenes look like; and temporal integration that lifts instantaneous estimates to multi-frame odometry through event-surface, event-rate, and inertial constraints.

The completed work establishes the pivot of this pipeline, segmentation first, through three projects. A graph-convolutional network operating on raw event clouds demonstrated that extended temporal context, together with local geometry rather than semantic appearance, resolves motion segmentation; a classical successor isolated moving objects through residual analysis of a normal-flow field, without a prior egomotion estimate; and our border-ownership network completed the argument: trained entirely in simulation, it segments moving objects faster than real time, transfers zero-shot to real cameras, and matches or surpasses methods supervised with real data.

Ongoing work supplies the stages on either side of segmentation: two complementary normal-flow estimators, one dense and real-time, the other asynchronous and geometry-exact, which classify events by reliability as a byproduct; and a physics-in-the-loop recurrent egomotion estimator that recovers the translation direction in real time without pose supervision. The proposed research completes the pipeline by closing the real-time gap for asynchronous estimation, extending segmentation-aware egomotion to all six degrees of freedom, and building the temporal-integration layer: a tightly coupled event-inertial backend that fuses event-surfaces, event-rate, and inertial constraints to anchor rotation, gravity, and metric scale. The result will be a principled, efficient, biologically motivated alternative to both end-to-end learning and classical optimization for event-based dynamic scene understanding.

Bio

Zhiyuan Hua is a PhD student in Perception and Robotics Lab, University of Maryland, advised by Cornelia Fermüller and Yiannis Aloimonos. His research focuses on event-based vision, particularly motion segmentation and egomotion estimation. He combines geometric reasoning, machine learning, and insights from biological vision to develop efficient, generalizable perception systems for dynamic environments.

This talk is organized by Migo Gui