Photorealistic mapping, which aims to reconstruct the three-dimensional world, is a foundational problem in computer vision and graphics. When this reconstruction faithfully captures both the geometry and appearance of the real environment, it produces a digital twin of the real environment. Such high-quality digital twins enable immersive user experiences and scalable data generation through real-to-simulation-to-real pipelines. However, constructing such digital twins remains challenging due to limitations in sensing, reconstruction accuracy, rendering fidelity, and data acquisition and synthesis efficiency. We advance photorealistic 3D mapping through three interconnected research goals: sensing and perception, mapping, and generation.

The first part of our dissertation focuses on sensing and perception, with an emphasis on single-view depth estimation for indoor and dynamic environments to enhance or replace external depth sensors. By leveraging self-supervised learning and integrating sparse geometric priors derived from 3D LiDAR sensors, we introduce SelfDeco. Our method enables accurate metric depth estimation without relying on dense ground-truth depth maps. Complementing this approach, we propose DnD, which incorporates multi-view 3D reconstructions to infer dense depth in dynamic environments, achieving a 3.6%–10.2% reduction in root-mean-square error compared to prior methods.

The second part studies mapping and explores how classical geometry and graphics pipelines can be combined with neural representations. For object-level reconstruction, we present TMO, which leverages neural implicit surface representations to reconstruct complete geometry, improving geometric accuracy while reducing texture artifacts. To support unbounded scene reconstruction, we introduce LTM, a mesh-based neural rendering approach that produces compact yet geometrically faithful representations. LTM generates optimized meshes that achieve comparable rendering quality while reducing triangle count by 73x and memory usage by 11x. Bridging mesh-based modeling and point-based neural rendering, we further present MeshGS, which integrates explicit mesh geometry with 3D Gaussian splatting to achieve high-quality rendering while preserving structural consistency. Finally, we extend these pipelines to omnidirectional sensing through IM360, a framework that enables spherical structure-from-motion and large-scale indoor mapping using 360-degree cameras.

The third part of the dissertation addresses data generation by leveraging reconstructed digital twins to synthesize high-quality training data for downstream perception tasks. Building on 3D Gaussian splatting, we propose UAVTwin, which constructs high-fidelity digital twins that support controllable data augmentation in scenes with multiple moving humans. By enabling realistic data generation with accurate annotations, our approach substantially improves learning in difficult-to-capture real-world scenarios, as validated by performance gains in downstream tasks such as person detection and human action recognition. Finally, we introduce UAV4D, which extends dynamic Gaussian splatting to aerial settings and enables robust novel-view rendering of dynamic UAV scenes, achieving a 1.5 dB increase in PSNR and improved visual sharpness.