Recent years have seen a number of robust technologies for 3D display and 3D spatial tracking hit the market. Devices like the Oculus Rift, the Microsoft Kinect, and the Leap Motion Controller are often touted as the “natural user interfaces” of the future, and there’s a resurgence of interest in virtual reality (VR) and augmented reality (AR). But what makes an interface “natural?” How “realistic” do VR and AR systems need to be? In this talk, I will argue that careful interaction design is more important than ever when designing 3D VR and AR interfaces, and that “magical” interaction can be highly effective in some cases. I will discuss the abilities and limitations of the technologies, describe the results of empirical studies addressing questions of fidelity and naturalism, and present principles for the design of effective 3D user interfaces. Case studies of both successful and unsuccessful VR and AR designs will be used to illustrate these principles. I will conclude with some reflections on the importance of realism and the continuing need for intentional design even as technologies improve.