Self-driving vehicles will bring us safer, cleaner, and more convenient transportation. To make this dream come true, we need our autonomous system to perceive, plan, and execute effectively in unstructured environments and have guaranteed safety. While machine learning has significantly enhanced autonomous capabilities, we are still missing key ingredients to achieve the desired goal. In this talk, I will present our approach towards autonomous driving. The core idea is to systematically integrate learning methods with structured models and human priors of the world. The effectiveness of our integrated approach has been demonstrated at the full spectrum of self-driving tasks, including localization, perception, planning and simulation, and our developed algorithms have been deployed in real-world production systems. Finally, I will give a brief personal outlook on open research topics towards realistically solving self-driving.