Impressive successes of large language models are based on a unified pipeline: pre-training on large datasets and fine-tuning with reinforcement learning (RL). However, this “recipe” does not directly work for robotics, because (i) robotics have a data scarcity problem, (ii) deploying robots to interact with the physical world is expensive, and (iii) RL for robotics is not data-efficient. In this talk, I will present my research on addressing these challenges to develop a pipeline towards building embodied intelligence through RL. I will first present data-efficient RL algorithms for robotics, which can solve robotic tasks that were not possible to solve with previous RL algorithms in a reasonable amount of time. Then, I will describe how to effectively train world models from videos so that we can use synthetic data for training robots with RL. Finally, I will describe my research on offline-to-online RL that identifies the challenges of fine-tuning RL agents trained on a narrow data distribution and addresses them. I will conclude by discussing future directions for robustifying this pipeline by incorporating large generative models trained on internet-scale data.