Multi-robot systems are becoming essential in applications ranging from warehouse automation to search and rescue, offering advantages in speed, coverage, and capability compared to single robots. However, getting multiple robots to learn and coordinate effectively remains challenging - training robots to work together requires extensive data collection and often fails to generalize beyond training conditions. This dissertation addresses two fundamental challenges in multi-robot learning: reducing the training effort required and improving generalization to reduce policy retraining. First, we propose methods to make training data collection easier - using human-drawn sketches instead of teleoperated demonstrations for manipulation tasks, and leveraging individual robot demonstrations rather than joint multi-robot demonstrations for learning collaborative behaviors. Second, we develop techniques to help learned policies adapt to new scenarios without retraining - introducing frameworks that maintain coordination under different observation conditions and enable effective information sharing across varying initial state distributions. Building on these completed works, we propose to tackle zero-shot coordination with new teammates and generalization to diverse opponent strategies, aiming to create multi-robot systems that can quickly adapt to new partners and adversaries without extensive retraining.