Multi-robot systems are becoming essential in applications ranging from warehouse automation to search and rescue, offering advantages in speed, coverage, and capability compared to single robots. However, getting multiple robots to learn and coordinate effectively remains challenging - training robots to work together requires extensive effort and often fails to generalize beyond training conditions. This proposal addresses two fundamental challenges in multi-robot learning: reducing the training effort required and improving generalization to reduce policy retraining. First, we propose methods to leverage more easily obtainable data to facilitate the learning - using human-drawn sketches as an alternative to costly teleoperated demonstrations for manipulation tasks, and utilizing individual robot demonstrations instead of harder-to-obtain joint multi-robot demonstrations for learning collaborative behaviors. Second, we develop techniques to help learned policies adapt to new scenarios without retraining - introducing frameworks that maintain coordination under different observation conditions and enable effective information sharing across varying initial state distributions. Building on these completed works, we propose to tackle zero-shot coordination with new teammates and generalization to diverse opponent strategies, aiming to create multi-robot systems that can quickly adapt to new partners and adversaries without extensive retraining.