Curriculum learning is an effective and natural strategy in human learning. It plays an important role in challenging tasks such as language learning. However, current machine learning (ML) paradigms are mostly built upon repeatedly practicing the same training data/tasks with a random order, which is non-adaptive to the learning process. Moreover, they do not plan multiple learning stages in advance as humans.

In the first part of this talk, I will introduce several novel formulations of curriculum learning, which lead to theoretically motivated and practically effective algorithms for a broad class of ML problems, e.g., supervised learning, semi-supervised learning, noisy-label learning, and diverse ensemble learning. We observe significant advantages of curriculum learning on weakly-labeled data. The second part of this talk will present a line of our works that develop scores from analyzing the training dynamics. Compared to the commonly used instantaneous feedback (e.g., the loss per step), these scores can save a lot of computation and capture richer information about the loss landscape's sharpness and the training inconsistency, hence improving the efficiency and test-set performance in experiments.

In the third part, I will discuss several potential forms of curriculum learning for pre-training language models or for model adaptation to downstream NLP tasks. According to the structures of NLP data, we can develop curricula at different levels. We will also discuss the design of scores and learning schedules for NLP models/tasks.