We are investigating how to effectively use machine learning to provide assistance in the process of expert data annotation and to facilitate data exploration. While labeled data are essential for supervised learning of predictive models in NLP, manual annotation of large data sets can be cost-prohibitive. We reduce annotation cost by offering human annotators various forms of machine assistance designed to increase annotator speed and accuracy and by leveraging their annotations for prediction on larger volumes of data. One aspect of this work focuses on practical cost-conscious active learning (AL) in annotator-initiated environments. Employed properly, AL uses learned models to select data instances that are both informative and inexpensive to annotate. We present results showing that cost-conscious AL can significantly reduce the cost of annotating data in practice. We present a method for eliminating human wait time in AL, independent of specific learning and scoring algorithms, by making scores always available for all instances, using old (stale) scores when necessary. While a human is annotating, the machine is also training models and scoring instances - in parallel - to maximize the recency of the scores. Our "no-wait" method can be seen as a parameter-free, dynamic batch AL algorithm. The performance of our method depends on the relative amounts of time required for annotating, training, and scoring. Time permitting, I will also present some new research involving the inference of labels from multiple, fallible annotators.

Joint work with: Robbie Haertel, Paul Felt, and Kevin Seppi