Technology-assisted review (TAR) refers to the use of supervised learning and other technologies to support large scale manual document reviews in the law, medicine, content moderation, and other areas. Statistical estimates play a critical role in deciding when to end such a review, but are often used in ways that introduce multiple comparisons bias. We present a new evaluation approach for TAR based on confidence sequences which allows a review manager complete freedom to control the selection and timing of review decisions while avoiding bias. The method is expensive in terms of sample size, but provides several directions for future improvements. We also discuss evaluation in electronic discovery (eDiscovery) in the law more broadly, including the impact of recent developments in generative AI.
Lenora Gray is a Data Scientist at Redgrave Strategic Data Solutions (Redgrave Data), with experience building predictive models for use in TAR workflows. Prior to joining Redgrave Data, she was an Advisor at Redgrave LLP. She is pursuing her M.S. in Data Science at John Hopkins as a Fellow of the National GEM Consortium.
David D. Lewis is Chief Scientific Officer and co-founder of Redgrave Data, a boutique legal technology services firm. He has worked for over 35 years at the intersections of information retrieval, machine learning, natural language processing, and statistics, in a wide variety of research, industry, and consulting roles. He was elected a Fellow of the American Association for the Advancement of Science in 2006 for foundational work in text analytics.