With the recent proliferation of automated text analysis methods, scholars are increasingly turning to open-ended survey responses as a measure of public opinion. However, the quality of open-ended responses on online surveys can be extremely variable.

Using Large Language Models, we rank the quality of open-ended responses to a survey question about the impact of social media and code them based on topics mentioned. We find substantial differences in the conclusions drawn from topic-level analysis of lower versus higher quality open-ended answers. Finally, we leverage a randomized experiment in answer modality (voice vs text) to randomly induce higher quality responses among some respondents. We find that inducing higher quality responses creates similar changes in topic coding.