Given the increasingly prominent role of NLP models in the world, it is crucial that we develop a better understanding of their behavior. In this talk, I present two different angles on this topic that rely on human intuitions about NLI data and NLI model behavior. First, we hand-annotate the development set of the difficult adversarial NLI dataset to better understand which types of reasoning are necessary to get the correct label. We show that the same types of reasoning are responsible for failures and successes of different state-of-the-art models, suggesting we might benefit from focusing our efforts on architectures that can handle those phenomena. Second, we ask whether human-generated explanations of models’ inference decisions align well with how models actually make those decisions. We propose an alignment metric based on human-generated natural language explanations and find that our models are only weakly aligned to humans. Taken together, these projects expose opportunities for using human perspectives to improve our current best models.