AI models have the potential to support and complement human decision-makers and users. And yet, the deployment of human-AI teams still faces practical challenges. I’m interested in developing a more principled workflow for building human-AI teams, which involves carefully examining points in the team setup and asking the following questions: (i) what are the right metrics to optimize the AI model for, and (ii) how can we facilitate effective communication between humans and AI. In this talk, I will discuss how existing literature has attempted to answer each of these questions, their limitations, and promising alternatives.