The field of computer vision has advanced remarkably during the past 10-15 years.  This is due to a variety of factors including the availability of the large annotated data sets needed to train deep learning models, software like AMT that enables the collection of these data sets at reasonable costs, important engineering improvements to  the training methodologies of deep networks, dramatic decreases in price/performance ratios of computing systems (especially GPU’s) and memory systems, widespread availability of source code that researchers make available to one another worldwide, and inexpensive sensors and robotic platforms like Kinect, Go-pro’s and UAV’s .  So, while the fundamental vision problems of detection and recognition of objects and human movements are not solved, they have improved to the point where it is important to ask: What’s next?  A workshop was held in the US late last year to address  exactly that question (chaired by me, Fei Fei Li and Devi Parikh) and this talk will discuss the conclusions of that workshop, and illustrate research in some of those future directions with work from some current students in the Computer Vision Laboratory.