Over the past decade and a half, Computer Vision has grown from an area of primarily academic research into a true part of people's everyday lives. But with its widespread use have come both successes and failures for deep models, ranging from highly powerful multi-purpose visual backbones to adversarial and backdoor attacks which jeopardize the use of these systems in critical scenarios. Looking ahead to the future, the use of Computer Vision in real-world applications will only continue to grow, and so it is more important than ever to learn from both the failures and successes of deep learning models.
In this thesis, I will examine the strengths and weaknesses of deep neural networks in a range of contexts, including backdoor attacks, self-supervised learning, and applications to a range of tasks. My research begins in the space of adversarial and backdoor attacks, and then expands into more general representation learning and network understanding. This work places a particular emphasis on the Vision Transformer (ViT), which has emerged as the new go-to architecture for Computer Vision tasks, overtaking the previously well-established Convolutional Neural Network (CNN). This study includes an in-depth examination of ViTs trained with different methods of supervision, application of ViTs to video representation learning, and lightweight methods to enhance ViT features for dense downstream tasks.
Throughout this work, I will aim to shine a light on some of the inner aspects of deep models. To quote the sadly passed Dr. David T. Yue: "Every so often, the veil of confusing experimental results is parted, and something deep and beautiful about how biological life works is revealed." The study of biology and human vision is what inspired my passion for Computer Vision, and I believe that same excitement manifests when we can find beautiful and complex emergent properties in deep neural networks. I hope that this thesis can help illuminate a small part of these black boxes, so that we can better understand these incredible inventions of humanity, and so that we can continue to learn from both the failures and successes of deep learning.
Matthew Walmer is a PhD student at the University of Maryland, College Park, Department of Computer Science. He works with Professor Abhinav Shrivastava studying Computer Vision and Machine Learning. He previously completed a Bachelor’s and Master’s degree in Biomedical Engineering at Johns Hopkins University with a focus on Computational Biology. His previous studies in neuroscience and human learning led to a deep interest in Machine Learning and the amazing advances the field has seen in the past 15 years. His current research focuses on investigating how and what neural networks learn as well as their failure cases.