Deep learning models have demonstrated excellent empirical performance in different application domains. However, a satisfactory understanding of deep learning foundations continues to elude us. In this talk, I will explain our recent results in understanding some of the fundamental problems in supervised and unsupervised deep learning.

In unsupervised learning, I will first establish a principled connection between two modern generative approaches, namely Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs). Leveraging this connection, I will show how sample likelihoods can be computed in GANs facilitating their use in statistical inference applications. Next, I will explain why the standard Wasserstein distance can lead to undesired results when applied to mixture distributions with imbalanced mixture proportions. To resolve this issue, I will present a new distance measure namely the normalized Wasserstein distance and show its effectiveness in GANs, domain adaptation, adversarial clustering and hypothesis testing.

In supervised learning, I will explain impacts of high-order loss approximations and features in two related problems of deep learning interpretation and robustness. In particular, by obtaining a closed-form formula for the Hessian matrix of a deep ReLU network, I will characterize differences between first and second order interpretation methods. Finally, I will explain our recent results on attack-agnostic robustness certificates for a multi-label classification problem using deep ReLU networks. In particular, I will present a certificate that has a closed-form, is differentiable and is an order of magnitude faster to compute than existing methods even for deep networks.