As anime-style content becomes more popular on the global stage, we ask whether new vision/graphics techniques could contribute to the artform. However, the highly-expressive and non-photorealistic nature of anime poses additional challenges not addressed by standard ML models, and much of the existing work in the domain does not align with real artist workflows. In this dissertation, we will present work building foundational 2D/3D infrastructure in the anime domain (including pose estimation, video frame interpolation, and 3D character reconstruction), as well as new tools for professional 2D animators that leverage novel vision/graphics techniques to assist drawing.
Shuhong (Shu) Chen is a CS PhD student at UMD, advised by Prof. Matthias Zwicker. Prior to joining UMD in 2019, he received his BS in CS and Math from Rutgers University, where he worked with Prof. Ivan Marsic on ML for healthcare informatics in trauma resuscitation. Shu has not only done research internships in tech (at Meta, TikTok, and MIT Lincoln Labs), but also with the Japanese anime industry (OLM Digital, Arch Inc.). Currently, his research interests are in non-photorealistic traditional 2D animation and 3D character modeling. He somewhat unfortunately spends his spare time on anime, vtubers, and gacha games.