As part of this larger system, in this proposal, I will focus on posture specific issues and will present the development of the following components: a) posture assessment b) machine learning for fine motor movement and c) user interface for visualization and feedback. I take a human-centered design approach for the development of Vaiolin, then take a top-down approach to build an augmented learning user interface and evaluation metrics. In this work, I will present my ongoing work for a novel visualization of posture issues and visual and auditory feedback mechanisms to explain and guide students on how to fix the issues. To achieve this, I will discuss models used to capture the meaning of posture issues, sophisticated bowing techniques, and fine hands movements. In this work, a combined human body and instrument geometry is learned through a spatio-temporal graph convolution network. For low level features, I will discuss temporally coherent human 3D pose refinement network. For violin and bow 3D pose, I will present the creation and generation of large synthetic data and learning the 3D pose. For the posture issues I will define ontology of salient issues due to posture and movement patterns grounded in kinesiology theories and music teaching. Finally I will summarize the entire system to connect the workflow to create a scalable system.
Dept rep: Dr. Ramani Duraiswami
Members: Dr. Cornelia Fermüller
Dr. Irina Muresanu
Snehesh Shrestha is a Ph.D. student in the Department of Computer Science at the University of Maryland College Park, under the supervision of Prof. Yiannis Aloimonos, and co-advised by Dr. Cornelia Fermüller. His research is at the intersection of Robotics, Computer Vision, Machine Learning, Human-Robot and Human-Computer Interaction, and Human Behavior.