E-Commerce has been growing at a rapid pace in recent years. Digital try-on systems, as one alternative way to improve the user experience and popularize online garment shopping, have drawn the attention of many researchers. However, the technology is still far from being practical and easy to use to replace physical try-on, mostly due to the gap in modeling and in demonstrating garment-fitting between the digital and the real worlds. The estimation of the hidden parameters of the garments plays an important role in closing the gap. In this talk, I will introduce several methods to address the key open research issues above by adopting machine learning and optimization techniques. First, I will present a unified garment representation based on body surface correspondences and a generative network for garment editing and creation. Next, I will talk about training an auto-encoder for garment geometry together with a joint estimation pipeline to predict the human body, garment geometry, and fabric material at the same time. Moreover, I will introduce the first differentiable cloth simulation to incorporate physics constraints into deep learning. At last, we will take a look at how this idea can be generalized and simplified to create vivid and physically accurate garment draping predictions for virtual try-on usage.