Large language models (LLMs) have utterly transformed the field of natural language processing. However, training LLMs comes at a massive financial and environmental cost, making them out of reach of academic research labs. Meanwhile, these models are costly to update and remain opaque in their reasoning process. In this talk, I will discuss two recent works from my group and share some thoughts on how we could move forward in the age of LLMs. The first half of the talk will introduce a new training approach for language modeling called TRIME, which is designed for training LMs with memory augmentation. Our approach can enable language models to better leverage long-range contexts or retrieve external knowledge at testing time, with negligible compute overhead compared to standard LM training. I will argue that augmenting language models with a large retrieval component can greatly ease the computational and privacy concerns in current LLMs. In the second part of the talk, I will introduce a new framework called NLProofS for deductive reasoning in natural language. I will argue that current LLMs still struggle with generating complex proofs in a single shot, while our step-by-step approach with an independently-trained verifier can generate both valid and relevant steps and achieve much stronger performance.