Over the last five years, the rapid increase in the capability and ubiquity of language models has revealed a clear and pressing need to incorporate transparency enhancing technologies into these systems. In the first part of this thesis we introduce a method for watermarking the outputs of language models and showcase how this technology can enable robust content provenance in realistic deployment scenarios. Next, we present studies on training data attribution and memorization in language models while shedding light on how web-scale pretraining presents unique and fundamental challenges in this space. To enable more controlled experiments in these research domains we develop a synthetic dataset pipeline that generates realistic but semantically isolated documents and questions suitable for further studies on memorization and knowledge acquisition. Finally, motivated by prescient issues at the intersection of intellectual property law and language model training, we conclude by demonstrating that proactive, selective watermarking by content creators or model providers can make training data membership testing---determining whether or not their data or model outputs were included in a training dataset---a more tractable problem.