To deploy AI agents in real world applications often requires considering number of factors that may not arise during relatively sanitized development stages like addressing domain shifts between training data and inference time, adapting to new categories, and performing efficient inference.  Many see large models trained on web-scale datasets as a way to combat many of the issues with deploying models as their scale means that a larger portion of the target distribution may be seen that effectively reduces domain gaps while also reducing the potential for dataset bias during pretraining stages. In this talk, I will showcase some recent work in my lab where we demonstrate that many of these issues still remain despite the scaling effects, and new issues are introduced.  In particular, we find that recent large model training simply shifts what data can be considered in-domain rather than making models more inherently generic.  Relying on web-scraped datasets also produces models more vulnerable to attacks by web artifacts, intensifying the need for mitigating strategies.   Finally, I will close by discussing methods to address energy, storage, and annotation issues stemming from adapting large models to a task.