Gradient-boosted decision tree (GBDT) algorithms such as XGBoost, CatBoost, and LightGBM have been the de facto standard for tabular and time series problems for the past decade. Meanwhile, tabular prior-fitted networks (PFNs)—a recent class of transformer-based foundational models trained for in-context learning—have demonstrated superior performance on small and medium-sized datasets. Large language models (LLMs) have also shown strong zero- and few-shot performance on tabular tasks by leveraging column headers and descriptive metadata. However, transformer models struggle to scale to large datasets due to inherent context-length limitations.

This dissertation presents a set of contributions aimed at improving the scalability of transformer-based models for tabular and time series prediction, enabling them to retain their pretraining benefits, natural-language understanding, and strong small-data reasoning capabilities while extending reliably to large-scale datasets and joint multivariate prediction settings.