Abstract: Despite years of empirical success with deep learning for many large-scale problems, existing theoretical frameworks fail to explain many of the most successful heuristics used by practitioners.  The primary weakness most approaches encounter is a reliance on the typical large data regime, which neural networks often do not operate in due to their large size.  To overcome this issue, I will describe how for any overparameterized (high-dimensional) model, there exists a dual underparameterized (low-dimensional) model that possesses the same marginal likelihood, establishing a form of Bayesian duality.  Applying classical methods to this dual model reveals the Interpolating Information Criterion, a measure of model quality that is consistent with current deep learning heuristics.  I will also describe how, in many modern machine learning settings, the benefits of ensembling are less ubiquitous and less obvious than classically.  Theoretically, we prove simple new results relating the ensemble improvement rate (a measure of how much ensembling decreases the error rate versus a single model, on a relative scale) to the disagreement-error ratio.  Empirically, the predictions made by our theory hold, and we identify practical scenarios where ensembling does and does not result in large performance improvements.  Perhaps most notably, we demonstrate a distinct difference in behavior between interpolating models (popular in current practice) and non-interpolating models (such as tree-based methods, where ensembling is popular), demonstrating that ensembling helps considerably more in the latter case than in the former.