Training dynamics in modern neural network optimization

Abstract

This article examines the critical role of training dynamics in the optimization of modern neural networks. Moving beyond a static analysis of algorithms and converged solutions, we argue that the time-evolving trajectory of parameters and internal representations - the dynamical system defined by the interaction of optimizer, architecture, and data - is fundamental to understanding contemporary deep learning phenomena. We analyze the journey from initialization through convergence, exploring how initial conditions implicitly regularize the optimization path, how stochasticity and loss landscape geometry interact to select flat minima, and how adaptive optimizers alter the fundamental search dynamics. The discussion extends to the co-evolution of internal representations, the emergent macroscopic scaling laws observed in large-scale training, and theoretical lenses such as the Neural Tangent Kernel that provide formal insight into these behaviors. Ultimately, a dynamic perspective is shown to be indispensable for explaining implicit regularization, generalization, and the emergence of capabilities, thereby guiding the development of more efficient, robust, and controllable optimization strategies for advanced artificial intelligence systems.