Research Seminar

Deep learning continues to achieve impressive breakthroughs across disciplines but relies on increasingly large neural network models that are trained on massive data sets. Their development inflicts costs that are only affordable by a few labs and prevent global participation in the creation of related technologies. But does it really have to be like this? We will identify some of the major challenges of deep learning at small scales and present solution strategies pertaining to the design of sparse training algorithms and problem specific neural network design in the context of agentic networks, which hold the promise to overcome a fundamental trade-off between model specialization and trainability.

Methods for unconstrained continuous optimization can be classified by the strength of the oracle for the objective function they require. First-order methods only require access to function values and gradients, second-order methods additionally require access to Hessians. Higher-order methods correspondingly require access to derivatives of order 1 to p for some p ≥ 3 and are called tensors methods. Intuitively, access to additional derivative information should speed up convergence to an approximate minimizer. We will cover the different perspectives from which this becomes true. First, convergence speed can be quantified from the perspective of global iteration complexity, for which the literature asserts that methods minimizing a regularized Taylor expansions, and in particular the adaptive ARp method, are optimal. Second, we will discuss local convergence results, showing how ARp can converge superlinearly even if the Hessian is singular at the minimizer. Third, numerical experiments on a range of test functions show that (after introducing certain heuristics) the third-order AR3 method needs fewer iterations and oracle calls than the second-order AR2 method.