Savoirs ENS

Optimization's Hidden Gift to Learning: Implicit Regularization

mardi 01 octobre 2019

Loading the player...

Descriptif

Exposé de Nathan Srebro dans le cadre du Data Science Colloqiuum de l'ENS.

It is becoming increasingly clear that implicit regularization afforded by the optimization algorithms play a central role in machine learning, and especially so when using large, deep, neural networks. We have a good understanding of the implicit regularization afforded by stochastic approximation algorithms, such as SGD, for convex problem, and we understand and can characterize the implicit bias of different algorithms, and can design algorithms with specific biases. But in this talk I will focus on implicit biases of local search algorithms for non-convex underdetermined problem, such as deep networks. In an effort to uncover the implicit biases of gradient-based optimization of neural networks, which holds the key to their empirical success, I will discuss recent work on implicit regularization for matrix factorization, linear convolutional networks, and two-layer ReLU networks, as well as a general bottom-up understanding on implicit regularization in terms of optimization geometry.

Thèmes : Informatique
Catégories: Data Science Colloquium
Mot-clés : mathématiques, algorithme, machine learning

Voir aussi