UCL  IRIS
Institutional Research Information Service
UCL Logo
Please report any queries concerning the funding data grouped in the sections named "Externally Awarded" or "Internally Disbursed" (shown on the profile page) to your Research Finance Administrator. Your can find your Research Finance Administrator at https://www.ucl.ac.uk/finance/research/rs-contacts.php by entering your department
Please report any queries concerning the student data shown on the profile page to:

Email: portico-services@ucl.ac.uk

Help Desk: http://www.ucl.ac.uk/ras/portico/helpdesk
Publication Detail
Generalisation dynamics of online learning in over-parameterised neural networks
  • Publication Type:
    Journal article
  • Publication Sub Type:
    article
  • Authors:
    Goldt S, Advani MS, Saxe AM, Krzakala F, Zdeborov√° L
  • Publication date:
    2019
  • Notes:
    Publisher: arXiv Version Number: 1 copyright: arXiv.org perpetual, non-exclusive license urldate: 2022-11-25 keywords: Disordered Systems and Neural Networks (cond-mat.dis-nn), FOS: Computer and information sciences, FOS: Physical sciences, Machine Learning (cs.LG), Machine Learning (stat.ML), Statistical Mechanics (cond-mat.stat-mech)
Abstract
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisation dynamics of two-layer neural networks in a teacher-student setup, where one network, the student, is trained using stochastic gradient descent (SGD) on data generated by another network, called the teacher. We show how for this problem, the dynamics of SGD are captured by a set of differential equations. In particular, we demonstrate analytically that the generalisation error of the student increases linearly with the network size, with other relevant parameters held constant. Our results indicate that achieving good generalisation in neural networks depends on the interplay of at least the algorithm, its learning rate, the model architecture, and the data set.
Publication data is maintained in RPS. Visit https://rps.ucl.ac.uk
 More search options
UCL Researchers
Author
Gatsby Computational Neurosci Unit
University College London - Gower Street - London - WC1E 6BT Tel:+44 (0)20 7679 2000

© UCL 1999–2011

Search by