UCL  IRIS
Institutional Research Information Service
UCL Logo
Please report any queries concerning the funding data grouped in the sections named "Externally Awarded" or "Internally Disbursed" (shown on the profile page) to your Research Finance Administrator. Your can find your Research Finance Administrator at https://www.ucl.ac.uk/finance/research/rs-contacts.php by entering your department
Please report any queries concerning the student data shown on the profile page to:

Email: portico-services@ucl.ac.uk

Help Desk: http://www.ucl.ac.uk/ras/portico/helpdesk
Publication Detail
MARTHE: Scheduling the learning rate via online hypergradients
  • Publication Type:
    Conference
  • Authors:
    Donini M, Franceschi L, Majumder O, Pontil M, Frasconi P
  • Publication date:
    07/01/2021
  • Pagination:
    2119, 2125
  • Published proceedings:
    IJCAI International Joint Conference on Artificial Intelligence
  • Volume:
    2021-January
  • ISBN-13:
    9780999241165
  • Status:
    Published
  • Name of conference:
    Twenty-Ninth International Joint Conference on Artificial Intelligence - IJCAI 20
  • Print ISSN:
    1045-0823
Abstract
© 2020 Inst. Sci. inf., Univ. Defence in Belgrade. All rights reserved. We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization, aiming at good generalization. We describe the structure of the gradient of a validation error w.r.t. the learning rate schedule - the hypergradient. Based on this, we introduce MARTHE, a novel online algorithm guided by cheap approximations of the hypergradient that uses past information from the optimization trajectory to simulate future behaviour. It interpolates between two recent techniques, RTHO [Franceschi et al., 2017] and HD [Baydin et al., 2018], and is able to produce learning rate schedules that are more stable leading to models that generalize better.
Publication data is maintained in RPS. Visit https://rps.ucl.ac.uk
 More search options
UCL Researchers
Author
Dept of Computer Science
University College London - Gower Street - London - WC1E 6BT Tel:+44 (0)20 7679 2000

© UCL 1999–2011

Search by