报告人：李建
报告地点：腾讯会议 256-907-699
报告时间：2022年11月3日星期四下午4:00
报告题目：Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability
报告简介：
Recent findings demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS). In this regime, the sharpness, i.e., the maximum Hessian eigenvalue, first increases to the value 2/(step size) (the progressive sharpening phase) and then oscillates around this value (the EOS phase). Such behavior is inconsistent with several widespread presumptions and traditional wisdom in the field of optimization and raises several intriguing questions about neural network training. This paper aims to analyze the GD dynamics and the sharpness along the optimization trajectory. Our analysis naturally divides the GD trajectory into four phases depending on the change in the sharpness value. We empirically identify the norm of output layer weight as an interesting indicator of the sharpness dynamics. Based on this empirical observation, we attempt to theoretically and empirically explain the dynamics of various key quantities that lead to the change of the sharpness in each phase of EOS.
报告人简介：
李建目前是清华大学交叉信息研究院长聘副教授，博士生导师。他在中山大学取得的学士学位和复旦大学取得的硕士学位，马里兰大学博士毕业。他的研究兴趣主要包括算法设计与分析，机器学习，数据库，金融科技。他已经在主流国际会议和杂志上发表了100余篇论文等，并获得了VLDB 2009和ESA 2010的最佳论文奖，ICDT 2017最佳新人奖，清华221基础研究青年人才支持计划，教育部新世纪人才支持计划，国家自然科学基金优秀青年基金。他主持并参与了多项科研项目，包括自然科学基金面上项目，中以国际合作项目，青年973计划等，以及多个企业级合作项目，包括蚂蚁金服、华泰证券、易方达、微软、百度、滴滴等。
