PhD student in Deep Learning
Ildus Sadrtdinov*, Maxim Kodryan*, Eduard Pokonechny*, Ekaterina Lobacheva†, Dmitry Vetrov† We show that only a narrow range of large LRs is beneficial for generalization and analyze it from the loss landscape and feature learning perspectives. |
Ildus Sadrtdinov, Maxim Kodryan, Eduard Pokonechny*, Ekaterina Lobacheva*, Dmitry Vetrov* We study feature learning properties of training with different initial LRs. We show that a narrow range of optimal initial LRs learns a sparse set of the most useful features. At the same time, smaller LRs do not have such specialization, while larger LRs fail to extract useful patterns from data. |
Ildus Sadrtdinov*, Dmitrii Pozdeev*, Dmitry Vetrov, Ekaterina Lobacheva We study the effectiveness of the exploration of the pre-train basin and its close vicinity for ensembling in transfer learning. We show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, while leaving the basin results in degradation of the ensemble quality. |
Alexander Shabalin*, Ildus Sadrtdinov*, Evgeniy Shabalin We successfully reproduce results of the paper “Towards Understanding Grokking: An Effective Theory of Representation Learning”. We investigate the consistency of training phases depending on data and weight initialization and propose smooth phase diagrams. |
Ildus Sadrtdinov, Nadezhda Chirkova, Ekaterina Lobacheva We study how different training paradigms (supervised learning, self-supervised learning, and training with random labels) learn training examples. We show that memorization of self-supervised algorithm (SimCLR) is similar to training with random labels. |
