Publications

You can also find my articles on Google Scholar

To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning

Ildus Sadrtdinov*, Dmitrii Pozdeev*, Dmitry Vetrov, Ekaterina Lobacheva
Neural Information Processing Systems (NeurIPS), 2023
arXiv / openreview / poster & video / code / bibtex

We study the effectiveness of the exploration of the pre-train basin and its close vicinity for ensembling in transfer learning. We show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, while leaving the basin results in degradation of the ensemble quality.

[Re] “Towards Understanding Grokking”

Alexander Shabalin*, Ildus Sadrtdinov*, Evgeniy Shabalin
ML Reproducibility Challenge 2022, 2023
Outstanding Paper Honorable Mention
pdf / openreview / code / bibtex

We successfully reproduce results of the paper “Towards Understanding Grokking: An Effective Theory of Representation Learning”. We investigate the consistency of training phases depending on data and weight initialization and propose smooth phase diagrams.

On the Memorization Properties of Contrastive Learning

Ildus Sadrtdinov, Nadezhda Chirkova, Ekaterina Lobacheva
Workshop on Overparameterization: Pitfalls & Opportunities at ICML, 2021
arXiv / bibtex

We study how different training paradigms (supervised learning, self-supervised learning, and training with random labels) learn training examples. We show that memorization of self-supervised algorithm (SimCLR) is similar to training with random labels.