Publications

Ildus Sadrtdinov, Ekaterina Lobacheva, Ivan Klimov, Mikhail Burtsev, Mikhail Katsnelson, Dmitry Vetrov
International Joint Conference on Artificial Intelligence (IJCAI), 2026
arXiv / code / bibtex

We show that the stationary distributions of scale-invariant networks can be expressed by an ideal gas law. In this interpretation, weight decay acts as pressure, and half of the squared parameter norm acts as volume.

Alexander Shabalin, Simon Elistratov, Viacheslav Meshchaninov, Ildus Sadrtdinov^†, Dmitry Vetrov^†
Conference on Language Modeling (COLM) 2026, 2026
arXiv / X thread / bibtex

We explain why Gaussian diffusion models with the DDPM solver perform poorly on discrete domains, and analyze heuristics (self-conditioning, q-sampling solver, MBR) for improving their quality.

Ildus Sadrtdinov*, Maxim Kodryan*, Eduard Pokonechny*, Ekaterina Lobacheva^†, Dmitry Vetrov^†
Neural Information Processing Systems (NeurIPS), 2024
arXiv / openreview / poster & video / X thread / code / bibtex

We show that only a narrow range of large LRs is beneficial for generalization and analyze it from the loss landscape and feature learning perspectives.

Ildus Sadrtdinov*, Dmitrii Pozdeev*, Dmitry Vetrov, Ekaterina Lobacheva
Neural Information Processing Systems (NeurIPS), 2023
arXiv / openreview / poster & video / X thread / code / bibtex

We study the effectiveness of the exploration of the pre-train basin and its close vicinity for ensembling in transfer learning. We show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, while leaving the basin results in degradation of the ensemble quality.

Alexander Shabalin*, Ildus Sadrtdinov*, Evgeniy Shabalin
ML Reproducibility Challenge 2022, 2023
Outstanding Paper Honorable Mention
pdf / openreview / code / bibtex

We successfully reproduce results of the paper “Towards Understanding Grokking: An Effective Theory of Representation Learning”. We investigate the consistency of training phases depending on data and weight initialization and propose smooth phase diagrams.

Ildus Sadrtdinov, Nadezhda Chirkova, Ekaterina Lobacheva
Workshop on Overparameterization: Pitfalls & Opportunities at ICML, 2021
arXiv / bibtex

We study how different training paradigms (supervised learning, self-supervised learning, and training with random labels) learn training examples. We show that memorization of self-supervised algorithm (SimCLR) is similar to training with random labels.