Laboratory employee Mikhail Rudakov was awarded for the best article “Compression techniques for layer and gradient activations for distributed training of artificial intelligence models” at the AI Journey conference. The event takes place from November 22 to 24, 2023. Alexander Beznosikov (Innopolis, MIPT), Yaroslav Kholodov (Innopolis) and Alexander Gasnikov (Innopolis, MIPT) worked on the article together with Mikhail.

As part of the SberAI Journey conference, scientific articles were selected for AI Journey Science. As part of this selection, the works will be published in the journal Reports of the Russian Academy of Sciences (Mathematics, computer science, management processes, as well as the English version of doklady mathematics).

“This was my first experience of submitting a scientific article to a conference, which allowed me to work with experts from the MIPT MMO laboratory and dive deeper into the problem being solved,” said Rudakov.

He clarified that the article “Techniques for compressing activations of layers and gradients for distributed training of artificial intelligence models” examines the effect of simultaneous compression of activations and gradients in model parallelization mode on the convergence of the learning process.

“Our work outlines the limits of using compression for training neural networks when dividing a model into several devices. Several new observations have also been made about the peculiarities of using compression in such a task. For example, we notice that activations of neural network layers are less sensitive to compression than gradients; and also that the use of error compensation techniques, although it does not speed up convergence, makes the model more stable, equalizing the quality of the model with and without compression,” the authors of the article report.

The article was also accompanied by a lecture "Techniques for compression of activation layers and gradients for distributed learning of AI models", which was read by the head of the laboratory, Alexander Gasnikov.

The laboratory staff congratulates the team on this achievement and wishes them further success in their research!

The work was supported by the Russian Science Foundation (project No. 23-11-00229).

Photo source: Machinelearning telegram channel.