FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W3166972615> ?p ?o ?g. }

• W3166972615 abstract "Error feedback (EF), also known as error compensation, is an immensely popular convergence stabilization mechanism in the context of distributed training of supervised machine learning models enhanced by the use of contractive communication compression mechanisms, such as Top-$k$. First proposed by Seide et al (2014) as a heuristic, EF resisted any theoretical understanding until recently [Stich et al., 2018, Alistarh et al., 2018]. However, all existing analyses either i) apply to the single node setting only, ii) rely on very strong and often unreasonable assumptions, such global boundedness of the gradients, or iterate-dependent assumptions that cannot be checked a-priori and may not hold in practice, or iii) circumvent these issues via the introduction of additional unbiased compressors, which increase the communication cost. In this work we fix all these deficiencies by proposing and analyzing a new EF mechanism, which we call EF21, which consistently and substantially outperforms EF in practice. Our theoretical analysis relies on standard assumptions only, works in the distributed heterogeneous data setting, and leads to better and more meaningful rates. In particular, we prove that EF21 enjoys a fast $O(1/T)$ convergence rate for smooth nonconvex problems, beating the previous bound of $O(1/T^{2/3})$, which was shown a bounded gradients assumption. We further improve this to a fast linear rate for PL functions, which is the first linear convergence result for an EF-type method not relying on unbiased compressors. Since EF has a large number of applications where it reigns supreme, we believe that our 2021 variant, EF21, can a large impact on the practice of communication efficient distributed learning." @default.
• W3166972615 created "2021-06-22" @default.
• W3166972615 creator A5003899242 @default.
• W3166972615 creator A5036598221 @default.
• W3166972615 creator A5090088069 @default.
• W3166972615 date "2021-06-09" @default.
• W3166972615 modified "2023-09-27" @default.
• W3166972615 title "EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback" @default.
• W3166972615 cites W2153635508 @default.
• W3166972615 cites W2194775991 @default.
• W3166972615 cites W2407022425 @default.
• W3166972615 cites W2808495300 @default.
• W3166972615 cites W2890924858 @default.
• W3166972615 cites W2911863041 @default.
• W3166972615 cites W2914138317 @default.
• W3166972615 cites W2935990182 @default.
• W3166972615 cites W2946198813 @default.
• W3166972615 cites W2946815927 @default.
• W3166972615 cites W2954365017 @default.
• W3166972615 cites W2963228337 @default.
• W3166972615 cites W2963376662 @default.
• W3166972615 cites W2963766684 @default.
• W3166972615 cites W2970971581 @default.
• W3166972615 cites W2971064744 @default.
• W3166972615 cites W2973096819 @default.
• W3166972615 cites W2996118901 @default.
• W3166972615 cites W3007528421 @default.
• W3166972615 cites W3012125688 @default.
• W3166972615 cites W3023387899 @default.
• W3166972615 cites W3034553394 @default.
• W3166972615 cites W3036623140 @default.
• W3166972615 cites W3037346940 @default.
• W3166972615 cites W3037842861 @default.
• W3166972615 cites W3080910898 @default.
• W3166972615 cites W3090675293 @default.
• W3166972615 cites W3100770245 @default.
• W3166972615 cites W3113984671 @default.
• W3166972615 cites W3118608800 @default.
• W3166972615 cites W3124367822 @default.
• W3166972615 cites W3130457203 @default.
• W3166972615 cites W3132163419 @default.
• W3166972615 cites W3133646186 @default.
• W3166972615 cites W3154043723 @default.
• W3166972615 cites W3161543042 @default.
• W3166972615 cites W3089975361 @default.
• W3166972615 hasPublicationYear "2021" @default.
• W3166972615 type Work @default.
• W3166972615 sameAs 3166972615 @default.
• W3166972615 citedByCount "6" @default.
• W3166972615 countsByYear W31669726152020 @default.
• W3166972615 countsByYear W31669726152021 @default.
• W3166972615 crossrefType "posted-content" @default.
• W3166972615 hasAuthorship W3166972615A5003899242 @default.
• W3166972615 hasAuthorship W3166972615A5036598221 @default.
• W3166972615 hasAuthorship W3166972615A5090088069 @default.
• W3166972615 hasConcept C111472728 @default.
• W3166972615 hasConcept C11171543 @default.
• W3166972615 hasConcept C126255220 @default.
• W3166972615 hasConcept C127413603 @default.
• W3166972615 hasConcept C134306372 @default.
• W3166972615 hasConcept C138885662 @default.
• W3166972615 hasConcept C151730666 @default.
• W3166972615 hasConcept C154945302 @default.
• W3166972615 hasConcept C15744967 @default.
• W3166972615 hasConcept C162324750 @default.
• W3166972615 hasConcept C173801870 @default.
• W3166972615 hasConcept C26517878 @default.
• W3166972615 hasConcept C2777303404 @default.
• W3166972615 hasConcept C2779343474 @default.
• W3166972615 hasConcept C2780023022 @default.
• W3166972615 hasConcept C33923547 @default.
• W3166972615 hasConcept C34388435 @default.
• W3166972615 hasConcept C38652104 @default.
• W3166972615 hasConcept C41008148 @default.
• W3166972615 hasConcept C50522688 @default.
• W3166972615 hasConcept C57869625 @default.
• W3166972615 hasConcept C62611344 @default.
• W3166972615 hasConcept C66938386 @default.
• W3166972615 hasConcept C75553542 @default.
• W3166972615 hasConcept C77553402 @default.
• W3166972615 hasConcept C86803240 @default.
• W3166972615 hasConceptScore W3166972615C111472728 @default.
• W3166972615 hasConceptScore W3166972615C11171543 @default.
• W3166972615 hasConceptScore W3166972615C126255220 @default.
• W3166972615 hasConceptScore W3166972615C127413603 @default.
• W3166972615 hasConceptScore W3166972615C134306372 @default.
• W3166972615 hasConceptScore W3166972615C138885662 @default.
• W3166972615 hasConceptScore W3166972615C151730666 @default.
• W3166972615 hasConceptScore W3166972615C154945302 @default.
• W3166972615 hasConceptScore W3166972615C15744967 @default.
• W3166972615 hasConceptScore W3166972615C162324750 @default.
• W3166972615 hasConceptScore W3166972615C173801870 @default.
• W3166972615 hasConceptScore W3166972615C26517878 @default.
• W3166972615 hasConceptScore W3166972615C2777303404 @default.
• W3166972615 hasConceptScore W3166972615C2779343474 @default.
• W3166972615 hasConceptScore W3166972615C2780023022 @default.
• W3166972615 hasConceptScore W3166972615C33923547 @default.
• W3166972615 hasConceptScore W3166972615C34388435 @default.
• W3166972615 hasConceptScore W3166972615C38652104 @default.
• W3166972615 hasConceptScore W3166972615C41008148 @default.

• next