Matches in SemOpenAlex for { <https://semopenalex.org/work/W4378510484> ?p ?o ?g. }
Showing items 1 to 75 of
75
with 100 items per page.
- W4378510484 abstract "Due to the high communication overhead when training machine learning models in a distributed environment, modern algorithms invariably rely on lossy communication compression. However, when untreated, the errors caused by compression propagate, and can lead to severely unstable behavior, including exponential divergence. Almost a decade ago, Seide et al [2014] proposed an error feedback (EF) mechanism, which we refer to as EF14, as an immensely effective heuristic for mitigating this issue. However, despite steady algorithmic and theoretical advances in the EF field in the last decade, our understanding is far from complete. In this work we address one of the most pressing issues. In particular, in the canonical nonconvex setting, all known variants of EF rely on very large batch sizes to converge, which can be prohibitive in practice. We propose a surprisingly simple fix which removes this issue both theoretically, and in practice: the application of Polyak's momentum to the latest incarnation of EF due to Richt'{a}rik et al. [2021] known as EF21. Our algorithm, for which we coin the name EF21-SGDM, improves the communication and sample complexities of previous error feedback algorithms under standard smoothness and bounded variance assumptions, and does not require any further strong assumptions such as bounded gradient dissimilarity. Moreover, we propose a double momentum version of our method that improves the complexities even further. Our proof seems to be novel even when compression is removed from the method, and as such, our proof technique is of independent interest in the study of nonconvex stochastic optimization enriched with Polyak's momentum." @default.
- W4378510484 created "2023-05-27" @default.
- W4378510484 creator A5003899242 @default.
- W4378510484 creator A5036598221 @default.
- W4378510484 creator A5070820740 @default.
- W4378510484 date "2023-05-24" @default.
- W4378510484 modified "2023-09-30" @default.
- W4378510484 title "Momentum Provably Improves Error Feedback!" @default.
- W4378510484 doi "https://doi.org/10.48550/arxiv.2305.15155" @default.
- W4378510484 hasPublicationYear "2023" @default.
- W4378510484 type Work @default.
- W4378510484 citedByCount "0" @default.
- W4378510484 crossrefType "posted-content" @default.
- W4378510484 hasAuthorship W4378510484A5003899242 @default.
- W4378510484 hasAuthorship W4378510484A5036598221 @default.
- W4378510484 hasAuthorship W4378510484A5070820740 @default.
- W4378510484 hasBestOaLocation W43785104841 @default.
- W4378510484 hasConcept C10138342 @default.
- W4378510484 hasConcept C102634674 @default.
- W4378510484 hasConcept C111472728 @default.
- W4378510484 hasConcept C111919701 @default.
- W4378510484 hasConcept C11413529 @default.
- W4378510484 hasConcept C121955636 @default.
- W4378510484 hasConcept C126255220 @default.
- W4378510484 hasConcept C134306372 @default.
- W4378510484 hasConcept C138885662 @default.
- W4378510484 hasConcept C144133560 @default.
- W4378510484 hasConcept C154945302 @default.
- W4378510484 hasConcept C162324750 @default.
- W4378510484 hasConcept C165021410 @default.
- W4378510484 hasConcept C173801870 @default.
- W4378510484 hasConcept C196083921 @default.
- W4378510484 hasConcept C2779960059 @default.
- W4378510484 hasConcept C2780586882 @default.
- W4378510484 hasConcept C33923547 @default.
- W4378510484 hasConcept C34388435 @default.
- W4378510484 hasConcept C41008148 @default.
- W4378510484 hasConcept C60718061 @default.
- W4378510484 hasConceptScore W4378510484C10138342 @default.
- W4378510484 hasConceptScore W4378510484C102634674 @default.
- W4378510484 hasConceptScore W4378510484C111472728 @default.
- W4378510484 hasConceptScore W4378510484C111919701 @default.
- W4378510484 hasConceptScore W4378510484C11413529 @default.
- W4378510484 hasConceptScore W4378510484C121955636 @default.
- W4378510484 hasConceptScore W4378510484C126255220 @default.
- W4378510484 hasConceptScore W4378510484C134306372 @default.
- W4378510484 hasConceptScore W4378510484C138885662 @default.
- W4378510484 hasConceptScore W4378510484C144133560 @default.
- W4378510484 hasConceptScore W4378510484C154945302 @default.
- W4378510484 hasConceptScore W4378510484C162324750 @default.
- W4378510484 hasConceptScore W4378510484C165021410 @default.
- W4378510484 hasConceptScore W4378510484C173801870 @default.
- W4378510484 hasConceptScore W4378510484C196083921 @default.
- W4378510484 hasConceptScore W4378510484C2779960059 @default.
- W4378510484 hasConceptScore W4378510484C2780586882 @default.
- W4378510484 hasConceptScore W4378510484C33923547 @default.
- W4378510484 hasConceptScore W4378510484C34388435 @default.
- W4378510484 hasConceptScore W4378510484C41008148 @default.
- W4378510484 hasConceptScore W4378510484C60718061 @default.
- W4378510484 hasLocation W43785104841 @default.
- W4378510484 hasOpenAccess W4378510484 @default.
- W4378510484 hasPrimaryLocation W43785104841 @default.
- W4378510484 hasRelatedWork W1979322129 @default.
- W4378510484 hasRelatedWork W2028024605 @default.
- W4378510484 hasRelatedWork W2075948250 @default.
- W4378510484 hasRelatedWork W2230691193 @default.
- W4378510484 hasRelatedWork W2327217847 @default.
- W4378510484 hasRelatedWork W2357857148 @default.
- W4378510484 hasRelatedWork W2800834205 @default.
- W4378510484 hasRelatedWork W4238075012 @default.
- W4378510484 hasRelatedWork W4248389398 @default.
- W4378510484 hasRelatedWork W2247596074 @default.
- W4378510484 isParatext "false" @default.
- W4378510484 isRetracted "false" @default.
- W4378510484 workType "article" @default.