Matches in SemOpenAlex for { <https://semopenalex.org/work/W4281656148> ?p ?o ?g. }
Showing items 1 to 95 of
95
with 100 items per page.
- W4281656148 abstract "In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. We consider the setting in which all workers sample from the same dataset, and communicate over a sparse graph (decentralized). In this setting, current theory fails to capture important aspects of real-world behavior. First, the 'spectral gap' of the communication graph is not predictive of its empirical performance in (deep) learning. Second, current theory does not explain that collaboration enables larger learning rates than training alone. In fact, it prescribes smaller learning rates, which further decrease as graphs become larger, failing to explain convergence in infinite graphs. This paper aims to paint an accurate picture of sparsely-connected distributed optimization when workers share the same data distribution. We quantify how the graph topology influences convergence in a quadratic toy problem and provide theoretical results for general smooth and (strongly) convex objectives. Our theory matches empirical observations in deep learning, and accurately describes the relative merits of different graph topologies." @default.
- W4281656148 created "2022-06-13" @default.
- W4281656148 creator A5020200055 @default.
- W4281656148 creator A5073756389 @default.
- W4281656148 creator A5080762759 @default.
- W4281656148 date "2022-06-07" @default.
- W4281656148 modified "2023-09-29" @default.
- W4281656148 title "Beyond spectral gap: The role of the topology in decentralized learning" @default.
- W4281656148 doi "https://doi.org/10.48550/arxiv.2206.03093" @default.
- W4281656148 hasPublicationYear "2022" @default.
- W4281656148 type Work @default.
- W4281656148 citedByCount "0" @default.
- W4281656148 crossrefType "posted-content" @default.
- W4281656148 hasAuthorship W4281656148A5020200055 @default.
- W4281656148 hasAuthorship W4281656148A5073756389 @default.
- W4281656148 hasAuthorship W4281656148A5080762759 @default.
- W4281656148 hasBestOaLocation W42816561481 @default.
- W4281656148 hasConcept C105795698 @default.
- W4281656148 hasConcept C111919701 @default.
- W4281656148 hasConcept C112680207 @default.
- W4281656148 hasConcept C114614502 @default.
- W4281656148 hasConcept C119857082 @default.
- W4281656148 hasConcept C120936955 @default.
- W4281656148 hasConcept C126255220 @default.
- W4281656148 hasConcept C127162648 @default.
- W4281656148 hasConcept C129844170 @default.
- W4281656148 hasConcept C132525143 @default.
- W4281656148 hasConcept C134306372 @default.
- W4281656148 hasConcept C154945302 @default.
- W4281656148 hasConcept C157406716 @default.
- W4281656148 hasConcept C15744967 @default.
- W4281656148 hasConcept C162324750 @default.
- W4281656148 hasConcept C184720557 @default.
- W4281656148 hasConcept C19417346 @default.
- W4281656148 hasConcept C199845137 @default.
- W4281656148 hasConcept C203776342 @default.
- W4281656148 hasConcept C22149727 @default.
- W4281656148 hasConcept C2524010 @default.
- W4281656148 hasConcept C2777303404 @default.
- W4281656148 hasConcept C2778114796 @default.
- W4281656148 hasConcept C2779582901 @default.
- W4281656148 hasConcept C31258907 @default.
- W4281656148 hasConcept C33923547 @default.
- W4281656148 hasConcept C41008148 @default.
- W4281656148 hasConcept C50522688 @default.
- W4281656148 hasConcept C57869625 @default.
- W4281656148 hasConcept C80444323 @default.
- W4281656148 hasConcept C88230418 @default.
- W4281656148 hasConceptScore W4281656148C105795698 @default.
- W4281656148 hasConceptScore W4281656148C111919701 @default.
- W4281656148 hasConceptScore W4281656148C112680207 @default.
- W4281656148 hasConceptScore W4281656148C114614502 @default.
- W4281656148 hasConceptScore W4281656148C119857082 @default.
- W4281656148 hasConceptScore W4281656148C120936955 @default.
- W4281656148 hasConceptScore W4281656148C126255220 @default.
- W4281656148 hasConceptScore W4281656148C127162648 @default.
- W4281656148 hasConceptScore W4281656148C129844170 @default.
- W4281656148 hasConceptScore W4281656148C132525143 @default.
- W4281656148 hasConceptScore W4281656148C134306372 @default.
- W4281656148 hasConceptScore W4281656148C154945302 @default.
- W4281656148 hasConceptScore W4281656148C157406716 @default.
- W4281656148 hasConceptScore W4281656148C15744967 @default.
- W4281656148 hasConceptScore W4281656148C162324750 @default.
- W4281656148 hasConceptScore W4281656148C184720557 @default.
- W4281656148 hasConceptScore W4281656148C19417346 @default.
- W4281656148 hasConceptScore W4281656148C199845137 @default.
- W4281656148 hasConceptScore W4281656148C203776342 @default.
- W4281656148 hasConceptScore W4281656148C22149727 @default.
- W4281656148 hasConceptScore W4281656148C2524010 @default.
- W4281656148 hasConceptScore W4281656148C2777303404 @default.
- W4281656148 hasConceptScore W4281656148C2778114796 @default.
- W4281656148 hasConceptScore W4281656148C2779582901 @default.
- W4281656148 hasConceptScore W4281656148C31258907 @default.
- W4281656148 hasConceptScore W4281656148C33923547 @default.
- W4281656148 hasConceptScore W4281656148C41008148 @default.
- W4281656148 hasConceptScore W4281656148C50522688 @default.
- W4281656148 hasConceptScore W4281656148C57869625 @default.
- W4281656148 hasConceptScore W4281656148C80444323 @default.
- W4281656148 hasConceptScore W4281656148C88230418 @default.
- W4281656148 hasLocation W42816561481 @default.
- W4281656148 hasOpenAccess W4281656148 @default.
- W4281656148 hasPrimaryLocation W42816561481 @default.
- W4281656148 hasRelatedWork W1706309217 @default.
- W4281656148 hasRelatedWork W2092358244 @default.
- W4281656148 hasRelatedWork W2341464167 @default.
- W4281656148 hasRelatedWork W2508017964 @default.
- W4281656148 hasRelatedWork W2908860425 @default.
- W4281656148 hasRelatedWork W2949219073 @default.
- W4281656148 hasRelatedWork W2968850319 @default.
- W4281656148 hasRelatedWork W3186127753 @default.
- W4281656148 hasRelatedWork W3217310956 @default.
- W4281656148 hasRelatedWork W4301950041 @default.
- W4281656148 isParatext "false" @default.
- W4281656148 isRetracted "false" @default.
- W4281656148 workType "article" @default.