FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W3096004388> ?p ?o ?g. }

• W3096004388 abstract "Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use gradient compression techniques like sparsification, quantization, or low-rank updates. The techniques usually require choosing a static compression ratio, often requiring users to balance the trade-off between model accuracy and per-iteration speedup. In this work, we show that such performance degradation due to choosing a high compression ratio is not fundamental. An adaptive compression strategy can reduce communication while maintaining final test accuracy. Inspired by recent findings on critical learning regimes, in which small gradient errors can have irrecoverable impact on model performance, we propose Accordion a simple yet effective adaptive compression algorithm. While Accordion maintains a high enough compression rate on average, it avoids over-compressing gradients whenever in critical learning regimes, detected by a simple gradient-norm based criterion. Our extensive experimental study over a number of machine learning tasks in distributed environments indicates that Accordion, maintains similar model accuracy to uncompressed training, yet achieves up to 5.5x better compression and up to 4.1x end-to-end speedup over static approaches. We show that Accordion also works for adjusting the batch size, another popular strategy for alleviating communication bottlenecks." @default.
• W3096004388 created "2020-11-09" @default.
• W3096004388 creator A5018630988 @default.
• W3096004388 creator A5049494334 @default.
• W3096004388 creator A5073193050 @default.
• W3096004388 creator A5082710354 @default.
• W3096004388 creator A5089058934 @default.
• W3096004388 date "2020-10-29" @default.
• W3096004388 modified "2023-09-26" @default.
• W3096004388 title "Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification." @default.
• W3096004388 cites W1686810756 @default.
• W3096004388 cites W2097117768 @default.
• W3096004388 cites W2135046866 @default.
• W3096004388 cites W2168231600 @default.
• W3096004388 cites W2194775991 @default.
• W3096004388 cites W2407022425 @default.
• W3096004388 cites W2622263826 @default.
• W3096004388 cites W2699326867 @default.
• W3096004388 cites W2749988060 @default.
• W3096004388 cites W2769644379 @default.
• W3096004388 cites W2773689216 @default.
• W3096004388 cites W2787998955 @default.
• W3096004388 cites W2895139741 @default.
• W3096004388 cites W2900167092 @default.
• W3096004388 cites W2902280036 @default.
• W3096004388 cites W2904243021 @default.
• W3096004388 cites W2911040189 @default.
• W3096004388 cites W2916967211 @default.
• W3096004388 cites W2962786385 @default.
• W3096004388 cites W2962819303 @default.
• W3096004388 cites W2962911728 @default.
• W3096004388 cites W2963112712 @default.
• W3096004388 cites W2963179579 @default.
• W3096004388 cites W2963420686 @default.
• W3096004388 cites W2963446712 @default.
• W3096004388 cites W2963540381 @default.
• W3096004388 cites W2963655672 @default.
• W3096004388 cites W2963664311 @default.
• W3096004388 cites W2963702144 @default.
• W3096004388 cites W2963803379 @default.
• W3096004388 cites W2963959597 @default.
• W3096004388 cites W2964004663 @default.
• W3096004388 cites W2964346549 @default.
• W3096004388 cites W2970289928 @default.
• W3096004388 cites W2970971581 @default.
• W3096004388 cites W2973096819 @default.
• W3096004388 cites W2982475424 @default.
• W3096004388 cites W2990352720 @default.
• W3096004388 cites W2994779554 @default.
• W3096004388 cites W2996309822 @default.
• W3096004388 cites W2996642285 @default.
• W3096004388 cites W3013308737 @default.
• W3096004388 cites W3015585426 @default.
• W3096004388 cites W3016081801 @default.
• W3096004388 cites W3037047862 @default.
• W3096004388 cites W3038098022 @default.
• W3096004388 cites W3038103902 @default.
• W3096004388 cites W3101036738 @default.
• W3096004388 cites W3118608800 @default.
• W3096004388 hasPublicationYear "2020" @default.
• W3096004388 type Work @default.
• W3096004388 sameAs 3096004388 @default.
• W3096004388 citedByCount "6" @default.
• W3096004388 countsByYear W30960043882021 @default.
• W3096004388 crossrefType "posted-content" @default.
• W3096004388 hasAuthorship W3096004388A5018630988 @default.
• W3096004388 hasAuthorship W3096004388A5049494334 @default.
• W3096004388 hasAuthorship W3096004388A5073193050 @default.
• W3096004388 hasAuthorship W3096004388A5082710354 @default.
• W3096004388 hasAuthorship W3096004388A5089058934 @default.
• W3096004388 hasConcept C11413529 @default.
• W3096004388 hasConcept C115961682 @default.
• W3096004388 hasConcept C127413603 @default.
• W3096004388 hasConcept C13481523 @default.
• W3096004388 hasConcept C136764020 @default.
• W3096004388 hasConcept C154945302 @default.
• W3096004388 hasConcept C171146098 @default.
• W3096004388 hasConcept C173608175 @default.
• W3096004388 hasConcept C25797200 @default.
• W3096004388 hasConcept C28855332 @default.
• W3096004388 hasConcept C41008148 @default.
• W3096004388 hasConcept C511840579 @default.
• W3096004388 hasConcept C68339613 @default.
• W3096004388 hasConcept C87732919 @default.
• W3096004388 hasConcept C9417928 @default.
• W3096004388 hasConcept C94835093 @default.
• W3096004388 hasConceptScore W3096004388C11413529 @default.
• W3096004388 hasConceptScore W3096004388C115961682 @default.
• W3096004388 hasConceptScore W3096004388C127413603 @default.
• W3096004388 hasConceptScore W3096004388C13481523 @default.
• W3096004388 hasConceptScore W3096004388C136764020 @default.
• W3096004388 hasConceptScore W3096004388C154945302 @default.
• W3096004388 hasConceptScore W3096004388C171146098 @default.
• W3096004388 hasConceptScore W3096004388C173608175 @default.
• W3096004388 hasConceptScore W3096004388C25797200 @default.
• W3096004388 hasConceptScore W3096004388C28855332 @default.
• W3096004388 hasConceptScore W3096004388C41008148 @default.
• W3096004388 hasConceptScore W3096004388C511840579 @default.
• W3096004388 hasConceptScore W3096004388C68339613 @default.
• W3096004388 hasConceptScore W3096004388C87732919 @default.

• next