FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2891970177> ?p ?o ?g. }

• W2891970177 abstract "Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, large-batch methods, and gradient sparsification, have been proposed. To date, gradient sparsification methods - where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally - are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to three orders of magnitude, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis and empirical validation also reveal that these methods do require analytical conditions to converge well, justifying existing heuristics." @default.
• W2891970177 created "2018-09-27" @default.
• W2891970177 creator A5024256519 @default.
• W2891970177 creator A5026990786 @default.
• W2891970177 creator A5027182415 @default.
• W2891970177 creator A5063262952 @default.
• W2891970177 creator A5083822059 @default.
• W2891970177 creator A5088091013 @default.
• W2891970177 date "2018-09-27" @default.
• W2891970177 modified "2023-09-27" @default.
• W2891970177 title "The Convergence of Sparsified Gradient Methods" @default.
• W2891970177 cites W1877037013 @default.
• W2891970177 cites W2079482358 @default.
• W2891970177 cites W2181607856 @default.
• W2891970177 cites W2186615578 @default.
• W2891970177 cites W2194775991 @default.
• W2891970177 cites W2405578611 @default.
• W2891970177 cites W2407022425 @default.
• W2891970177 cites W2563343794 @default.
• W2891970177 cites W2617766261 @default.
• W2891970177 cites W2622263826 @default.
• W2891970177 cites W2625885504 @default.
• W2891970177 cites W2757910899 @default.
• W2891970177 cites W2766140019 @default.
• W2891970177 cites W2789218400 @default.
• W2891970177 cites W2789516847 @default.
• W2891970177 cites W2951781666 @default.
• W2891970177 cites W2963803379 @default.
• W2891970177 cites W778657980 @default.
• W2891970177 cites W2701971652 @default.
• W2891970177 hasPublicationYear "2018" @default.
• W2891970177 type Work @default.
• W2891970177 sameAs 2891970177 @default.
• W2891970177 citedByCount "4" @default.
• W2891970177 countsByYear W28919701772020 @default.
• W2891970177 countsByYear W28919701772021 @default.
• W2891970177 countsByYear W28919701772023 @default.
• W2891970177 crossrefType "posted-content" @default.
• W2891970177 hasAuthorship W2891970177A5024256519 @default.
• W2891970177 hasAuthorship W2891970177A5026990786 @default.
• W2891970177 hasAuthorship W2891970177A5027182415 @default.
• W2891970177 hasAuthorship W2891970177A5063262952 @default.
• W2891970177 hasAuthorship W2891970177A5083822059 @default.
• W2891970177 hasAuthorship W2891970177A5088091013 @default.
• W2891970177 hasConcept C111335779 @default.
• W2891970177 hasConcept C112680207 @default.
• W2891970177 hasConcept C11413529 @default.
• W2891970177 hasConcept C121332964 @default.
• W2891970177 hasConcept C126255220 @default.
• W2891970177 hasConcept C126691448 @default.
• W2891970177 hasConcept C127413603 @default.
• W2891970177 hasConcept C1276947 @default.
• W2891970177 hasConcept C127705205 @default.
• W2891970177 hasConcept C153258448 @default.
• W2891970177 hasConcept C154945302 @default.
• W2891970177 hasConcept C162324750 @default.
• W2891970177 hasConcept C206688291 @default.
• W2891970177 hasConcept C2524010 @default.
• W2891970177 hasConcept C2777303404 @default.
• W2891970177 hasConcept C28826006 @default.
• W2891970177 hasConcept C28855332 @default.
• W2891970177 hasConcept C33923547 @default.
• W2891970177 hasConcept C41008148 @default.
• W2891970177 hasConcept C50522688 @default.
• W2891970177 hasConcept C50644808 @default.
• W2891970177 hasConcept C62611344 @default.
• W2891970177 hasConcept C66938386 @default.
• W2891970177 hasConcept C81917197 @default.
• W2891970177 hasConceptScore W2891970177C111335779 @default.
• W2891970177 hasConceptScore W2891970177C112680207 @default.
• W2891970177 hasConceptScore W2891970177C11413529 @default.
• W2891970177 hasConceptScore W2891970177C121332964 @default.
• W2891970177 hasConceptScore W2891970177C126255220 @default.
• W2891970177 hasConceptScore W2891970177C126691448 @default.
• W2891970177 hasConceptScore W2891970177C127413603 @default.
• W2891970177 hasConceptScore W2891970177C1276947 @default.
• W2891970177 hasConceptScore W2891970177C127705205 @default.
• W2891970177 hasConceptScore W2891970177C153258448 @default.
• W2891970177 hasConceptScore W2891970177C154945302 @default.
• W2891970177 hasConceptScore W2891970177C162324750 @default.
• W2891970177 hasConceptScore W2891970177C206688291 @default.
• W2891970177 hasConceptScore W2891970177C2524010 @default.
• W2891970177 hasConceptScore W2891970177C2777303404 @default.
• W2891970177 hasConceptScore W2891970177C28826006 @default.
• W2891970177 hasConceptScore W2891970177C28855332 @default.
• W2891970177 hasConceptScore W2891970177C33923547 @default.
• W2891970177 hasConceptScore W2891970177C41008148 @default.
• W2891970177 hasConceptScore W2891970177C50522688 @default.
• W2891970177 hasConceptScore W2891970177C50644808 @default.
• W2891970177 hasConceptScore W2891970177C62611344 @default.
• W2891970177 hasConceptScore W2891970177C66938386 @default.
• W2891970177 hasConceptScore W2891970177C81917197 @default.
• W2891970177 hasLocation W28919701771 @default.
• W2891970177 hasOpenAccess W2891970177 @default.
• W2891970177 hasPrimaryLocation W28919701771 @default.
• W2891970177 hasRelatedWork W2136667596 @default.
• W2891970177 hasRelatedWork W2484096406 @default.
• W2891970177 hasRelatedWork W2618001342 @default.
• W2891970177 hasRelatedWork W2741938793 @default.
• W2891970177 hasRelatedWork W2751113853 @default.

• next