Matches in SemOpenAlex for { <https://semopenalex.org/work/W2786899637> ?p ?o ?g. }
- W2786899637 abstract "It has been experimentally observed that distributed implementations of mini-batch stochastic gradient descent (SGD) algorithms exhibit speedup saturation and decaying generalization ability beyond a particular batch-size. In this work, we present an analysis hinting that high similarity between concurrently processed gradients may be a cause of this performance degradation. We introduce the notion of gradient diversity that measures the dissimilarity between concurrent gradient updates, and show its key role in the performance of mini-batch SGD. We prove that on problems with high gradient diversity, mini-batch SGD is amenable to better speedups, while maintaining the generalization performance of serial (one sample) SGD. We further establish lower bounds on convergence where mini-batch SGD slows down beyond a particular batch-size, solely due to the lack of gradient diversity. We provide experimental evidence indicating the key role of gradient diversity in distributed learning, and discuss how heuristics like dropout, Langevin dynamics, and quantization can improve it." @default.
- W2786899637 created "2018-02-23" @default.
- W2786899637 creator A5001139821 @default.
- W2786899637 creator A5018630988 @default.
- W2786899637 creator A5027263026 @default.
- W2786899637 creator A5030261391 @default.
- W2786899637 creator A5081804374 @default.
- W2786899637 creator A5085869933 @default.
- W2786899637 date "2017-06-18" @default.
- W2786899637 modified "2023-09-27" @default.
- W2786899637 title "Gradient Diversity: a Key Ingredient for Scalable Distributed Learning" @default.
- W2786899637 cites W1442374986 @default.
- W2786899637 cites W1636243882 @default.
- W2786899637 cites W1697545848 @default.
- W2786899637 cites W1780115997 @default.
- W2786899637 cites W193761913 @default.
- W2786899637 cites W1980147176 @default.
- W2786899637 cites W1987083649 @default.
- W2786899637 cites W2035720976 @default.
- W2786899637 cites W2072566913 @default.
- W2786899637 cites W2095705004 @default.
- W2786899637 cites W2100007248 @default.
- W2786899637 cites W2103628399 @default.
- W2786899637 cites W2106709023 @default.
- W2786899637 cites W2111453803 @default.
- W2786899637 cites W2130062883 @default.
- W2786899637 cites W2139338362 @default.
- W2786899637 cites W2144839971 @default.
- W2786899637 cites W2166706236 @default.
- W2786899637 cites W2167433878 @default.
- W2786899637 cites W2168231600 @default.
- W2786899637 cites W2186615578 @default.
- W2786899637 cites W2194775991 @default.
- W2786899637 cites W2235185944 @default.
- W2786899637 cites W2263490141 @default.
- W2786899637 cites W2271840356 @default.
- W2786899637 cites W2336650964 @default.
- W2786899637 cites W2436219157 @default.
- W2786899637 cites W2508844316 @default.
- W2786899637 cites W2523060838 @default.
- W2786899637 cites W2592206372 @default.
- W2786899637 cites W2611778843 @default.
- W2786899637 cites W2617019596 @default.
- W2786899637 cites W2622263826 @default.
- W2786899637 cites W27434444 @default.
- W2786899637 cites W2751791039 @default.
- W2786899637 cites W2752512710 @default.
- W2786899637 cites W2949212702 @default.
- W2786899637 cites W2951101735 @default.
- W2786899637 cites W2951488730 @default.
- W2786899637 cites W2951781666 @default.
- W2786899637 cites W2951833684 @default.
- W2786899637 cites W2953281140 @default.
- W2786899637 cites W2963248893 @default.
- W2786899637 cites W2963794891 @default.
- W2786899637 cites W2963992805 @default.
- W2786899637 cites W2964106499 @default.
- W2786899637 cites W3103657382 @default.
- W2786899637 cites W4919037 @default.
- W2786899637 hasPublicationYear "2017" @default.
- W2786899637 type Work @default.
- W2786899637 sameAs 2786899637 @default.
- W2786899637 citedByCount "12" @default.
- W2786899637 countsByYear W27868996372017 @default.
- W2786899637 countsByYear W27868996372018 @default.
- W2786899637 countsByYear W27868996372019 @default.
- W2786899637 countsByYear W27868996372020 @default.
- W2786899637 countsByYear W27868996372021 @default.
- W2786899637 crossrefType "posted-content" @default.
- W2786899637 hasAuthorship W2786899637A5001139821 @default.
- W2786899637 hasAuthorship W2786899637A5018630988 @default.
- W2786899637 hasAuthorship W2786899637A5027263026 @default.
- W2786899637 hasAuthorship W2786899637A5030261391 @default.
- W2786899637 hasAuthorship W2786899637A5081804374 @default.
- W2786899637 hasAuthorship W2786899637A5085869933 @default.
- W2786899637 hasConcept C105795698 @default.
- W2786899637 hasConcept C111919701 @default.
- W2786899637 hasConcept C11413529 @default.
- W2786899637 hasConcept C127705205 @default.
- W2786899637 hasConcept C134306372 @default.
- W2786899637 hasConcept C153258448 @default.
- W2786899637 hasConcept C154945302 @default.
- W2786899637 hasConcept C162324750 @default.
- W2786899637 hasConcept C173608175 @default.
- W2786899637 hasConcept C177148314 @default.
- W2786899637 hasConcept C206688291 @default.
- W2786899637 hasConcept C26517878 @default.
- W2786899637 hasConcept C2777303404 @default.
- W2786899637 hasConcept C2780004032 @default.
- W2786899637 hasConcept C33923547 @default.
- W2786899637 hasConcept C38652104 @default.
- W2786899637 hasConcept C41008148 @default.
- W2786899637 hasConcept C48044578 @default.
- W2786899637 hasConcept C50522688 @default.
- W2786899637 hasConcept C50644808 @default.
- W2786899637 hasConcept C68339613 @default.
- W2786899637 hasConcept C77088390 @default.
- W2786899637 hasConceptScore W2786899637C105795698 @default.
- W2786899637 hasConceptScore W2786899637C111919701 @default.
- W2786899637 hasConceptScore W2786899637C11413529 @default.