Matches in SemOpenAlex for { <https://semopenalex.org/work/W2940453604> ?p ?o ?g. }
- W2940453604 abstract "Synchronous mini-batch SGD is state-of-the-art for large-scale distributed machine learning. However, in practice, its convergence is bottlenecked by slow communication rounds between worker nodes. A natural solution to reduce communication is to use the emph{`local-SGD'} model in which the workers train their model independently and synchronize every once in a while. This algorithm improves the computation-communication trade-off but its convergence is not understood very well. We propose a non-asymptotic error analysis, which enables comparison to emph{one-shot averaging} i.e., a single communication round among independent workers, and emph{mini-batch averaging} i.e., communicating at every step. We also provide adaptive lower bounds on the communication frequency for large step-sizes ($ t^{-alpha} $, $ alphain (1/2 , 1 ) $) and show that emph{Local-SGD} reduces communication by a factor of $OBig(frac{sqrt{T}}{P^{3/2}}Big)$, with $T$ the total number of gradients and $P$ machines." @default.
- W2940453604 created "2019-05-03" @default.
- W2940453604 creator A5023396823 @default.
- W2940453604 creator A5056989506 @default.
- W2940453604 date "2019-04-25" @default.
- W2940453604 modified "2023-10-01" @default.
- W2940453604 title "Communication trade-offs for synchronized distributed SGD with large step size" @default.
- W2940453604 cites W1501362825 @default.
- W2940453604 cites W1516903196 @default.
- W2940453604 cites W1518479949 @default.
- W2940453604 cites W1626317705 @default.
- W2940453604 cites W182774073 @default.
- W2940453604 cites W1842094663 @default.
- W2940453604 cites W1845277745 @default.
- W2940453604 cites W1877037013 @default.
- W2940453604 cites W1919113324 @default.
- W2940453604 cites W1980412153 @default.
- W2940453604 cites W1992208280 @default.
- W2940453604 cites W1999083556 @default.
- W2940453604 cites W2028683648 @default.
- W2940453604 cites W2066623346 @default.
- W2940453604 cites W2072566913 @default.
- W2940453604 cites W2078670469 @default.
- W2940453604 cites W2086161653 @default.
- W2940453604 cites W2091825929 @default.
- W2940453604 cites W2103971893 @default.
- W2940453604 cites W2112269233 @default.
- W2940453604 cites W2112284959 @default.
- W2940453604 cites W2113547287 @default.
- W2940453604 cites W2116612304 @default.
- W2940453604 cites W2117499659 @default.
- W2940453604 cites W2127941149 @default.
- W2940453604 cites W2130062883 @default.
- W2940453604 cites W2138243089 @default.
- W2940453604 cites W2138302120 @default.
- W2940453604 cites W2138855972 @default.
- W2940453604 cites W2144839971 @default.
- W2940453604 cites W2156779765 @default.
- W2940453604 cites W2159930037 @default.
- W2940453604 cites W2162287622 @default.
- W2940453604 cites W2162576315 @default.
- W2940453604 cites W2164278908 @default.
- W2940453604 cites W2165905842 @default.
- W2940453604 cites W2166706236 @default.
- W2940453604 cites W2169342250 @default.
- W2940453604 cites W2199097987 @default.
- W2940453604 cites W2235185944 @default.
- W2940453604 cites W2238591594 @default.
- W2940453604 cites W2283463896 @default.
- W2940453604 cites W2460815216 @default.
- W2940453604 cites W2510078447 @default.
- W2940453604 cites W2513289469 @default.
- W2940453604 cites W2530874098 @default.
- W2940453604 cites W2531786980 @default.
- W2940453604 cites W2535838896 @default.
- W2940453604 cites W2556660792 @default.
- W2940453604 cites W2560804315 @default.
- W2940453604 cites W2571425027 @default.
- W2940453604 cites W2591577911 @default.
- W2940453604 cites W2598713166 @default.
- W2940453604 cites W2608239888 @default.
- W2940453604 cites W2617766261 @default.
- W2940453604 cites W2622263826 @default.
- W2940453604 cites W2734739919 @default.
- W2940453604 cites W2736068458 @default.
- W2940453604 cites W2737665727 @default.
- W2940453604 cites W2738275598 @default.
- W2940453604 cites W2741269719 @default.
- W2940453604 cites W2755172545 @default.
- W2940453604 cites W2766367231 @default.
- W2940453604 cites W2783257164 @default.
- W2940453604 cites W2794969685 @default.
- W2940453604 cites W2804227275 @default.
- W2940453604 cites W2807915830 @default.
- W2940453604 cites W2808495300 @default.
- W2940453604 cites W2886404071 @default.
- W2940453604 cites W2949198759 @default.
- W2940453604 cites W2949227242 @default.
- W2940453604 cites W2949585412 @default.
- W2940453604 cites W2951781666 @default.
- W2940453604 cites W2952033860 @default.
- W2940453604 cites W2952986481 @default.
- W2940453604 cites W2962863496 @default.
- W2940453604 cites W2962952793 @default.
- W2940453604 cites W2963228337 @default.
- W2940453604 cites W2963372166 @default.
- W2940453604 cites W2963374099 @default.
- W2940453604 cites W2963408258 @default.
- W2940453604 cites W2963422939 @default.
- W2940453604 cites W2963632834 @default.
- W2940453604 cites W2963959597 @default.
- W2940453604 cites W2963992805 @default.
- W2940453604 cites W3103741939 @default.
- W2940453604 cites W3141595720 @default.
- W2940453604 cites W59018853 @default.
- W2940453604 cites W778657980 @default.
- W2940453604 cites W88685657 @default.
- W2940453604 cites W2963587506 @default.
- W2940453604 hasPublicationYear "2019" @default.
- W2940453604 type Work @default.