Matches in SemOpenAlex for { <https://semopenalex.org/work/W2913357890> ?p ?o ?g. }
- W2913357890 abstract "Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stochastic oracle model where noisy estimates of the gradients are available. For strongly convex problems, we show that the distribution of the iterates of AG converges with the accelerated $O(sqrt{kappa}log(1/varepsilon))$ linear rate to a ball of radius $varepsilon$ centered at a unique invariant distribution in the 1-Wasserstein metric where $kappa$ is the condition number as long as the noise variance is smaller than an explicit upper bound we can provide. Our analysis also certifies linear convergence rates as a function of the stepsize, momentum parameter and the noise variance; recovering the accelerated rates in the noiseless case and quantifying the level of noise that can be tolerated to achieve a given performance. In the special case of strongly convex quadratic objectives, we can show accelerated linear rates in the $p$-Wasserstein metric for any $pgeq 1$ with improved sensitivity to noise for both AG and HB through a non-asymptotic analysis under some additional assumptions on the noise structure. Our analysis for HB and AG also leads to improved non-asymptotic convergence bounds in suboptimality for both deterministic and stochastic settings which is of independent interest. To the best of our knowledge, these are the first linear convergence results for stochastic momentum methods under the stochastic oracle model. We also extend our results to the APG method and weakly convex functions showing accelerated rates when the noise magnitude is sufficiently small." @default.
- W2913357890 created "2019-02-21" @default.
- W2913357890 creator A5025900972 @default.
- W2913357890 creator A5034370705 @default.
- W2913357890 creator A5084475092 @default.
- W2913357890 date "2019-01-22" @default.
- W2913357890 modified "2023-09-22" @default.
- W2913357890 title "Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances" @default.
- W2913357890 cites W104184427 @default.
- W2913357890 cites W1877303889 @default.
- W2913357890 cites W1895076743 @default.
- W2913357890 cites W191916825 @default.
- W2913357890 cites W1946256299 @default.
- W2913357890 cites W1988720110 @default.
- W2913357890 cites W1992926795 @default.
- W2913357890 cites W1995713768 @default.
- W2913357890 cites W2028493795 @default.
- W2913357890 cites W2045744861 @default.
- W2913357890 cites W2076810630 @default.
- W2913357890 cites W2096199223 @default.
- W2913357890 cites W2121210949 @default.
- W2913357890 cites W2124450277 @default.
- W2913357890 cites W2124768887 @default.
- W2913357890 cites W2127447444 @default.
- W2913357890 cites W2154094669 @default.
- W2913357890 cites W2156909104 @default.
- W2913357890 cites W2161466752 @default.
- W2913357890 cites W2167302917 @default.
- W2913357890 cites W2168909589 @default.
- W2913357890 cites W2192240806 @default.
- W2913357890 cites W2263490141 @default.
- W2913357890 cites W2328649617 @default.
- W2913357890 cites W2337540838 @default.
- W2913357890 cites W2528062157 @default.
- W2913357890 cites W2552959509 @default.
- W2913357890 cites W2590513847 @default.
- W2913357890 cites W2608239888 @default.
- W2913357890 cites W2624692385 @default.
- W2913357890 cites W2738275598 @default.
- W2913357890 cites W2763081248 @default.
- W2913357890 cites W2767036780 @default.
- W2913357890 cites W2777387026 @default.
- W2913357890 cites W2782643025 @default.
- W2913357890 cites W2807915830 @default.
- W2913357890 cites W2889691929 @default.
- W2913357890 cites W2890773834 @default.
- W2913357890 cites W2896204105 @default.
- W2913357890 cites W2904091245 @default.
- W2913357890 cites W2910207440 @default.
- W2913357890 cites W2913535645 @default.
- W2913357890 cites W2914932468 @default.
- W2913357890 cites W2952586095 @default.
- W2913357890 cites W2964102336 @default.
- W2913357890 cites W2964106499 @default.
- W2913357890 cites W385466589 @default.
- W2913357890 hasPublicationYear "2019" @default.
- W2913357890 type Work @default.
- W2913357890 sameAs 2913357890 @default.
- W2913357890 citedByCount "15" @default.
- W2913357890 countsByYear W29133578902018 @default.
- W2913357890 countsByYear W29133578902019 @default.
- W2913357890 countsByYear W29133578902020 @default.
- W2913357890 countsByYear W29133578902021 @default.
- W2913357890 crossrefType "posted-content" @default.
- W2913357890 hasAuthorship W2913357890A5025900972 @default.
- W2913357890 hasAuthorship W2913357890A5034370705 @default.
- W2913357890 hasAuthorship W2913357890A5084475092 @default.
- W2913357890 hasConcept C112680207 @default.
- W2913357890 hasConcept C122041747 @default.
- W2913357890 hasConcept C127162648 @default.
- W2913357890 hasConcept C134306372 @default.
- W2913357890 hasConcept C140479938 @default.
- W2913357890 hasConcept C145446738 @default.
- W2913357890 hasConcept C2524010 @default.
- W2913357890 hasConcept C2777634741 @default.
- W2913357890 hasConcept C28826006 @default.
- W2913357890 hasConcept C31258907 @default.
- W2913357890 hasConcept C33923547 @default.
- W2913357890 hasConcept C41008148 @default.
- W2913357890 hasConcept C57869625 @default.
- W2913357890 hasConcept C77553402 @default.
- W2913357890 hasConceptScore W2913357890C112680207 @default.
- W2913357890 hasConceptScore W2913357890C122041747 @default.
- W2913357890 hasConceptScore W2913357890C127162648 @default.
- W2913357890 hasConceptScore W2913357890C134306372 @default.
- W2913357890 hasConceptScore W2913357890C140479938 @default.
- W2913357890 hasConceptScore W2913357890C145446738 @default.
- W2913357890 hasConceptScore W2913357890C2524010 @default.
- W2913357890 hasConceptScore W2913357890C2777634741 @default.
- W2913357890 hasConceptScore W2913357890C28826006 @default.
- W2913357890 hasConceptScore W2913357890C31258907 @default.
- W2913357890 hasConceptScore W2913357890C33923547 @default.
- W2913357890 hasConceptScore W2913357890C41008148 @default.
- W2913357890 hasConceptScore W2913357890C57869625 @default.
- W2913357890 hasConceptScore W2913357890C77553402 @default.
- W2913357890 hasLocation W29133578901 @default.
- W2913357890 hasOpenAccess W2913357890 @default.
- W2913357890 hasPrimaryLocation W29133578901 @default.
- W2913357890 hasRelatedWork W104184427 @default.
- W2913357890 hasRelatedWork W1987083649 @default.