Matches in SemOpenAlex for { <https://semopenalex.org/work/W3033432775> ?p ?o ?g. }
- W3033432775 abstract "We investigate stochastic optimization problems under relaxed assumptions on the distribution of noise that are motivated by empirical observations in neural network training. Standard results on optimal convergence rates for stochastic optimization assume either there exists a uniform bound on the moments of the gradient noise, or that the noise decays as the algorithm progresses. These assumptions do not match the empirical behavior of optimization algorithms used in neural network training where the noise level in stochastic gradients could even increase with time. We address this behavior by studying convergence rates of stochastic gradient methods subject to changing second moment (or variance) of the stochastic oracle as the iterations progress. When the variation in the noise is known, we show that it is always beneficial to adapt the step-size and exploit the noise variability. When the noise statistics are unknown, we obtain similar improvements by developing an online estimator of the noise level, thereby recovering close variants of RMSProp. Consequently, our results reveal an important scenario where adaptive stepsize methods outperform SGD." @default.
- W3033432775 created "2020-06-12" @default.
- W3033432775 creator A5011183160 @default.
- W3033432775 creator A5058767558 @default.
- W3033432775 creator A5064766573 @default.
- W3033432775 creator A5078288116 @default.
- W3033432775 creator A5083701507 @default.
- W3033432775 date "2020-06-08" @default.
- W3033432775 modified "2023-09-27" @default.
- W3033432775 title "Stochastic Optimization with Non-stationary Noise" @default.
- W3033432775 cites W1505731132 @default.
- W3033432775 cites W1522301498 @default.
- W3033432775 cites W1988526405 @default.
- W3033432775 cites W1992208280 @default.
- W3033432775 cites W2045744861 @default.
- W3033432775 cites W2112269233 @default.
- W3033432775 cites W2122678453 @default.
- W3033432775 cites W2146502635 @default.
- W3033432775 cites W2156779765 @default.
- W3033432775 cites W2618001342 @default.
- W3033432775 cites W2755172545 @default.
- W3033432775 cites W2806803448 @default.
- W3033432775 cites W2818059595 @default.
- W3033432775 cites W2886294525 @default.
- W3033432775 cites W2886463271 @default.
- W3033432775 cites W2912323147 @default.
- W3033432775 cites W2946511237 @default.
- W3033432775 cites W2947578309 @default.
- W3033432775 cites W2962832505 @default.
- W3033432775 cites W2962930655 @default.
- W3033432775 cites W2963049774 @default.
- W3033432775 cites W2963326510 @default.
- W3033432775 cites W2963411541 @default.
- W3033432775 cites W2963433607 @default.
- W3033432775 cites W2963470657 @default.
- W3033432775 cites W2963487351 @default.
- W3033432775 cites W2963562522 @default.
- W3033432775 cites W2963563140 @default.
- W3033432775 cites W2963613486 @default.
- W3033432775 cites W2963698657 @default.
- W3033432775 cites W2964054583 @default.
- W3033432775 cites W2978586064 @default.
- W3033432775 cites W2980197666 @default.
- W3033432775 cites W2980398138 @default.
- W3033432775 cites W2992505801 @default.
- W3033432775 cites W2993258424 @default.
- W3033432775 cites W2994689640 @default.
- W3033432775 cites W2994747431 @default.
- W3033432775 cites W2995509798 @default.
- W3033432775 cites W3101098636 @default.
- W3033432775 hasPublicationYear "2020" @default.
- W3033432775 type Work @default.
- W3033432775 sameAs 3033432775 @default.
- W3033432775 citedByCount "1" @default.
- W3033432775 countsByYear W30334327752021 @default.
- W3033432775 crossrefType "posted-content" @default.
- W3033432775 hasAuthorship W3033432775A5011183160 @default.
- W3033432775 hasAuthorship W3033432775A5058767558 @default.
- W3033432775 hasAuthorship W3033432775A5064766573 @default.
- W3033432775 hasAuthorship W3033432775A5078288116 @default.
- W3033432775 hasAuthorship W3033432775A5083701507 @default.
- W3033432775 hasConcept C105795698 @default.
- W3033432775 hasConcept C111919701 @default.
- W3033432775 hasConcept C115903868 @default.
- W3033432775 hasConcept C115961682 @default.
- W3033432775 hasConcept C126255220 @default.
- W3033432775 hasConcept C147168706 @default.
- W3033432775 hasConcept C149672232 @default.
- W3033432775 hasConcept C154945302 @default.
- W3033432775 hasConcept C162324750 @default.
- W3033432775 hasConcept C185429906 @default.
- W3033432775 hasConcept C194387892 @default.
- W3033432775 hasConcept C206688291 @default.
- W3033432775 hasConcept C26517878 @default.
- W3033432775 hasConcept C2777303404 @default.
- W3033432775 hasConcept C33923547 @default.
- W3033432775 hasConcept C38652104 @default.
- W3033432775 hasConcept C41008148 @default.
- W3033432775 hasConcept C50522688 @default.
- W3033432775 hasConcept C50644808 @default.
- W3033432775 hasConcept C55166926 @default.
- W3033432775 hasConcept C55479107 @default.
- W3033432775 hasConcept C86582703 @default.
- W3033432775 hasConcept C99498987 @default.
- W3033432775 hasConceptScore W3033432775C105795698 @default.
- W3033432775 hasConceptScore W3033432775C111919701 @default.
- W3033432775 hasConceptScore W3033432775C115903868 @default.
- W3033432775 hasConceptScore W3033432775C115961682 @default.
- W3033432775 hasConceptScore W3033432775C126255220 @default.
- W3033432775 hasConceptScore W3033432775C147168706 @default.
- W3033432775 hasConceptScore W3033432775C149672232 @default.
- W3033432775 hasConceptScore W3033432775C154945302 @default.
- W3033432775 hasConceptScore W3033432775C162324750 @default.
- W3033432775 hasConceptScore W3033432775C185429906 @default.
- W3033432775 hasConceptScore W3033432775C194387892 @default.
- W3033432775 hasConceptScore W3033432775C206688291 @default.
- W3033432775 hasConceptScore W3033432775C26517878 @default.
- W3033432775 hasConceptScore W3033432775C2777303404 @default.
- W3033432775 hasConceptScore W3033432775C33923547 @default.
- W3033432775 hasConceptScore W3033432775C38652104 @default.