Matches in SemOpenAlex for { <https://semopenalex.org/work/W3132287789> ?p ?o ?g. }
Showing items 1 to 99 of
99
with 100 items per page.
- W3132287789 abstract "We investigate stochastic optimization under weaker assumptions on the distribution of noise than those used in usual analysis. Our assumptions are motivated by empirical observations in training neural networks. In particular, standard results on optimal convergence rates for stochastic optimization assume either there exists a uniform bound on the moments of the gradient noise, or that the noise decays as the algorithm progresses. These assumptions do not match the empirical behavior of optimization algorithms used in neural network training where the noise level in stochastic gradients could even increase with time. We address this nonstationary behavior of noise by analyzing convergence rates of stochastic gradient methods subject to changing second moment (or variance) of the stochastic oracle. When the noise variation is known, we show that it is always beneficial to adapt the step-size and exploit the noise variability. When the noise statistics are unknown, we obtain similar improvements by developing an online estimator of the noise level, thereby recovering close variants of RMSProp~citep{tieleman2012lecture}. Consequently, our results reveal why adaptive step size methods can outperform SGD, while still enjoying theoretical guarantees." @default.
- W3132287789 created "2021-03-01" @default.
- W3132287789 creator A5011183160 @default.
- W3132287789 creator A5058767558 @default.
- W3132287789 creator A5064766573 @default.
- W3132287789 creator A5078288116 @default.
- W3132287789 creator A5083701507 @default.
- W3132287789 date "2021-05-04" @default.
- W3132287789 modified "2023-09-24" @default.
- W3132287789 title "Stochastic Optimization with Non-stationary Noise: The Power of Moment Estimation" @default.
- W3132287789 hasPublicationYear "2021" @default.
- W3132287789 type Work @default.
- W3132287789 sameAs 3132287789 @default.
- W3132287789 citedByCount "0" @default.
- W3132287789 crossrefType "journal-article" @default.
- W3132287789 hasAuthorship W3132287789A5011183160 @default.
- W3132287789 hasAuthorship W3132287789A5058767558 @default.
- W3132287789 hasAuthorship W3132287789A5064766573 @default.
- W3132287789 hasAuthorship W3132287789A5078288116 @default.
- W3132287789 hasAuthorship W3132287789A5083701507 @default.
- W3132287789 hasConcept C105795698 @default.
- W3132287789 hasConcept C115961682 @default.
- W3132287789 hasConcept C121332964 @default.
- W3132287789 hasConcept C126255220 @default.
- W3132287789 hasConcept C147168706 @default.
- W3132287789 hasConcept C154945302 @default.
- W3132287789 hasConcept C162324750 @default.
- W3132287789 hasConcept C163258240 @default.
- W3132287789 hasConcept C163294075 @default.
- W3132287789 hasConcept C179254644 @default.
- W3132287789 hasConcept C185429906 @default.
- W3132287789 hasConcept C187612029 @default.
- W3132287789 hasConcept C194387892 @default.
- W3132287789 hasConcept C200378446 @default.
- W3132287789 hasConcept C203234222 @default.
- W3132287789 hasConcept C206688291 @default.
- W3132287789 hasConcept C2777303404 @default.
- W3132287789 hasConcept C29265498 @default.
- W3132287789 hasConcept C33923547 @default.
- W3132287789 hasConcept C41008148 @default.
- W3132287789 hasConcept C50522688 @default.
- W3132287789 hasConcept C50644808 @default.
- W3132287789 hasConcept C62520636 @default.
- W3132287789 hasConcept C74650414 @default.
- W3132287789 hasConcept C86582703 @default.
- W3132287789 hasConcept C99498987 @default.
- W3132287789 hasConceptScore W3132287789C105795698 @default.
- W3132287789 hasConceptScore W3132287789C115961682 @default.
- W3132287789 hasConceptScore W3132287789C121332964 @default.
- W3132287789 hasConceptScore W3132287789C126255220 @default.
- W3132287789 hasConceptScore W3132287789C147168706 @default.
- W3132287789 hasConceptScore W3132287789C154945302 @default.
- W3132287789 hasConceptScore W3132287789C162324750 @default.
- W3132287789 hasConceptScore W3132287789C163258240 @default.
- W3132287789 hasConceptScore W3132287789C163294075 @default.
- W3132287789 hasConceptScore W3132287789C179254644 @default.
- W3132287789 hasConceptScore W3132287789C185429906 @default.
- W3132287789 hasConceptScore W3132287789C187612029 @default.
- W3132287789 hasConceptScore W3132287789C194387892 @default.
- W3132287789 hasConceptScore W3132287789C200378446 @default.
- W3132287789 hasConceptScore W3132287789C203234222 @default.
- W3132287789 hasConceptScore W3132287789C206688291 @default.
- W3132287789 hasConceptScore W3132287789C2777303404 @default.
- W3132287789 hasConceptScore W3132287789C29265498 @default.
- W3132287789 hasConceptScore W3132287789C33923547 @default.
- W3132287789 hasConceptScore W3132287789C41008148 @default.
- W3132287789 hasConceptScore W3132287789C50522688 @default.
- W3132287789 hasConceptScore W3132287789C50644808 @default.
- W3132287789 hasConceptScore W3132287789C62520636 @default.
- W3132287789 hasConceptScore W3132287789C74650414 @default.
- W3132287789 hasConceptScore W3132287789C86582703 @default.
- W3132287789 hasConceptScore W3132287789C99498987 @default.
- W3132287789 hasLocation W31322877891 @default.
- W3132287789 hasOpenAccess W3132287789 @default.
- W3132287789 hasPrimaryLocation W31322877891 @default.
- W3132287789 hasRelatedWork W1169135902 @default.
- W3132287789 hasRelatedWork W13282426 @default.
- W3132287789 hasRelatedWork W1508077025 @default.
- W3132287789 hasRelatedWork W1977405322 @default.
- W3132287789 hasRelatedWork W2088163039 @default.
- W3132287789 hasRelatedWork W2114029332 @default.
- W3132287789 hasRelatedWork W2260172210 @default.
- W3132287789 hasRelatedWork W2523331129 @default.
- W3132287789 hasRelatedWork W2780035115 @default.
- W3132287789 hasRelatedWork W2787748180 @default.
- W3132287789 hasRelatedWork W2796940913 @default.
- W3132287789 hasRelatedWork W2923420596 @default.
- W3132287789 hasRelatedWork W2950266408 @default.
- W3132287789 hasRelatedWork W2972680475 @default.
- W3132287789 hasRelatedWork W2981382427 @default.
- W3132287789 hasRelatedWork W3007796267 @default.
- W3132287789 hasRelatedWork W3045587504 @default.
- W3132287789 hasRelatedWork W3149743631 @default.
- W3132287789 hasRelatedWork W3161614516 @default.
- W3132287789 hasRelatedWork W3161862726 @default.
- W3132287789 isParatext "false" @default.
- W3132287789 isRetracted "false" @default.
- W3132287789 magId "3132287789" @default.
- W3132287789 workType "article" @default.