FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W3121837399> ?p ?o ?g. }

• W3121837399 abstract "The adaptive stochastic gradient descent (SGD) with momentum has been widely adopted in deep learning as well as convex optimization. In practice, the last iterate is commonly used as the final solution to make decisions. However, the available regret analysis and the setting of constant momentum parameters only guarantee the optimal convergence of the averaged solution. In this paper, we fill this theory-practice gap by investigating the convergence of the last iterate (referred to as {it individual convergence}), which is a more difficult task than convergence analysis of the averaged solution. Specifically, in the constrained convex cases, we prove that the adaptive Polyak's Heavy-ball (HB) method, in which only the step size is updated using the exponential moving average strategy, attains an optimal individual convergence rate of O(1t), as opposed to the optimality of O(log⁡tt) of SGD, where t is the number of iteration. Our new analysis not only shows how the HB momentum and its time-varying weight help us to achieve the acceleration in convex optimization but also gives valuable hints how the momentum parameters should be scheduled in deep learning. Empirical results on optimizing convex functions and training deep networks validate the correctness of our convergence analysis and demonstrate the improved performance of the adaptive HB methods." @default.
• W3121837399 created "2021-02-01" @default.
• W3121837399 creator A5025184358 @default.
• W3121837399 creator A5046355512 @default.
• W3121837399 creator A5052908126 @default.
• W3121837399 creator A5053102123 @default.
• W3121837399 date "2021-05-03" @default.
• W3121837399 modified "2023-09-23" @default.
• W3121837399 title "The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods" @default.
• W3121837399 cites W104184427 @default.
• W3121837399 cites W1505731132 @default.
• W3121837399 cites W1988720110 @default.
• W3121837399 cites W2000462146 @default.
• W3121837399 cites W2112269233 @default.
• W3121837399 cites W2137731592 @default.
• W3121837399 cites W2140451424 @default.
• W3121837399 cites W2146502635 @default.
• W3121837399 cites W2148825261 @default.
• W3121837399 cites W2163605009 @default.
• W3121837399 cites W2167302917 @default.
• W3121837399 cites W2337540838 @default.
• W3121837399 cites W2398337421 @default.
• W3121837399 cites W2523246573 @default.
• W3121837399 cites W2785523195 @default.
• W3121837399 cites W2896146108 @default.
• W3121837399 cites W2923133561 @default.
• W3121837399 cites W2947578309 @default.
• W3121837399 cites W2962749006 @default.
• W3121837399 cites W2962811981 @default.
• W3121837399 cites W2963208657 @default.
• W3121837399 cites W2963563140 @default.
• W3121837399 cites W2963783084 @default.
• W3121837399 cites W2966614174 @default.
• W3121837399 cites W2970948121 @default.
• W3121837399 cites W2971452831 @default.
• W3121837399 cites W2994635124 @default.
• W3121837399 cites W2996449522 @default.
• W3121837399 cites W3009948090 @default.
• W3121837399 cites W3035289993 @default.
• W3121837399 cites W3035358246 @default.
• W3121837399 cites W3153737196 @default.
• W3121837399 hasPublicationYear "2021" @default.
• W3121837399 type Work @default.
• W3121837399 sameAs 3121837399 @default.
• W3121837399 citedByCount "1" @default.
• W3121837399 countsByYear W31218373992021 @default.
• W3121837399 crossrefType "proceedings-article" @default.
• W3121837399 hasAuthorship W3121837399A5025184358 @default.
• W3121837399 hasAuthorship W3121837399A5046355512 @default.
• W3121837399 hasAuthorship W3121837399A5052908126 @default.
• W3121837399 hasAuthorship W3121837399A5053102123 @default.
• W3121837399 hasConcept C10138342 @default.
• W3121837399 hasConcept C112680207 @default.
• W3121837399 hasConcept C122041747 @default.
• W3121837399 hasConcept C126255220 @default.
• W3121837399 hasConcept C127162648 @default.
• W3121837399 hasConcept C134306372 @default.
• W3121837399 hasConcept C145446738 @default.
• W3121837399 hasConcept C154945302 @default.
• W3121837399 hasConcept C157972887 @default.
• W3121837399 hasConcept C162324750 @default.
• W3121837399 hasConcept C206688291 @default.
• W3121837399 hasConcept C2524010 @default.
• W3121837399 hasConcept C2777303404 @default.
• W3121837399 hasConcept C28826006 @default.
• W3121837399 hasConcept C31258907 @default.
• W3121837399 hasConcept C33923547 @default.
• W3121837399 hasConcept C41008148 @default.
• W3121837399 hasConcept C50522688 @default.
• W3121837399 hasConcept C50644808 @default.
• W3121837399 hasConcept C57869625 @default.
• W3121837399 hasConcept C60718061 @default.
• W3121837399 hasConceptScore W3121837399C10138342 @default.
• W3121837399 hasConceptScore W3121837399C112680207 @default.
• W3121837399 hasConceptScore W3121837399C122041747 @default.
• W3121837399 hasConceptScore W3121837399C126255220 @default.
• W3121837399 hasConceptScore W3121837399C127162648 @default.
• W3121837399 hasConceptScore W3121837399C134306372 @default.
• W3121837399 hasConceptScore W3121837399C145446738 @default.
• W3121837399 hasConceptScore W3121837399C154945302 @default.
• W3121837399 hasConceptScore W3121837399C157972887 @default.
• W3121837399 hasConceptScore W3121837399C162324750 @default.
• W3121837399 hasConceptScore W3121837399C206688291 @default.
• W3121837399 hasConceptScore W3121837399C2524010 @default.
• W3121837399 hasConceptScore W3121837399C2777303404 @default.
• W3121837399 hasConceptScore W3121837399C28826006 @default.
• W3121837399 hasConceptScore W3121837399C31258907 @default.
• W3121837399 hasConceptScore W3121837399C33923547 @default.
• W3121837399 hasConceptScore W3121837399C41008148 @default.
• W3121837399 hasConceptScore W3121837399C50522688 @default.
• W3121837399 hasConceptScore W3121837399C50644808 @default.
• W3121837399 hasConceptScore W3121837399C57869625 @default.
• W3121837399 hasConceptScore W3121837399C60718061 @default.
• W3121837399 hasOpenAccess W3121837399 @default.
• W3121837399 hasRelatedWork W2000698507 @default.
• W3121837399 hasRelatedWork W2049556680 @default.
• W3121837399 hasRelatedWork W2051691992 @default.
• W3121837399 hasRelatedWork W2183021192 @default.
• W3121837399 hasRelatedWork W2695988161 @default.
• W3121837399 hasRelatedWork W2765599067 @default.

• next