Matches in SemOpenAlex for { <https://semopenalex.org/work/W4287869624> ?p ?o ?g. }
Showing items 1 to 89 of
89
with 100 items per page.
- W4287869624 abstract "We provide an improved analysis of normalized SGD showing that adding momentum provably removes the need for large batch sizes on non-convex objectives. Then, we consider the case of objectives with bounded second derivative and show that in this case a small tweak to the momentum formula allows normalized SGD with momentum to find an $epsilon$-critical point in $O(1/epsilon^{3.5})$ iterations, matching the best-known rates without accruing any logarithmic factors or dependence on dimension. We also provide an adaptive method that automatically improves convergence rates when the variance in the gradients is small. Finally, we show that our method is effective when employed on popular large scale tasks such as ResNet-50 and BERT pretraining, matching the performance of the disparate methods used to get state-of-the-art results on both tasks." @default.
- W4287869624 created "2022-07-26" @default.
- W4287869624 creator A5053378034 @default.
- W4287869624 creator A5066011484 @default.
- W4287869624 date "2020-02-09" @default.
- W4287869624 modified "2023-10-15" @default.
- W4287869624 title "Momentum Improves Normalized SGD" @default.
- W4287869624 doi "https://doi.org/10.48550/arxiv.2002.03305" @default.
- W4287869624 hasPublicationYear "2020" @default.
- W4287869624 type Work @default.
- W4287869624 citedByCount "0" @default.
- W4287869624 crossrefType "posted-content" @default.
- W4287869624 hasAuthorship W4287869624A5053378034 @default.
- W4287869624 hasAuthorship W4287869624A5066011484 @default.
- W4287869624 hasBestOaLocation W42878696241 @default.
- W4287869624 hasConcept C10138342 @default.
- W4287869624 hasConcept C105795698 @default.
- W4287869624 hasConcept C112680207 @default.
- W4287869624 hasConcept C11413529 @default.
- W4287869624 hasConcept C121332964 @default.
- W4287869624 hasConcept C121955636 @default.
- W4287869624 hasConcept C126255220 @default.
- W4287869624 hasConcept C134306372 @default.
- W4287869624 hasConcept C144133560 @default.
- W4287869624 hasConcept C145446738 @default.
- W4287869624 hasConcept C162324750 @default.
- W4287869624 hasConcept C165064840 @default.
- W4287869624 hasConcept C196083921 @default.
- W4287869624 hasConcept C202444582 @default.
- W4287869624 hasConcept C2524010 @default.
- W4287869624 hasConcept C26517878 @default.
- W4287869624 hasConcept C2777303404 @default.
- W4287869624 hasConcept C2778755073 @default.
- W4287869624 hasConcept C28826006 @default.
- W4287869624 hasConcept C33676613 @default.
- W4287869624 hasConcept C33923547 @default.
- W4287869624 hasConcept C34388435 @default.
- W4287869624 hasConcept C38652104 @default.
- W4287869624 hasConcept C39927690 @default.
- W4287869624 hasConcept C41008148 @default.
- W4287869624 hasConcept C50522688 @default.
- W4287869624 hasConcept C57869625 @default.
- W4287869624 hasConcept C60718061 @default.
- W4287869624 hasConcept C62520636 @default.
- W4287869624 hasConceptScore W4287869624C10138342 @default.
- W4287869624 hasConceptScore W4287869624C105795698 @default.
- W4287869624 hasConceptScore W4287869624C112680207 @default.
- W4287869624 hasConceptScore W4287869624C11413529 @default.
- W4287869624 hasConceptScore W4287869624C121332964 @default.
- W4287869624 hasConceptScore W4287869624C121955636 @default.
- W4287869624 hasConceptScore W4287869624C126255220 @default.
- W4287869624 hasConceptScore W4287869624C134306372 @default.
- W4287869624 hasConceptScore W4287869624C144133560 @default.
- W4287869624 hasConceptScore W4287869624C145446738 @default.
- W4287869624 hasConceptScore W4287869624C162324750 @default.
- W4287869624 hasConceptScore W4287869624C165064840 @default.
- W4287869624 hasConceptScore W4287869624C196083921 @default.
- W4287869624 hasConceptScore W4287869624C202444582 @default.
- W4287869624 hasConceptScore W4287869624C2524010 @default.
- W4287869624 hasConceptScore W4287869624C26517878 @default.
- W4287869624 hasConceptScore W4287869624C2777303404 @default.
- W4287869624 hasConceptScore W4287869624C2778755073 @default.
- W4287869624 hasConceptScore W4287869624C28826006 @default.
- W4287869624 hasConceptScore W4287869624C33676613 @default.
- W4287869624 hasConceptScore W4287869624C33923547 @default.
- W4287869624 hasConceptScore W4287869624C34388435 @default.
- W4287869624 hasConceptScore W4287869624C38652104 @default.
- W4287869624 hasConceptScore W4287869624C39927690 @default.
- W4287869624 hasConceptScore W4287869624C41008148 @default.
- W4287869624 hasConceptScore W4287869624C50522688 @default.
- W4287869624 hasConceptScore W4287869624C57869625 @default.
- W4287869624 hasConceptScore W4287869624C60718061 @default.
- W4287869624 hasConceptScore W4287869624C62520636 @default.
- W4287869624 hasLocation W42878696241 @default.
- W4287869624 hasOpenAccess W4287869624 @default.
- W4287869624 hasPrimaryLocation W42878696241 @default.
- W4287869624 hasRelatedWork W1973911476 @default.
- W4287869624 hasRelatedWork W2039013946 @default.
- W4287869624 hasRelatedWork W2952551997 @default.
- W4287869624 hasRelatedWork W3032929727 @default.
- W4287869624 hasRelatedWork W3035012470 @default.
- W4287869624 hasRelatedWork W4205132716 @default.
- W4287869624 hasRelatedWork W4299789052 @default.
- W4287869624 hasRelatedWork W4301602261 @default.
- W4287869624 hasRelatedWork W657818131 @default.
- W4287869624 hasRelatedWork W3024856659 @default.
- W4287869624 isParatext "false" @default.
- W4287869624 isRetracted "false" @default.
- W4287869624 workType "article" @default.