FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2995594417> ?p ?o ?g. }

• W2995594417 abstract "Variance reduction methods which use a mixture of large and small batch gradients, such as SVRG (Johnson & Zhang, 2013) and SpiderBoost (Wang et al., 2018), require significantly more computational resources per update than SGD (Robbins & Monro, 1951). We reduce the computational cost per update of variance reduction methods by introducing a sparse gradient operator blending the top-K operator (Stich et al., 2018; Aji & Heafield, 2017) and the randomized coordinate descent operator. While the computational cost of computing the derivative of a model parameter is constant, we make the observation that the gains in variance reduction are proportional to the magnitude of the derivative. In this paper, we show that a sparse gradient based on the magnitude of past gradients reduces the computational cost of model updates without a significant loss in variance reduction. Theoretically, our algorithm is at least as good as the best available algorithm (e.g. SpiderBoost) under appropriate settings of parameters and can be much more efficient if our algorithm succeeds in capturing the sparsity of the gradients. Empirically, our algorithm consistently outperforms SpiderBoost using various models to solve various image classification tasks. We also provide empirical evidence to support the intuition behind our algorithm via a simple gradient entropy computation, which serves to quantify gradient sparsity at every iteration." @default.
• W2995594417 created "2019-12-26" @default.
• W2995594417 creator A5032933169 @default.
• W2995594417 creator A5049812527 @default.
• W2995594417 creator A5071822572 @default.
• W2995594417 date "2020-04-30" @default.
• W2995594417 modified "2023-09-23" @default.
• W2995594417 title "Variance Reduction With Sparse Gradients" @default.
• W2995594417 cites W1726370773 @default.
• W2995594417 cites W1992208280 @default.
• W2995594417 cites W1994616650 @default.
• W2995594417 cites W2047152541 @default.
• W2995594417 cites W2064675550 @default.
• W2995594417 cites W2093647425 @default.
• W2995594417 cites W2105875671 @default.
• W2995594417 cites W2107438106 @default.
• W2995594417 cites W2135482703 @default.
• W2995594417 cites W2140310134 @default.
• W2995594417 cites W2146502635 @default.
• W2995594417 cites W2194775991 @default.
• W2995594417 cites W2219888463 @default.
• W2995594417 cites W2301983558 @default.
• W2995594417 cites W2304667012 @default.
• W2995594417 cites W2306875213 @default.
• W2995594417 cites W2335728318 @default.
• W2995594417 cites W2787873267 @default.
• W2995594417 cites W2890924858 @default.
• W2995594417 cites W2898280890 @default.
• W2995594417 cites W2899771611 @default.
• W2995594417 cites W2914000446 @default.
• W2995594417 cites W2963409219 @default.
• W2995594417 cites W2963411541 @default.
• W2995594417 cites W2963545805 @default.
• W2995594417 cites W2963607709 @default.
• W2995594417 cites W2963655672 @default.
• W2995594417 cites W2963702144 @default.
• W2995594417 cites W2964121744 @default.
• W2995594417 cites W2964312760 @default.
• W2995594417 cites W2970224333 @default.
• W2995594417 cites W3024230214 @default.
• W2995594417 cites W3039770199 @default.
• W2995594417 cites W3101036738 @default.
• W2995594417 hasPublicationYear "2020" @default.
• W2995594417 type Work @default.
• W2995594417 sameAs 2995594417 @default.
• W2995594417 citedByCount "2" @default.
• W2995594417 countsByYear W29955944172020 @default.
• W2995594417 countsByYear W29955944172021 @default.
• W2995594417 crossrefType "proceedings-article" @default.
• W2995594417 hasAuthorship W2995594417A5032933169 @default.
• W2995594417 hasAuthorship W2995594417A5049812527 @default.
• W2995594417 hasAuthorship W2995594417A5071822572 @default.
• W2995594417 hasConcept C104317684 @default.
• W2995594417 hasConcept C105795698 @default.
• W2995594417 hasConcept C111335779 @default.
• W2995594417 hasConcept C11413529 @default.
• W2995594417 hasConcept C126255220 @default.
• W2995594417 hasConcept C153258448 @default.
• W2995594417 hasConcept C154945302 @default.
• W2995594417 hasConcept C158448853 @default.
• W2995594417 hasConcept C17020691 @default.
• W2995594417 hasConcept C185592680 @default.
• W2995594417 hasConcept C19499675 @default.
• W2995594417 hasConcept C2524010 @default.
• W2995594417 hasConcept C33923547 @default.
• W2995594417 hasConcept C41008148 @default.
• W2995594417 hasConcept C45374587 @default.
• W2995594417 hasConcept C50644808 @default.
• W2995594417 hasConcept C55493867 @default.
• W2995594417 hasConcept C62644790 @default.
• W2995594417 hasConcept C86339819 @default.
• W2995594417 hasConceptScore W2995594417C104317684 @default.
• W2995594417 hasConceptScore W2995594417C105795698 @default.
• W2995594417 hasConceptScore W2995594417C111335779 @default.
• W2995594417 hasConceptScore W2995594417C11413529 @default.
• W2995594417 hasConceptScore W2995594417C126255220 @default.
• W2995594417 hasConceptScore W2995594417C153258448 @default.
• W2995594417 hasConceptScore W2995594417C154945302 @default.
• W2995594417 hasConceptScore W2995594417C158448853 @default.
• W2995594417 hasConceptScore W2995594417C17020691 @default.
• W2995594417 hasConceptScore W2995594417C185592680 @default.
• W2995594417 hasConceptScore W2995594417C19499675 @default.
• W2995594417 hasConceptScore W2995594417C2524010 @default.
• W2995594417 hasConceptScore W2995594417C33923547 @default.
• W2995594417 hasConceptScore W2995594417C41008148 @default.
• W2995594417 hasConceptScore W2995594417C45374587 @default.
• W2995594417 hasConceptScore W2995594417C50644808 @default.
• W2995594417 hasConceptScore W2995594417C55493867 @default.
• W2995594417 hasConceptScore W2995594417C62644790 @default.
• W2995594417 hasConceptScore W2995594417C86339819 @default.
• W2995594417 hasLocation W29955944171 @default.
• W2995594417 hasOpenAccess W2995594417 @default.
• W2995594417 hasPrimaryLocation W29955944171 @default.
• W2995594417 hasRelatedWork W1486662790 @default.
• W2995594417 hasRelatedWork W1580520382 @default.
• W2995594417 hasRelatedWork W2164447077 @default.
• W2995594417 hasRelatedWork W2554489331 @default.
• W2995594417 hasRelatedWork W2561209984 @default.
• W2995594417 hasRelatedWork W2585228289 @default.
• W2995594417 hasRelatedWork W2594924640 @default.

• next