Matches in SemOpenAlex for { <https://semopenalex.org/work/W4377013551> ?p ?o ?g. }
- W4377013551 endingPage "423" @default.
- W4377013551 startingPage "394" @default.
- W4377013551 abstract "We introduce a general framework for nonlinear stochastic gradient descent (SGD) for the scenarios when gradient noise exhibits heavy tails. The proposed framework subsumes several popular nonlinearity choices, like clipped, normalized, signed, or quantized gradient, but we also consider novel nonlinearity choices. We establish for the considered class of methods strong convergence guarantees assuming a strongly convex cost function with Lipschitz continuous gradients under very general assumptions on the gradient noise. Most notably, we show that, for a nonlinearity with bounded outputs and for the gradient noise that may not have finite moments of order greater than one, the nonlinear SGD’s mean squared error (MSE), or equivalently, the expected cost function’s optimality gap, converges to zero at rate , . In contrast, for the same noise setting, the linear SGD generates a sequence with unbounded variances. Furthermore, for general nonlinearities that can be decoupled componentwise and a class of joint nonlinearities, we show that the nonlinear SGD asymptotically (locally) achieves an rate in the weak convergence sense and explicitly quantify the corresponding asymptotic variance. Experiments show that, while our framework is more general than existing studies of SGD under heavy-tail noise, several easy-to-implement nonlinearities from our framework are competitive with state-of-the-art alternatives on real datasets with heavy-tail noises." @default.
- W4377013551 created "2023-05-19" @default.
- W4377013551 creator A5019965945 @default.
- W4377013551 creator A5021969767 @default.
- W4377013551 creator A5070836307 @default.
- W4377013551 creator A5077197425 @default.
- W4377013551 creator A5077268766 @default.
- W4377013551 creator A5078640354 @default.
- W4377013551 date "2023-05-16" @default.
- W4377013551 modified "2023-10-17" @default.
- W4377013551 title "Nonlinear Gradient Mappings and Stochastic Optimization: A General Framework with Applications to Heavy-Tail Noise" @default.
- W4377013551 cites W1498711961 @default.
- W4377013551 cites W1991083751 @default.
- W4377013551 cites W1992208280 @default.
- W4377013551 cites W2045744861 @default.
- W4377013551 cites W2080335539 @default.
- W4377013551 cites W2083657634 @default.
- W4377013551 cites W2108306501 @default.
- W4377013551 cites W2114346036 @default.
- W4377013551 cites W2153368486 @default.
- W4377013551 cites W2153635508 @default.
- W4377013551 cites W2168909589 @default.
- W4377013551 cites W2937268935 @default.
- W4377013551 cites W2962952793 @default.
- W4377013551 cites W2963433607 @default.
- W4377013551 cites W2972939971 @default.
- W4377013551 cites W3123607434 @default.
- W4377013551 cites W4252698487 @default.
- W4377013551 doi "https://doi.org/10.1137/21m145896x" @default.
- W4377013551 hasPublicationYear "2023" @default.
- W4377013551 type Work @default.
- W4377013551 citedByCount "0" @default.
- W4377013551 crossrefType "journal-article" @default.
- W4377013551 hasAuthorship W4377013551A5019965945 @default.
- W4377013551 hasAuthorship W4377013551A5021969767 @default.
- W4377013551 hasAuthorship W4377013551A5070836307 @default.
- W4377013551 hasAuthorship W4377013551A5077197425 @default.
- W4377013551 hasAuthorship W4377013551A5077268766 @default.
- W4377013551 hasAuthorship W4377013551A5078640354 @default.
- W4377013551 hasBestOaLocation W43770135512 @default.
- W4377013551 hasConcept C112680207 @default.
- W4377013551 hasConcept C115961682 @default.
- W4377013551 hasConcept C119857082 @default.
- W4377013551 hasConcept C121332964 @default.
- W4377013551 hasConcept C126255220 @default.
- W4377013551 hasConcept C127162648 @default.
- W4377013551 hasConcept C134306372 @default.
- W4377013551 hasConcept C14036430 @default.
- W4377013551 hasConcept C145446738 @default.
- W4377013551 hasConcept C154945302 @default.
- W4377013551 hasConcept C158622935 @default.
- W4377013551 hasConcept C162324750 @default.
- W4377013551 hasConcept C206688291 @default.
- W4377013551 hasConcept C22324862 @default.
- W4377013551 hasConcept C2524010 @default.
- W4377013551 hasConcept C2777303404 @default.
- W4377013551 hasConcept C28826006 @default.
- W4377013551 hasConcept C31258907 @default.
- W4377013551 hasConcept C33923547 @default.
- W4377013551 hasConcept C34388435 @default.
- W4377013551 hasConcept C41008148 @default.
- W4377013551 hasConcept C50522688 @default.
- W4377013551 hasConcept C50644808 @default.
- W4377013551 hasConcept C57869625 @default.
- W4377013551 hasConcept C62520636 @default.
- W4377013551 hasConcept C78458016 @default.
- W4377013551 hasConcept C86803240 @default.
- W4377013551 hasConcept C99498987 @default.
- W4377013551 hasConceptScore W4377013551C112680207 @default.
- W4377013551 hasConceptScore W4377013551C115961682 @default.
- W4377013551 hasConceptScore W4377013551C119857082 @default.
- W4377013551 hasConceptScore W4377013551C121332964 @default.
- W4377013551 hasConceptScore W4377013551C126255220 @default.
- W4377013551 hasConceptScore W4377013551C127162648 @default.
- W4377013551 hasConceptScore W4377013551C134306372 @default.
- W4377013551 hasConceptScore W4377013551C14036430 @default.
- W4377013551 hasConceptScore W4377013551C145446738 @default.
- W4377013551 hasConceptScore W4377013551C154945302 @default.
- W4377013551 hasConceptScore W4377013551C158622935 @default.
- W4377013551 hasConceptScore W4377013551C162324750 @default.
- W4377013551 hasConceptScore W4377013551C206688291 @default.
- W4377013551 hasConceptScore W4377013551C22324862 @default.
- W4377013551 hasConceptScore W4377013551C2524010 @default.
- W4377013551 hasConceptScore W4377013551C2777303404 @default.
- W4377013551 hasConceptScore W4377013551C28826006 @default.
- W4377013551 hasConceptScore W4377013551C31258907 @default.
- W4377013551 hasConceptScore W4377013551C33923547 @default.
- W4377013551 hasConceptScore W4377013551C34388435 @default.
- W4377013551 hasConceptScore W4377013551C41008148 @default.
- W4377013551 hasConceptScore W4377013551C50522688 @default.
- W4377013551 hasConceptScore W4377013551C50644808 @default.
- W4377013551 hasConceptScore W4377013551C57869625 @default.
- W4377013551 hasConceptScore W4377013551C62520636 @default.
- W4377013551 hasConceptScore W4377013551C78458016 @default.
- W4377013551 hasConceptScore W4377013551C86803240 @default.
- W4377013551 hasConceptScore W4377013551C99498987 @default.
- W4377013551 hasFunder F4320332999 @default.
- W4377013551 hasIssue "2" @default.