Matches in SemOpenAlex for { <https://semopenalex.org/work/W4286795904> ?p ?o ?g. }
Showing items 1 to 69 of
69
with 100 items per page.
- W4286795904 abstract "The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance $sigma^2$ of the parameters at initialization $theta_0$. For DLNs of width $w$, we show a phase transition w.r.t. the scaling $gamma$ of the variance $sigma^2=w^{-gamma}$ as $wtoinfty$: for large variance ($gamma<1$), $theta_0$ is very close to a global minimum but far from any saddle point, and for small variance ($gamma>1$), $theta_0$ is close to a saddle point and far from any global minimum. While the first case corresponds to the well-studied NTK regime, the second case is less understood. This motivates the study of the case $gamma to +infty$, where we conjecture a Saddle-to-Saddle dynamics: throughout training, gradient descent visits the neighborhoods of a sequence of saddles, each corresponding to linear maps of increasing rank, until reaching a sparse global minimum. We support this conjecture with a theorem for the dynamics between the first two saddles, as well as some numerical experiments." @default.
- W4286795904 created "2022-07-24" @default.
- W4286795904 creator A5058306146 @default.
- W4286795904 creator A5061199946 @default.
- W4286795904 creator A5063861960 @default.
- W4286795904 creator A5071085659 @default.
- W4286795904 creator A5079254681 @default.
- W4286795904 date "2021-06-30" @default.
- W4286795904 modified "2023-10-17" @default.
- W4286795904 title "Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity" @default.
- W4286795904 doi "https://doi.org/10.48550/arxiv.2106.15933" @default.
- W4286795904 hasPublicationYear "2021" @default.
- W4286795904 type Work @default.
- W4286795904 citedByCount "0" @default.
- W4286795904 crossrefType "posted-content" @default.
- W4286795904 hasAuthorship W4286795904A5058306146 @default.
- W4286795904 hasAuthorship W4286795904A5061199946 @default.
- W4286795904 hasAuthorship W4286795904A5063861960 @default.
- W4286795904 hasAuthorship W4286795904A5071085659 @default.
- W4286795904 hasAuthorship W4286795904A5079254681 @default.
- W4286795904 hasBestOaLocation W42867959041 @default.
- W4286795904 hasConcept C114466953 @default.
- W4286795904 hasConcept C114614502 @default.
- W4286795904 hasConcept C121332964 @default.
- W4286795904 hasConcept C126255220 @default.
- W4286795904 hasConcept C134306372 @default.
- W4286795904 hasConcept C199360897 @default.
- W4286795904 hasConcept C2524010 @default.
- W4286795904 hasConcept C2681867 @default.
- W4286795904 hasConcept C2777127463 @default.
- W4286795904 hasConcept C2778049214 @default.
- W4286795904 hasConcept C2779886137 @default.
- W4286795904 hasConcept C2780990831 @default.
- W4286795904 hasConcept C33923547 @default.
- W4286795904 hasConcept C41008148 @default.
- W4286795904 hasConcept C62520636 @default.
- W4286795904 hasConcept C99844830 @default.
- W4286795904 hasConceptScore W4286795904C114466953 @default.
- W4286795904 hasConceptScore W4286795904C114614502 @default.
- W4286795904 hasConceptScore W4286795904C121332964 @default.
- W4286795904 hasConceptScore W4286795904C126255220 @default.
- W4286795904 hasConceptScore W4286795904C134306372 @default.
- W4286795904 hasConceptScore W4286795904C199360897 @default.
- W4286795904 hasConceptScore W4286795904C2524010 @default.
- W4286795904 hasConceptScore W4286795904C2681867 @default.
- W4286795904 hasConceptScore W4286795904C2777127463 @default.
- W4286795904 hasConceptScore W4286795904C2778049214 @default.
- W4286795904 hasConceptScore W4286795904C2779886137 @default.
- W4286795904 hasConceptScore W4286795904C2780990831 @default.
- W4286795904 hasConceptScore W4286795904C33923547 @default.
- W4286795904 hasConceptScore W4286795904C41008148 @default.
- W4286795904 hasConceptScore W4286795904C62520636 @default.
- W4286795904 hasConceptScore W4286795904C99844830 @default.
- W4286795904 hasLocation W42867959041 @default.
- W4286795904 hasOpenAccess W4286795904 @default.
- W4286795904 hasPrimaryLocation W42867959041 @default.
- W4286795904 hasRelatedWork W1419571898 @default.
- W4286795904 hasRelatedWork W2069499874 @default.
- W4286795904 hasRelatedWork W2470003561 @default.
- W4286795904 hasRelatedWork W2761104639 @default.
- W4286795904 hasRelatedWork W2963964579 @default.
- W4286795904 hasRelatedWork W2966614174 @default.
- W4286795904 hasRelatedWork W2999007559 @default.
- W4286795904 hasRelatedWork W3113763809 @default.
- W4286795904 hasRelatedWork W4287717979 @default.
- W4286795904 hasRelatedWork W4311892914 @default.
- W4286795904 isParatext "false" @default.
- W4286795904 isRetracted "false" @default.
- W4286795904 workType "article" @default.